The problem

Collecting public data from Instagram hits two walls: login walls and bot detection.

The stack that works

My internal tool pairs instagrapi with a few tricks:

• Fresh sessions — regenerate the session file, never reuse stale ones
• Rate limits — a sleep between requests tuned per endpoint
• Streamlit frontend — a thin UI on top so teammates can trigger scrapes without touching code

Practical rules

1. Respect the rate limits you set
2. Cache aggressively — never re-fetch what you already stored
3. Log failures distinctly from blocked
4. Keep daily volume deliberately low

Closing

Treat scraping as engineering at scale: the value is in a scraper that stays alive, not in throughput.