How to Avoid IP Bans When Scraping: A Polite Crawler Guide

<h2>Why Sites Block Scrapers (and Why That Is Normal)</h2> <p>Getting blocked while scraping is rarely personal. A website cannot tell the difference between your careful research script and a bot hammering its servers at a hundred requests per second — unless you behave differently. Blocking is how sites protect their bandwidth, keep pages fast for human visitors, and stop aggressive crawlers from copying content at scale.</p> <p>Most blocks are triggered by patterns, not by a single request. The usual suspects are: too many requests from one IP in a short window, identical headers on every request, no pauses between pages, requests at superhuman speed around the clock, and crawling URLs a human would never visit in that order. Fix the pattern and most blocks disappear. That is the whole idea behind the polite crawler playbook: act like a considerate guest, and the site has little reason to throw you out.</p> <p>It is worth stating the boundary up front. This guide is about collecting public data responsibly: respecting a site's terms of service, obeying robots.txt, and never bypassing logins, paywalls, or technical barriers to reach data you are not meant to see. If a site offers an official API, use it. If it clearly says no automated access, respect that. Polite crawling is both an ethical choice and a practical one — scrapers that ignore the rules get banned faster, and stay banned.</p> <h2>The Polite Crawler Mindset: Take Only What You Need</h2> <p>Before writing any code, decide exactly which pages and fields you need. A crawler that fetches every page on a site to extract one price is doing ten times the work of a crawler that goes straight to the product pages. Narrow your URL list, skip images, stylesheets, and scripts you do not need, and cache pages locally so a bug in your parser does not force a second full crawl.</p> <p>Three habits separate polite crawlers from noisy ones. First, identify yourself honestly with a descriptive user agent that includes a contact address — many site operators will email you before blocking you if they can reach you. Second, crawl during the site's off-peak hours when you have a large job, rather than adding load at peak traffic. Third, build in a kill switch: if error rates spike, your crawler should pause itself and alert you instead of retrying furiously until the whole IP range is banned.</p> <h2>Design Your Request Rate Before You Write the Loop</h2> <p>Rate is the single biggest factor in whether you get banned. A human reading product pages takes several seconds per page; a script can request fifty pages a second. Sites notice that gap instantly. Start slow, watch the response codes, and only then increase speed — and never exceed what the site can comfortably serve.</p> <p>Use these starting points as rules of thumb, not as guarantees. Every site has its own tolerance, and the only reliable signal is its actual responses to your traffic.</p> <table> <tr><th>Site type</th><th>Suggested delay between requests</th><th>Concurrency</th></tr> <tr><td>Small site or personal blog</td><td>5–10 seconds</td><td>1 request at a time</td></tr> <tr><td>Mid-size e-commerce store</td><td>2–5 seconds</td><td>1–2 concurrent</td></tr> <tr><td>Large marketplace or directory</td><td>1–3 seconds</td><td>2–5 concurrent</td></tr> <tr><td>Any site that returns 429s</td><td>Back off immediately (see below)</td><td>Drop to 1, then pause</td></tr> </table> <p>Add randomness to your delays. A perfectly regular one-request-every-two-seconds pattern is itself a bot signature. A delay of two seconds plus a random zero-to-two seconds looks far more human, and costs you almost nothing in total runtime. Also check the site's robots.txt for a crawl-delay directive — if the operator asks for ten seconds, ten seconds is the floor, not a suggestion to negotiate.</p> <h2>Header and Fingerprint Hygiene</h2> <p>Headers are your crawler's body language. Sending the exact same user agent, accept-language, and header order on ten thousand requests is a fingerprint, even if the IP address changes. Rotate through a small set of realistic, current browser user agents, keep accept-language consistent with the country you are browsing from, and send the headers a real browser would send: accept, accept-encoding, and a plausible referer for follow-up page views.</p> <p>Cookies deserve the same care. Many sites set session cookies on the first visit and expect them on later requests. A crawler that discards cookies and re-requests the homepage before every product page looks broken. Keep one cookie jar per browsing session, and keep one session per identity — if you rotate IPs, do it between sessions, not in the middle of one. A checkout flow or login sequence that suddenly changes country mid-session is a classic fraud signal.</p> <h3>What not to fake</h3> <p>Do not spoof a Googlebot user agent, do not forge headers to impersonate another service, and do not rotate user agents dozens of times per minute. Realism is a narrow band: a handful of genuine browser profiles, used consistently within each session, beats a thousand randomly generated ones.</p> <h2>Handle 429s and Blocks Gracefully</h2> <p>A 429 (Too Many Requests) is a site asking you to slow down. A 403 after a run of successes usually means a rule tripped. In both cases the correct response is the same: stop, wait, and back off — never retry immediately in a tight loop. Exponential backoff with jitter is the standard pattern: wait a short base delay, double it after each failure, and add randomness so a fleet of workers does not retry in lockstep.</p> <pre><code>import random, time, requests

def fetch_with_backoff(url, proxies, max_retries=5): for attempt in range(max_retries): try: resp = requests.get(url, proxies=proxies, timeout=30) if resp.status_code == 200: return resp if resp.status_code in (429, 503): wait = (2 ** attempt) + random.uniform(0, 2) print(f"Rate limited, waiting {wait:.1f}s") time.sleep(wait) continue resp.raise_for_status() except requests.RequestException as e: wait = (2 ** attempt) + random.uniform(0, 2) print(f"Request failed ({e}), retrying in {wait:.1f}s") time.sleep(wait) return None </code></pre>

<p>Two details matter here. First, treat repeated 429s as information, not as an obstacle to brute-force through: if you are still being limited after several backoffs, your base rate is too high and needs lowering. Second, log every failure with the URL, timestamp, status code, and the IP that served the request. When a ban does happen, that log is what lets you diagnose whether the cause was rate, headers, or one poisoned IP in your pool.</p> <h2>Rotate IPs Without Looking Like an Attack</h2> <p>IP rotation spreads your requests across many addresses so no single IP crosses a site's rate threshold. It works, but only as part of the playbook above — rotation alone will not save a crawler that requests a page every 200 milliseconds with identical headers. Sites also track behavior across IPs: identical request timing, identical header order, and identical crawl paths can link rotated requests back to one actor.</p> <p>Residential proxies are the usual choice for scraping because they come from real consumer internet connections, so they look like ordinary visitors. Rotating residential IPs suit broad crawls across many pages, while sticky sessions — keeping the same IP for 10 or 30 minutes — suit multi-step flows such as logging in, searching, and paging through results. Static ISP IPs suit long-running monitoring jobs where the same address checking the same pages on a schedule looks perfectly normal. Match the IP type to the job instead of defaulting to one tool for everything, and keep your pool healthy: retire IPs that start returning blocks, and never keep hammering a site from an IP that has already been flagged.</p> <h2>Read robots.txt Before You Crawl</h2> <p>Every polite crawler starts with robots.txt. Fetch https://example.com/robots.txt and read three things: which paths are disallowed for all crawlers, whether a crawl-delay is specified, and whether a sitemap is listed. A sitemap is a gift — it tells you exactly which pages the site considers public and worth indexing, so you can crawl that list instead of discovering URLs by brute force.</p> <p>Disallowed paths are not a puzzle to solve. Admin pages, internal search results, and checkout flows are disallowed because crawling them wastes the site's resources or touches non-public data. Skip them. If the data you need lives behind a disallowed path, that is a signal to ask the site owner for permission or an API — not to find a way around the rule.</p> <h2>When to Use an Official API Instead</h2> <p>Sometimes the polite answer is not to crawl at all. If a site publishes a product, pricing, or search API, it is faster, more stable, and explicitly permitted — and it usually returns cleaner structured data than HTML parsing ever will. APIs also survive site redesigns that would break a scraper overnight. Before building a crawler, spend fifteen minutes checking the site's developer pages, its robots.txt, and its terms. A crawler you do not have to build is the one that never gets banned.</p> <h2>The Polite Crawler Checklist</h2> <table> <tr><th>Check</th><th>Why it matters</th></tr> <tr><td>robots.txt read, disallowed paths skipped</td><td>Respects the operator's stated boundaries</td></tr> <tr><td>Request delay with randomness, crawl-delay honored</td><td>Avoids rate-limit triggers</td></tr> <tr><td>Realistic headers, one session per identity</td><td>Avoids fingerprint-based blocks</td></tr> <tr><td>Exponential backoff on 429/503, never tight retry loops</td><td>Turns a warning into a pause, not a ban</td></tr> <tr><td>IP rotation matched to the job, flagged IPs retired</td><td>Spreads load without looking like an attack</td></tr> <tr><td>Pages cached, only needed URLs fetched</td><td>Less load on the site, faster reruns for you</td></tr> <tr><td>Official API checked first</td><td>The most reliable source is the permitted one</td></tr> </table> <p>Avoiding IP bans is not about finding a clever trick the site has not thought of. It is about removing every reason the site has to block you: reasonable speed, honest identification, respect for its rules, and graceful behavior when it asks you to slow down. Crawlers built that way run for months without drama. Crawlers built the other way spend their lives rotating through burned IPs and wondering why nothing works.</p> <h2>FAQ</h2> <h3>How many requests per minute is safe when scraping?</h3> <p>There is no universal number — it depends on the site's size, infrastructure, and rules. A small site may struggle with ten requests a minute, while a large marketplace may tolerate several per second. Start with a delay of a few seconds between requests, watch for 429 responses, and let the site's actual responses set your rate.</p> <h3>Will using a proxy alone stop IP bans?</h3> <p>No. A proxy changes which IP address the site sees, but it does not fix aggressive timing, identical headers, or crawling disallowed paths. Proxies work best combined with polite rates, realistic headers, and backoff handling — they spread polite traffic, they do not disguise rude traffic.</p> <h3>Is web scraping legal?</h3> <p>It depends on what you collect, how you collect it, and where you operate. Scraping publicly available data while respecting a site's terms and robots.txt is widely practiced for research, price monitoring, and market analysis, but bypassing logins or paywalls, violating terms of service, or collecting personal data can create legal risk. When in doubt, check the site's terms and get permission first.</p> <h3>What should I do after getting a 403 or IP ban?</h3> <p>Stop crawling that site immediately and diagnose before retrying. Check your logs for what changed: a rate increase, a new header pattern, or a blocked IP. Fix the cause, wait — sometimes hours, sometimes a day — and resume at a slower rate from a clean IP. Retrying aggressively from the same setup is the fastest way to turn a temporary block into a permanent one.</p> <h2>Try It</h2> <p>Before your next crawl, check that your proxy setup is actually working: run your IPs through <a href="https://ipstation.cc/tools/proxy-test">IPStation's free proxy tester</a> to verify anonymity, location, and speed. If you need residential IPs for a scraping project, see the current plans on the <a href="https://ipstation.cc/pricing">pricing page</a> — rotating residential for broad crawls, sticky sessions for multi-step flows, and static ISP IPs for steady monitoring jobs.</p>