Proxies in Scrapy

A minimal spider

Nothing needs switching on: HttpProxyMiddleware sits in the default pipeline and reads the proxy key from meta. The credentials go inside the address, exactly as in a normal connection string.

import scrapy


class IpSpider(scrapy.Spider):
    name = "ip"
    proxy = "http://LOGIN__cr.de:PASSWORD@gw.dataimpulse.com:823"

    def start_requests(self):
        yield scrapy.Request(
            "https://ipinfo.io/json",
            meta={"proxy": self.proxy},
        )

    def parse(self, response):
        self.logger.info(response.text)

What to keep in mind while scraping

A retry on port 823 arrives from a different address — that is rotation working, not a fault. Scrapy retries behave correctly here, but nothing may depend on the address staying put between requests; the sticky ports 10000–20000 exist for that.

Set DOWNLOAD_TIMEOUT generously: a residential exit answers slower than a datacenter one, and a request without a timeout can hang for minutes. The plan allows up to two thousand concurrent connections, which ordinary scraping will not approach.