How To Follow Redirects With Curl

When you send a request with curl and the server responds with a 301, 302, or 307 status code, curl does not follow that redirect by default. This catches many developers off guard, especially when building scrapers or automated data collection pipelines where landing pages frequently redirect to regional or localized URLs before returning content.

Understanding how to configure curl to handle redirects properly is essential for reliable scraping. It becomes even more important when routing requests through proxies, since redirects can expose your actual IP, leak headers, or behave differently depending on the proxy protocol you are using.

Why curl Does Not Follow Redirects by Default

The curl command-line tool is designed to be explicit. By default, it returns exactly what the server sends back — including 3xx redirect responses — without automatically chasing the new location. This design is intentional: it gives you full visibility into what a server is actually returning, which matters when debugging APIs or verifying that redirect chains are set up correctly.

For casual testing, this behavior is useful. For scraping or data collection at scale, it means you must opt in to redirect following every time, or configure it as part of your script defaults.

The Core Flag: -L or --location

The primary way to tell curl to follow redirects is the -L flag, also written as --location. Adding this to any curl command instructs it to re-issue the request to whatever URL appears in the server's Location header.

For example, a basic request that follows redirects looks like this:

curl -L https://example.com/page

This will transparently follow any redirect chain until it reaches a final response. You can combine it with other flags such as -v (verbose) to see each hop in the chain, which is helpful when diagnosing where a redirect leads or whether it is looping.

Controlling the Maximum Number of Redirects

By default, curl will follow up to 30 redirects before stopping and returning an error. For most scraping tasks this is more than enough, but when dealing with misconfigured servers or unusual redirect chains, you may want to set this explicitly using --max-redirs.

• --max-redirs 5 — caps redirect following at five hops, useful for preventing runaway loops
• --max-redirs -1 — removes the cap entirely (use with caution)
• --max-redirs 0 — disables redirect following even if -L is set

Setting a sensible cap is good practice in automated pipelines, where an unexpected redirect loop could otherwise cause a script to hang or consume excessive resources.

How Proxies Interact With Redirect Handling

When you route curl through a proxy, redirect behavior becomes more nuanced. Each redirect causes curl to issue a new HTTP request, and that new request also passes through the proxy. This is actually desirable for web scraping proxies, because it means your outbound IP stays consistent for the entire redirect chain rather than switching mid-session.

However, there are scenarios where this creates problems:

• Some servers redirect HTTP to HTTPS. If your proxy only handles one protocol, the redirect may fail silently.
• Redirect targets may land on different domains. Certain proxy types restrict traffic to the original host, breaking cross-domain redirects.
• Cookies set during a redirect are sometimes not forwarded correctly, which matters for session-based scraping tasks.

Using rotating proxies adds another layer of consideration: if your proxy rotates IP addresses between requests, consecutive redirect hops may appear to come from different IPs, which some servers detect as suspicious activity. For redirect-heavy scraping targets, sticky sessions or session-pinned proxies tend to produce more reliable results.

Preserving the Request Method Across Redirects

By default, curl converts POST requests to GET when following a 301 or 302 redirect. This matches browser behavior but is often not what you want in a scraping or API context. To preserve the original HTTP method across redirects, use the --post301, --post302, or --post303 flags depending on which status codes you expect to encounter.

Alternatively, for proxies for scraping scenarios where you are replaying form submissions or API calls, you may find it cleaner to detect the redirect manually using -I (HEAD request) first, capture the Location header, and then issue a fresh POST to the resolved URL directly.

Debugging Redirect Chains With Verbose Output

The -v flag is your best tool when redirect behavior is not what you expect. It prints each request and response header, making it straightforward to trace exactly which URLs curl is visiting and what status codes are being returned at each hop.

For data collection proxies setups, combining verbose output with curl's --write-out option lets you log structured data about each redirect, including the final effective URL (using %{url_effective}) and the total number of redirects followed (using %{num_redirects}). This metadata is valuable when auditing large scraping runs to confirm that redirects resolved as expected.

If you are evaluating proxy providers for redirect-heavy scraping work, Cheapest Proxies is worth considering for buyers comparing affordable proxy services, particularly if your use case involves high request volumes where per-request cost adds up quickly.

Practical Tips for Redirect Handling in Scraping Scripts

When building scripts that rely on curl for data collection, a few habits will save significant debugging time:

• Always include -L unless you have a specific reason to inspect raw redirect responses.
• Set an explicit --max-redirs value to prevent loops from stalling your pipeline.
• Log the effective URL after each request to confirm you landed on the intended page.
• Test redirect behavior without a proxy first, then retest through your proxy to isolate any proxy-introduced issues.
• For session-sensitive targets, prefer sticky session proxies over purely rotating ones to keep cookie state intact across redirect hops.