How To Find All Urls Using Selenium

Extracting all URLs from a web page is one of the most common tasks in web scraping and automated testing. Selenium, because it drives a real browser, captures links that are rendered dynamically by JavaScript — something simpler HTTP-based tools often miss. Whether you are building a site crawler, auditing internal links, or gathering data for research, knowing how to pull every anchor href efficiently is a foundational skill.

This reference covers the core Selenium techniques for finding all URLs on a page using Python, explains how to handle edge cases like relative links and fragment-only anchors, and shows how proxies fit naturally into scraping workflows once your URL-discovery logic is solid.

Why Selenium Is Useful for URL Discovery

Many modern websites load links asynchronously after the initial HTML is served. Standard HTTP requests return only the static source, which may be missing navigation menus, infinite-scroll content, or modal dialogs populated by JavaScript. Selenium renders the full page inside a real browser engine, so every anchor tag present in the final DOM is accessible — including those injected after page load.

This makes Selenium a strong choice for python web scraping projects where completeness matters more than raw speed. The trade-off is higher resource usage per request, so combining it with a controlled proxy rotation strategy is often necessary at scale.

Setting Up Selenium for Link Extraction

Before writing any extraction logic, confirm your environment is ready:

• Install Selenium via pip install selenium.
• Download the matching WebDriver for your browser (ChromeDriver for Chrome, GeckoDriver for Firefox). The driver version must match the installed browser version.
• Optionally install webdriver-manager to handle driver downloads automatically: pip install webdriver-manager.
• For headless (server-side) operation, pass --headless to the browser options so no GUI window is opened.

A minimal headless Chrome setup looks like this in Python:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless")
options.add_argument("--no-sandbox")
driver = webdriver.Chrome(options=options)

Finding All URLs on a Page

Once the driver has loaded a target page with driver.get(url), use find_elements with the By.TAG_NAME locator to retrieve every anchor element, then read the href attribute from each:

from selenium.webdriver.common.by import By

driver.get("https://example.com")
anchors = driver.find_elements(By.TAG_NAME, "a")
urls = [a.get_attribute("href") for a in anchors]

Selenium automatically resolves relative URLs to absolute ones when reading href, so /about becomes https://example.com/about. This saves a manual resolution step that is common in raw HTML parsing.

Filtering and Cleaning the Results

Raw output from the above snippet will include None values (anchors without an href), JavaScript pseudo-URLs like javascript:void(0), and fragment-only links such as #section. Clean the list before processing:

clean_urls = [
    u for u in urls
    if u
    and u.startswith("http")
    and not u.startswith("javascript")
]
# Remove duplicates while preserving order
seen = set()
unique_urls = []
for u in clean_urls:
    if u not in seen:
        seen.add(u)
        unique_urls.append(u)

For web scraping pipelines that follow links recursively, also filter out URLs pointing to external domains if you want to stay within a single site:

from urllib.parse import urlparse

base_domain = urlparse("https://example.com").netloc
internal = [u for u in unique_urls if urlparse(u).netloc == base_domain]

Waiting for Dynamic Content Before Extracting

JavaScript-heavy pages may not have finished rendering all links by the time Selenium's get() call returns. Use explicit waits to ensure the DOM is ready:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.TAG_NAME, "a"))
)

For single-page applications that populate navigation menus only after a user interaction, you may also need to simulate a scroll or click before extracting links. driver.execute_script("window.scrollTo(0, document.body.scrollHeight)") followed by a short explicit wait is a common pattern for infinite-scroll pages.

Integrating Proxies for Reliable Scraping

When running URL-discovery scripts across many pages or on sites with rate limiting, routing requests through proxies is a practical necessity. Selenium supports proxy configuration through browser options:

from selenium.webdriver.common.proxy import Proxy, ProxyType

proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = "your.proxy.host:port"
proxy.ssl_proxy = "your.proxy.host:port"

capabilities = webdriver.DesiredCapabilities.CHROME.copy()
proxy.add_to_capabilities(capabilities)

driver = webdriver.Chrome(desired_capabilities=capabilities, options=options)

For proxies for scraping at any meaningful volume, residential or rotating datacenter proxies help avoid blocks and deliver consistent results. Cheapest Proxies is worth considering for buyers comparing affordable proxy services who need reliable rotating IPs without significant overhead. Always verify that your proxy provider supports HTTPS tunneling, since Selenium routes SSL traffic through the proxy as well.