courlan — URL Filtering and Normalization for Python

courlan provides an additional “brain” for web crawling, scraping, and document management. It facilitates web navigation through a set of filters to enhance the quality of resulting document collections: save bandwidth by steering clear of low-value pages, identify content by language, and deduplicate URL collections at scale.

Common tasks

Validate & filter URLs

check_url — validate, normalize, and filter in one call.

URL Checking, Cleaning, and Filtering in Python

Clean & normalize

clean_url, normalize_url — fix up messy URLs without full filtering.

URL Checking, Cleaning, and Filtering in Python

Extract links from HTML

extract_links, filter_links — general-purpose and crawl-aware extraction.

URL Checking, Cleaning, and Filtering in Python

Sample by domain

sample_urls — pick N URLs per domain from a larger collection.

URL Checking, Cleaning, and Filtering in Python

URL Store

UrlStore — domain-classified storage with visit tracking and persistence.

URL Store — Domain-Classified URL Storage

Web crawling

Crawl delays, robots.txt, download scheduling, and frontier management.

Web Crawling with Courlan

Command line

courlan CLI — filter and sample URL files from the terminal.

Courlan Command-Line Interface

Settings

Customize blacklists, allowed parameters, and language rules.

courlan.settings

Contents

• Getting Started with Courlan
  • Prerequisites
  • Install
  • Quick check
  • Troubleshooting
  • Where to go next
• URL Checking, Cleaning, and Filtering in Python
  • Checking URLs with check_url
  • Cleaning and normalizing URLs
  • URL parsing and decomposition utilities
  • Extracting links from HTML with extract_links
  • Sampling URLs by domain with sample_urls
  • Scope and crawlability checks
  • Cache management
• URL Store — Domain-Classified URL Storage
  • Basic UrlStore usage
  • UrlStore constructor options
  • Tracking visited and unvisited URLs
  • Adding links from HTML
  • UrlStore statistics and inspection
  • Printing and resetting
  • Discarding domains
  • Saving and loading UrlStore state
  • Thread safety
  • Performance tips
  • Next steps
• Web Crawling with Courlan
  • Basic crawler loop
  • Crawl delays and robots.txt
  • Scheduled downloads
  • Frontier management
  • Extracting links from HTML
  • Best practices
  • Complete example
  • Troubleshooting crawls
• Courlan Command-Line Interface
  • Flags
  • Behavior
  • Examples
  • Large inputs
  • Redirect checking (-r)
• Courlan Troubleshooting & FAQ
  • CLI: nothing written to output
  • Slow performance / high memory
  • Redirect checks are slow
  • UrlStore “missing” URLs
  • Pickle / load_store warnings
  • Language detection surprises
  • Tests failing locally
  • Network timeouts and retries
  • TypeError with language parameter
  • Still stuck?
• Migration notes
• History / Changelog
  • 1.4.0
  • 1.3.2
  • 1.3.1
  • 1.3.0
  • 1.2.0
  • 1.1.0
  • 1.0.0
  • 0.9.5
  • 0.9.4
  • 0.9.3
  • 0.9.2
  • 0.9.1
  • 0.9.0
  • 0.8.3
  • 0.8.2
  • 0.8.1
  • 0.8.0
  • 0.7.2
  • 0.7.1
  • 0.7.0
  • 0.6.0
  • 0.5.0
  • 0.4.2
  • 0.4.1
  • 0.4.0
  • 0.3.1
  • 0.3.0
  • 0.2.3
  • 0.2.2
  • 0.2.1
  • 0.2.0
  • 0.1.0
• API Reference
  • courlan.core
  • courlan.filters
  • courlan.clean
  • courlan.urlutils
  • courlan.urlstore
  • courlan.sampling
  • courlan.settings
  • Cache management
  • Internal modules
• About
  • License
  • Author
  • Similar work
  • References