API Reference

This section is generated from the courlan package. Click a module to jump to its reference.

Modules

• courlan.core
• courlan.filters
• courlan.clean
• courlan.urlutils
• courlan.urlstore
• courlan.sampling
• courlan.settings

Cache management

courlan.meta

clear_caches() resets the LRU caches behind langcodes_score and get_registrable_domain, as well as urllib.parse’s own cache. Call it in long-running processes to release memory.

Meta-functions to be applied module-wide.

courlan.meta.clear_caches()

Reset all known LRU caches used to speed up processing. This may release some memory.

Return type:

None

Internal modules

courlan.langcodes

CLDR-derived language and territory code sets plus the langcodes_score helper, which drives path- and subdomain-based language detection.

Language detection for URL segments: the scoring helper plus the CLDR-derived language/territory code sets it checks (from babel 2.18, no longer a dependency).

courlan.langcodes.langcodes_score(language, segment)

Score a URL segment as a language cue: +1 matching locale, -1 mismatching, 0 unrecognized.

Parameters:

• language (str)

• segment (str)

Return type:

int

courlan.hosts

Host utilities shared by urlutils and filters: IP-literal canonicalization, IDNA/punycode encoding, and public-suffix (eTLD+1) extraction via get_registrable_domain.

Host utilities shared by urlutils and filters: IP-literal canonicalization, IDNA/punycode encoding, public-suffix data and eTLD+1 extraction.

courlan.hosts.get_registrable_domain(host)

Return (domain_label, registrable_domain) from a host name.

Parameters:

host (str | None)

Return type:

tuple[str | None, str | None]

courlan.network

HTTP redirect resolution used by check_url(with_redirects=True).

Functions devoted to requests over the WWW.

courlan.network.redirection_test(url)

Test final URL to handle redirects :param url: url to check

Returns:

The final URL seen.

Raises:

ValueError – if the URL cannot be reached or returns an unacceptable status.

Parameters:

url (str)

Return type:

str

courlan.cli

Entry point for the courlan command. See the CLI reference for usage.

Implements a basic command-line interface.

courlan.cli.main()

Run as a command-line utility.

Return type:

None

courlan.cli.parse_args(args)

Define parser for command-line arguments

Parameters:

args (Sequence[str] | None)

Return type:

Namespace

courlan.cli.process_args(args)

Start processing according to the arguments

Parameters:

args (Namespace)

Return type:

None