courlan.core

Core URL checking, link extraction, and filtering.

For usage examples, see the Python Usage guide.

Core functions needed to make the module work.

courlan.core.check_url(url, strict=False, with_redirects=False, language=None, with_nav=False, trailing_slash=True)

Check links for appropriateness and sanity

Parameters:

• url (str) – url to check

• strict (bool) – set to True for stricter filtering

• with_redirects (bool) – set to True for redirection test (per HTTP HEAD request)

• language (str | None) – set target language (ISO 639-1 codes)

• with_nav (bool) – set to True to include navigation pages instead of discarding them

• trailing_slash (bool) – preserve trailing slashes (default True); when False, strip them from paths without a query string. With the default True and strict=False, a bare root slash is only kept when a query or fragment follows it, otherwise it is stripped

Returns:

A tuple consisting of canonical URL and extracted domain

Raises:

Nothing – invalid URLs are caught internally and None is returned.

Return type:

tuple[str, str] | None

courlan.core.extract_links(pagecontent, url=None, external_bool=False, *, no_filter=False, language=None, strict=True, trailing_slash=True, with_nav=False, redirects=False, reference=None, base_url=None)

Filter links in a HTML document using a series of heuristics

Parameters:

• pagecontent (str) – whole page as a string

• url (str | None) – full URL of the original page

• external_bool (bool) – set to True for external links only, False for internal links only

• no_filter (bool) – override settings and bypass checks to return all possible URLs

• language (str | None) – set target language (ISO 639-1 codes)

• strict (bool) – set to True for stricter filtering

• trailing_slash (bool) – preserve trailing slashes (default True); when False, strip them from paths without a query string. With the default True and strict=False, a bare root slash is only kept when a query or fragment follows it, otherwise it is stripped

• with_nav (bool) – set to True to include navigation pages instead of discarding them

• redirects (bool) – set to True for redirection test (per HTTP HEAD request)

• reference (str | None) – provide a host reference for external/internal evaluation

• base_url (str | None)

Returns:

A set containing filtered HTTP links checked for sanity and consistency.

Raises:

ValueError – if the deprecated ‘base_url’ argument is provided.

Return type:

set[str]

courlan.core.filter_links(htmlstring, url, *, lang=None, rules=None, external=False, strict=False, with_nav=True, base_url=None)

Find links in a HTML document, filter and prioritize them for crawling purposes.

Parameters:

• htmlstring (str)

• url (str | None)

• lang (str | None)

• rules (RobotFileParser | None)

• external (bool)

• strict (bool)

• with_nav (bool)

• base_url (str | None)

Return type:

tuple[list[str], list[str]]