API reference

notnews: News classification library.

A simple, unified library for classifying news articles as hard/soft news using URL patterns, machine learning models, and Large Language Models.

notnews.classify_by_url(df, url_col='url', region='us')[source]

Classify news articles as hard/soft based on URL patterns.

Parameters:

  • df (DataFrame) – DataFrame containing URLs to classify.

  • url_col (str) – Column name containing URLs. Defaults to “url”.

  • region (str) – Region-specific patterns to use (“us” or “uk”). Defaults to “us”.

Returns:

  • hard_news: 1 if URL matches hard news patterns, None otherwise

  • soft_news: 1 if URL matches soft news patterns, None otherwise

Return type:

DataFrame with original columns plus

Raises:

ValueError – If url_col not found in DataFrame or region not supported.

Example

>>> import pandas as pd
>>> import notnews
>>> df = pd.DataFrame({"url": ["cnn.com/politics/election", "espn.com/sports/football"]})
>>> result = notnews.classify_by_url(df, region="us")
>>> print(result[["url", "hard_news", "soft_news"]])
notnews.predict_soft_news(df, text_col='text', region='us')[source]

Predict soft news probability using trained ML models.

Parameters:

  • df (DataFrame) – DataFrame containing article text to classify.

  • text_col (str) – Column name containing article text. Defaults to “text”.

  • region (str) – Region-specific model to use (“us” or “uk”). Defaults to “us”.

Returns:

  • prob_soft_news_{region}: Predicted probability of soft news (0-1)

Return type:

DataFrame with original columns plus

Raises:

ValueError – If text_col not found in DataFrame or region not supported.

Example

>>> import pandas as pd
>>> import notnews
>>> df = pd.DataFrame({"text": ["Election coverage from Washington", "Celebrity wedding photos"]})
>>> result = notnews.predict_soft_news(df, region="us")
>>> print(result[["text", "prob_soft_news_us"]])
notnews.predict_news_category(df, text_col='text')[source]

Predict detailed news categories using US model.

Parameters:

  • df (DataFrame) – DataFrame containing text

  • text_col (str) – Column name containing text

Returns:

  • pred_category: Predicted category

  • prob_soft_news: Probability of soft news categories

Return type:

DataFrame with additional columns

Raises:

  • ValueError – If the requested text column is absent.

  • RuntimeError – If the classifier or vectorizer cannot be loaded.

notnews.classify_with_llm(df, text_col='text', provider='claude', categories=None, api_key=None, model=None)[source]

Classify news articles using Large Language Models.

Parameters:

  • df (DataFrame) – DataFrame containing articles to classify.

  • text_col (str) – Column name containing text to classify. Defaults to “text”.

  • provider (str) – LLM provider to use (“claude” or “openai”). Defaults to “claude”.

  • categories (dict | None) – Custom categories dictionary with descriptions and examples. Uses DEFAULT_CATEGORIES if None.

  • api_key (str | None) – API key for the LLM provider. Uses environment variable if None.

  • model (str | None) – Model name to use. Uses provider defaults if None (claude-3-haiku-20240307 for Claude, gpt-3.5-turbo for OpenAI).

Returns:

  • llm_category: Predicted category name

  • llm_confidence: Confidence score (0-1)

  • llm_reasoning: Brief explanation of classification

Return type:

DataFrame with original columns plus

Raises:

ValueError – If text_col not found in DataFrame or provider not supported.

Example

>>> import pandas as pd
>>> import notnews
>>> df = pd.DataFrame({
...     "text": ["Election results announced", "Celebrity wedding"]
... })
>>> result = notnews.classify_with_llm(df, provider="claude")
>>> print(result[["text", "llm_category"]].head())
notnews.clean_text(text)[source]

Clean and normalize text for machine learning processing.

Performs deterministic tokenization and normalization.

Parameters:

text (object) – Scalar input to clean and normalize. Missing values produce an empty string.

Returns:

Normalized, whitespace-separated text.

Return type:

str

Example

>>> import notnews
>>> clean = notnews.clean_text("The politician announced new policies today!")
>>> print(clean)
the politician announced new policies today
notnews.fetch_web_content(url, timeout=10)[source]

Fetch and extract clean text content from a web page.

Downloads the web page, parses HTML, and extracts the main article content using common content selectors. Automatically cleans extracted text.

Parameters:

  • url (str) – URL to fetch content from. Must include scheme (http/https).

  • timeout (int) – Request timeout in seconds. Defaults to 10.

Returns:

Extracted and cleaned text content, or None if fetching fails or content is too short (< 100 characters).

Return type:

str | None

Example

>>> import notnews
>>> content = notnews.fetch_web_content("https://example.com")
>>> if content:
...     print(f"Extracted {len(content)} characters")