netgoPython Search Toolkit & Wikimedia APIs

Lightweight Python library for scraping search engines, extracting clean page content, parsing sitemaps, and querying Wikimedia APIs.

Unified Search Engines

Engine-agnostic API querying Google and Bing, returning consistent Result objects with pagination and safe search.

Download any web page and extract the core article text, filtering out navigation, ads, headers, and footers.

Discover sitemaps via robots.txt, parse XML sitemap indexes, and crawl or filter URLs efficiently.

Full-text search, section outlines, summaries, backlinks, categories, and Commons media file inspection.

Resolve titles to QIDs, inspect claims, labels, descriptions, aliases, and run live SPARQL queries.

Look up parts of speech, numbered definitions, etymologies, and cross-language terms across editions.

netgo: Top-level package exports and convenient high-level functions (search, search_many, fetch).
netgo.search: Google and Bing backends, consistent Result models, pagination, error handling.
netgo.page: Article body extraction engine, HTML boilerplate filtering, and Page data model.
netgo.sitemap: Robots.txt discovery, XML/text sitemap parsing, index crawling, and prefix filtering.
netgo.wiki: MediaWiki Action API client, Wikipedia, Wikidata, Commons, and Wiktionary interfaces.

Module	Documentation Link	Documented Classes	Documented Functions
`netgo`	netgo	0	0
`netgo.page`	netgo.page	0	0
`netgo.page.errors`	netgo.page.errors	3	0
`netgo.page.extract`	netgo.page.extract	0	10
`netgo.page.fetch`	netgo.page.fetch	0	1
`netgo.page.models`	netgo.page.models	1	0
`netgo.search`	netgo.search	0	2
`netgo.search.bing`	netgo.search.bing	0	6
`netgo.search.errors`	netgo.search.errors	2	0
`netgo.search.google`	netgo.search.google	0	6
`netgo.search.models`	netgo.search.models	2	0
`netgo.sitemap`	netgo.sitemap	0	0
`netgo.sitemap.errors`	netgo.sitemap.errors	3	0
`netgo.sitemap.fetch`	netgo.sitemap.fetch	0	6
`netgo.sitemap.models`	netgo.sitemap.models	2	0
`netgo.sitemap.parse`	netgo.sitemap.parse	0	4
`netgo.wiki`	netgo.wiki	0	0
`netgo.wiki.client`	netgo.wiki.client	1	3
`netgo.wiki.core`	netgo.wiki.core	0	15
`netgo.wiki.errors`	netgo.wiki.errors	3	0
`netgo.wiki.models`	netgo.wiki.models	11	0
`netgo.wiki.wikidata`	netgo.wiki.wikidata	0	13
`netgo.wiki.wikimedia`	netgo.wiki.wikimedia	0	5
`netgo.wiki.wiktionary`	netgo.wiki.wiktionary	0	7

End-to-end runnable scripts are located in the GitHub Examples Directory and documented on the Code Examples Page.

Example File	Purpose / Feature	GitHub Source
`01_search.py`	Search Google and Bing through the same engine-agnostic API.	View Source
`02_parallel_search.py`	Run many search queries at once with `netgo.search_many`.	View Source
`03_read_page.py`	Fetch any web page and read just its main content.	View Source
`04_wikipedia.py`	Deep-dive into a Wikipedia article with `netgo.wiki`.	View Source
`05_wikidata.py`	From a Wikipedia title to structured Wikidata facts.	View Source
`06_commons.py`	Find Wikimedia Commons files and inspect their metadata.	View Source
`07_wiktionary.py`	Dictionary data from Wiktionary.	View Source
`08_research_pipeline.py`	The full journey in one script: search -> fetch -> enrich with Wikipedia.	View Source
`09_sitemap.py`	Discover and read a site's XML sitemaps.	View Source