CensorAlert generates text embeddings (OpenAI text-embedding-3-small) from each item's summary, title, and tags to cluster semantically similar reports within configurable time windows; near-duplicates such as reposts, copied headlines, and translations are collapsed into a single canonical post that preserves all original source URLs and metadata.

CensorAlert aggregates censorship signals from heterogeneous open sources — including OONI, Cloudflare Radar, NetBlocks, GitHub Issues of circumvention tools (Hysteria, Xray), Net4People BBS, NTC Party forum, Mastodon, X, Telegram channels, and arXiv — normalizing each item into a common schema preserving timestamp, source, raw data, and provenance with a link to the original content.

§2 (Data Sources) deployment

Systematic measurement platforms (OONI, Censored Planet, Cloudflare Radar, NetBlocks) have inherent blind spots due to geographic coverage and protocol-specific test constraints; critical censorship discoveries — including GFW fully-encrypted protocol blocking and regional Chinese censorship — were first surfaced by user reports on forums and GitHub issue pages of circumvention tools, not by automated measurement infrastructure.

§1 evaluation

CensorAlert's LLM agent scores each ingested item 0–5 on five independent dimensions — credibility, novelty, impact, timeliness, and verifiability — then computes a normalized significance score (0–10) from these components; items are processed every two hours using OpenAI GPT-5 Thinking (hosted on Azure) constrained to return structured JSON output.

§2 (AI-based Scoring) deployment

The largest single source of censored domains in the GNL is MESA lab's SNI monitoring dataset (E21-SNI-Top200w.txt) containing 57,362 censored domains, and E21-SNI-Top120W-20221020.txt with 36,467 domains—totaling over 93K domains from network tap data alone for a single country (E21 = Ethiopia per InterSecLab attribution). A separate Xinjiang dataset (XJ-CUCC-SNI-Top200w.txt) contains 13,604 domains. These datasets "do not seem to come from popular domain lists, and instead appear to be gathered from network taps," confirming that Geedge builds censorship target lists directly from passive traffic observation.

2026-sheffey-geedge §4.2, Table 3 dpisni-blockingmeasurement-platform

Of 6,915,266 domains extracted from the 572 GiB Geedge Networks Leak (GNL), 298,955 censored domains (93.7% of all GNL-censored domains) appear in neither Tranco top-1M nor CitizenLab test lists. Measurements across China (Guangzhou/Nanjing), Myanmar, Pakistan, and Algeria confirmed censorship via DNS injection and SNI-based TLS connection termination. The GNL covers 25–62% of Tranco-censored domains across countries, showing substantial but incomplete overlap. This vendor-side ground truth reveals a censorship surface roughly two orders of magnitude larger than curated academic test lists.

2026-sheffey-geedge §4.1, Table 2 dpisni-blockingdns-poisoningmeasurement-platform

The Ahmia search engine provided the most onion addresses (18,069 in a single day, ranging 18,000–22,000 week-to-week), outperforming five other sources combined (36,028 total across six engines). However, Ahmia's intentional exclusion blacklist contains 46,000+ hashed addresses, and crawling onion services for 20 days yielded 48,745 unique v3 addresses, 11,809 of which were on Ahmia's blacklist — meaning any index-based collection systematically misses a significant share of the onion ecosystem by design.

2025-h-ller-evaluating §4.1, §4.3 measurement-platform

Combining six onion search engines/repositories plus clearnet search engines, Tor2web-style DNS leakage, and 20 days of self-run crawling (2.9 million pages), the authors assembled 482,614 unique v3 onion addresses — the largest known collection. Verifying against HSDir blinded public keys showed the collected addresses accounted for 25% of observed blinded keys but were responsible for 66% of all successful service descriptor downloads, confirming a heavy-tailed usage distribution.

2025-h-ller-evaluating §4, §5 measurement-platform

152 of 5,478 crawled domains (approximately 2.8%) deployed active bot-detection measures—captcha delivery or perimeter protection—that blocked automated OpenWPM crawling entirely. The authors note this disproportionately excludes untrustworthy sites, biasing the training dataset toward well-resourced trustworthy outlets and limiting recall on the untrustworthy class.

2025-sivan-sevilla-probing §4 Limitations measurement-platform

Wrana et al. systematically assess how well existing surveillance and censorship mechanisms can target users of Future Internet Architectures (FIAs) — including NDN, SCION, XIA, and MobilityFirst — finding that DPI and flow-correlation techniques from the current internet map onto FIA traffic with moderate adaptation. The paper identifies that FIA naming/addressing schemes introduce new censorship attack surfaces (e.g., content-name-based filtering in NDN) not present in IP-based architectures.

2025-wrana-sok-surveillance §4, §5 dpiflow-correlationmeasurement-platform