Robots exclusions and gaps in historical archive coverage | Dead Internet Theory
Dead Internet Theory: How robots.txt, a file written for search-engine crawlers, has decided which pages get archived and which stored captures can be seen, and what a missing capture actually does and does not prove.