Ancient Web: Matthew Gray Counted the Web Before There Were a Thousand Sites

Part of the Ancient Web Collection.

Today nobody can comfortably count every public webpage. In December 1993, Matthew Gray's experimental program found just 623 web sites, and that was a remarkable number.

Visit Matthew Gray's Internet Statistics archive.

The Massachusetts Institute of Technology homepage says Gray personally researched much of its data about web growth. It is dated June 20, 1996 and even retains links to a print-friendly copy and an older edition of the report. The page grants permission to reproduce Gray's original statistics with credit, carefully distinguishing his measurements from data supplied by other people.

Its value goes deeper than one numerical table. In 1993 Gray created the World Wide Web Wanderer, an automated program that followed hyperlinks to discover web sites. The goal was to measure how fast the new hypertext network was expanding.

The Web when six hundred sites looked enormous

An original December 1993 message preserved in an Internet mailing-list archive gives a fascinating look at the experiment. Gray reports 130 sites detected in June, 204 in September, 212 in mid-October, 272 in November, and 623 by December 13. These are results of a particular crawler, not a complete census of every existing server.

His note describes the difficulty of discovering obscure pages connected by few links. It points out that web browsers already could reach material from FTP, Gopher, and WAIS services, complicating the question of what counted as part of the Web.

Gray invited owners of missing sites to contact him but warned that his program might not immediately visit them. He also joked about being embarrassed to release its source code because it was messy. These are small comments, but they preserve the human reality of very early web measurement.

The surviving statistics page dates from a later stage of the project, while the mailing-list message documents the 1993 measurements directly. It is important not to collapse those dates into one supposed launch.

This archive records a moment when the Web was becoming too large for people to comprehend by browsing, but was still small enough for an experimental crawler to measure in hundreds.

Read the original 1993 report and Gray's surviving growth-statistics homepage.

Part of the Ancient Web Collection.