Recent studies have shown that today's popular Web search engines cover only a relatively small fraction of the Web's URLs. (See our July 12, 1999 NewsBreak, "New Study of WWW Search Engine Coverage Published.") Now two Web search engine services have promised to rectify that problem with a massive expansion of coverage. Fast Search and Transfer (FAST), a Norwegian company, has unveiled "The World's Biggest Search Engine," which, it claims, scans 200 million of the Web's estimated 800 million pages. Excite@Home claims its new beefed-up search engine operation has dozens of new spiders, each capable of handling the 50 million pages that it previously took all of Excite's efforts to index. With the Web expected to grow to 1 billion pages within a year or two, this effort marks a necessary move to keep pace with its growth.A July 1999 Nature magazine article indicated that the percentage of Web coverage has actually dropped since December 1997, until no single service covers more than 16 percent of the Web's estimated 800 million pages. Currently the reported leaders in coverage of Web URLs are Northern Light, with 150 million pages, and Inktomi Corporation, with 110 million pages.
From Oslo, Fast Search and Transfer ASA (http://www.fast.no) has already reached 200 million pages with the August expansion of its "World's Biggest Search Engine" (http://www.alltheweb.com) and promises to keep expanding. It also promises to provide search results in 1 second and to update its database every 15 days, purging dead, out-of-date Web pages.
FAST uses a collection of Dell Computer Corporation PowerEdge servers and Dell PowerVault storage subsystems. They boast a parallel server architecture dissimilar to that used by most major search engines. The systems operate in parallel to distribute user queries, search the document catalog, and spider search the Web to scale linearly in both query volume (number of searches) and catalog size (number of documents). By working in parallel, FAST Search can build its document index in only 12 hours.
Forty-five percent of FAST is owned by the Norwegian firm Opticom ASA, the company from which it spun off in 1997. It has U.S. offices in Boston and San Francisco. FAST trades on the over-the-counter market in Oslo but plans a public offering before the end of the year. In the last few months, the company has received $11.5 million in funding in private placements and from stock sales. To generate revenue, it licenses and revenue-shares FAST Search to major portals, search engines, ISPs, and content sites, charging between $1 and $4 per 1,000 queries. The company also develops and markets image/video compression products.
Excite (http://www.excite.com) also announced a mid-August massive expansion of its Web search coverage. Currently Excite indexes about 50 million Web pages using around 10 spiders. It plans to add dozens more spiders, each capable of covering 50 million pages. Excite spokespeople claim their system will ultimately be able to reach all 800 million Web pages, though some commentators, such as Danny Sullivan of SearchEngineWatch, consider that claim more publicity than reality.
At the same time, Excite has also announced improvements in methods for identifying and tracking Web page content, detecting and suppressing spam, and boosting visibility of top Web quality content on search results pages. It will highlight user-specific content, including vertical collections such as news articles, online discussions, audio clips, and featured Web sites.
Excite@Home was formed through the merger of the portal/search engine Excite with AtHome Corporation, producer of cable-based Internet access reaching an estimated 67 million homes. AtHome paid $6.7 billion for Excite in a stock deal.
