In a general search, the top two news results are displayed above the Web search results with a link to additional news stories. The clustered results for a search are presented as labeled folders (à la Northern Light) in a box at the top of the results page. According to the announcement, these category folders are pulled from the Open Directory Project (ODP) or created on the fly if there's no ODP category match.

Users can choose to search for news-only results. The advanced search capability will show a screen that allows users to select categories of news, such as U.S. News, International, Business, Sports, Technology, and several others, or to search all categories. Users can also limit the search by freshness, ranging from 2 hours to the past week. FAST continuously crawls and updates its news catalog, reportedly at the rate of up to 800 articles per minute, and does a complete crawl of the 3,000 news sources every 2 hours.

I asked Rob Rubin, general manager of FAST's Internet Business Unit, about those 3,000 sources. While he indicated that searchers couldn't currently get a list of the sources indexed, the company was considering making this available in a later release. He provided an example of a query syntax that searchers could use to check for a particular source. To look for inclusion of CNN.com, for example, a user would type <:url.all:cnn.com>. He said that while the recent enhancements are designed to aid the average Web searcher, the company wanted AllTheWeb to be attractive to professional searchers as well.

Rubin elaborated on how the technology worked to achieve the best possible results. "Two of the pillars of great search are freshness and relevancy," he said. "By integrating news into AllTheWeb we have clearly improved freshness, which in turn improves relevancy. We've increased relevancy even more by introducing linguistic technology that interprets the user's intention instead of just matching words and phrases."

Rubin said FAST bought a small German linguistics company last year. That company's technology is now being deployed in a number of ways, including automatic phrase detection and "anti-phrasing," which removes the nonessential parts of a query.

John M. Lervik, FAST's CEO, said: "With this release, AllTheWeb further improves its lead in search engine freshness and raises the bar on relevancy. People now have a single resource they can count on to deliver a combination of fresh Web results, news, software downloads, multimedia, and music files. We're constantly focused on providing the best possible search experience to our users and private-label search partners."

FAST's enhanced search technology is available on AllTheWeb first, where the company hopes a loyal user base of searchers, press, analysts, and consultants will try it out. These advanced features will then be made available to all of FAST's portal customers, such as Terra Lycos. FAST claims to be the only search technology company whose core architecture is the same for both searching the Web and searching enterprise data.

FAST has a refresh cycle of 9 to 11 days on its main index of over 600 million Web pages. While Google claims to have 1.6 billion pages, Rubin said how you count matters and that most of Google's count consists of text links, adding that it does a crawl only every 28 days. "FAST is fresher and very comprehensive and is committed to winning the size race." He said, "In the next quarter you will see us reach 2 billion pages."

The new AllTheWeb quickly caught the attention of all the well-known search engine gurus and has already garnered wide acclaim for its enhancements. Chris Sherman, writing in his SearchDay newsletter, said: "FAST Search & Transfer isn't known for stealth technology, but it should be. Its AllTheWeb.com search engine has snuck up on all of the major search engines with new features, speed, and customization capabilities, and is now a viable challenger to pack leader Google." Kevin Elliot, About.com's Web Search guide, wrote: "I have tested and compared between AllTheWeb, Google, AltaVista, and others over the past few days. The short verdict: AllTheWeb Rules!"

In my tests, AllTheWeb also proved to be a viable contender for a preferred search tool, especially for news. While finishing this report and testing the site, I received a phone call from my husband who was stuck in the Atlanta airport. Shortly thereafter, I was able to find a number of breaking stories about an airport evacuation due to a security breach. At Yahoo! News I found only a Reuters report, and Moreover.com listed nothing.

AllTheWeb did perform better in tests of general and popular news topics than for more specialized news within the information industry—something I track closely. In some cases, I found additional and unique coverage of recent news by searching the free headlines at Moreover.com. In any case, the new version of AllTheWeb certainly warrants the attention of serious Web searchers, especially those looking for an integrated search and news resource.