The Unreasonable Effectiveness of Data

The Unreasonable Effectiveness of Data,10.1109/MIS.2009.36,IEEE Expert / IEEE Intelligent Systems,Alon Y. Halevy,Peter Norvig,Fernando Pereira

The Unreasonable Effectiveness of Data   (Citations: 40)
BibTex | RIS | RefWorks Download
At Brown University, there is excitement of having access to the Brown Corpus, containing one million English words. Since then, we have seen several notable corpora that are about 100 times larger, and in 2006, Google released a trillion-word corpus with frequency counts for all sequences up to five words long. In some ways this corpus is a step backwards from the Brown Corpus: it's taken from unfiltered Web pages and thus contains incomplete sentences, spelling errors, grammatical errors, and all sorts of other errors. It's not annotated with carefully hand-corrected part-of-speech tags. But the fact that it's a million times larger than the Brown Corpus outweighs these drawbacks. A trillion-word corpus - along with other Web-derived corpora of millions, billions, or trillions of links, videos, images, tables, and user interactions - captures even very rare aspects of human behavior. So, this corpus could serve as the basis of a complete model for certain tasks - if only we knew how to extract the model from the data.
Journal: IEEE Expert / IEEE Intelligent Systems - EXPERT , vol. 24, no. 2, pp. 8-12, 2009
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
Sort by: