This article has been brought to you by Digital Bubble Media. We treat Digital Marketing as an Investment.
During the early years of internet the power to publish websites to a global audience was spoiled by the rudimentary categorisation and search technologies available at the time. The first generation of websites from the early 1990s to about 2002 shared a number of limitations. This was the ‘I go get Web’ where the user sought Web gems among the deluge. Even before the invention of the www it had become clear that tools would be needed to sift through the mass of data available on the Internet. The first search tool was called ‘Archie’, released in 1990 by Alan Emtage, a graduate student at McGill University. Archie allowed users to narrow down the scope of their search for files to specific servers. It searched FTP servers, indexed their contents and allowed users to search for specific files. Archie was limited in that the user had to enter the precise filename of a specific item rather than entering an approximate phrase. One had to know what one sought in advance. Archie would then direct the user to the appropriate FTP server, and the user would then have to manually peruse the server’s folder to find the file. The remarkable thing was that, even to the creator of Archie, the first tool in what would become a multi billion-dollar industry that revolved around searching for things on the Internet, the need for search services was not foreseen:
It was only after we made it known that we had the database and got lots of requests to search it that we realised that there was a need for this kind of service.
In 1993 a search tool for the Gopher system called ‘Veronica’ brought the user directly to the item on the relevant Gopher server, but still searched only for items by precise title. Also in 1993 Matthew Gray at MIT developed ‘www Wanderer’, perhaps the first ‘web crawler’. Wanderer was an automated software known as a ‘bot’ that travelled from website to website gathering data on the number of sites as it went. In 1994 Brian Pinkerton was working on a browser for the NEXT OS and released ‘Web Crawler’. Informed by his experience working on information management at NEXT, Pinkerton made WebCrawler index all text within the documents it found. The crawler and indexer were the core components of a new generation of search engines that could return relevant results without being given precise research requests. Finally, the Web was coming to a point where the newcomer could find material with relative ease.
In the same year as WebCrawler appeared four other search engines were established: AltaVista, Lycos, Excite, and Yahoo! Yahoo!, one of the most popular sites on the Web, began life under the surprisingly unassuming name ‘Jerry and David’s Guide to the Web’. Procrastinating in the fourth year of their PhD in 1993, Jerry Yang and David Filo focused their attention on fantasy league baseball. Yang created a crawler program that would seek out any available data on baseball players from FTP and Gopher resources. Upon the release of Mosaic, Yang spotted an opportunity for further diversion from his PhD work and started to browse the new www compulsively. He established a list of the interesting site as he browsed. They put the growing list on the Web in 1994. Yahoo! had a friendly and easy approach totally at odds with standard business practice. For example it included links to its competitors’ sites to allow users to check for further results.
AltaVista was initiated within DEC which, though it had led the microcomputer revolution, had failed to adjust to the PC era. Although AltaVista was one of the most popular destinations on the Web in 1996, DEC was in its death throes. AltaVista, strangled of support, was repeatedly sold to a succession on new owners. Ironically AltaVista’s founder, Louis Monier, was an alumnus of Xerox PARC, the scene of the most egregious cases of an early technology innovator neglecting next generation development from within its own ranks. Another entrant, Lycos, was developed at the Carnegie Mellon University and supported by ARPA. It pioneered the examination of links between sites to gauge thir seirch relevance. Excite, like Yahoo!, was started by graduate students at Stanford. It pioneered free e-mail and personalisation, allowing users to filter the content they wanted to see according to their personal preferences. Web searching was blossoming into a vital business. From the late 1990s it would come of age in the form of Google.
An unassuming Sergey Brin wrote on his PhD profile page at Stanford in 1999:
Research on the web seems to be fashionable these days and I guess I am no exception. Recently I have been working on the Google search with Larry Page.
The previous year he and his colleague Larry Page presented a paper announcing their new search engine, Google. They argued in much the same vein as Vannevar Bush had half a century previously that ‘the number of documents in the indices has been increasing by many orders of magnitude, but the user’s ability to look at documents has not’. The duo had begun to work together in 1996 on an unexplored element in the relationship between websites: back links. Though each website contains code outlining its outgoing links to other side, it was not possible to know what other sites had incoming links to other sites, it was not possible to know what other sites had incoming links to back to it. By developing ‘BackRub’, a system to determine which sites were most cited by other sites. The most cited sites would be given higher ‘page ranks’, appearing first in the search results and their backrub would be more important to other sites.
Google works because it relies on the millions of individuals posting websites to determine which other sites offer content of value. Instead of relying on a group of editors or solely on the frequency with which certain terms appear, Google ranks every web page using a breakthrough techniques called PageRank. PageRank evaluates all of the sites linking to a web page and assigns them a value, based in part on the sites linking to them. By analysing the full structure of the web, Google is able to determine which sites have been ‘voted’ the best sources of information by those most interested in the information they offer.
The system not only judged the relevance of websites to search queries on the basis of their own text but also on the basis of third-party descriptions found beside the outgoing links of other sites.
The backrub system proved to be enormously effective, returning more relevant results to search queries than any of its competitors. Thus when Yahoo! offered to buy Google for $3 billion, Google rejected the offer. Indeed such was Google’s popularity that in June 2006 the Oxford English Dictionary added the word ‘google’ as a verb. The name Google, which Page and Brin chose in 1997, was inspired by the mathematical word ‘googol’, which denotes a massive number: one with a hundred zeros. The enormity of the googol, according to Google, ‘reflects their mission on organise a seemingly infinite amount of information on the Web’.
Google has a long history of famous algorithm updates, search index changes and refreshes.
Below are links to some of the most important resources for search marketers: