Making Money from the Web:
Has Google Lost the Plot?
Part 1
As a web developer and webmaster of the www.celtnet.org.uk website I have a vested interest in the way that search engines work and operate. A recent article in the Economist Newspaper indicates that the market share of the various search engines is as follows:
Google: 50%
Yahoo!: 28%
MSN: 13%
Ask: 6%
Google has more market penetration that this, however, as it tends to be used by more 'web savvy' users who are far more likely to buy products on-line. Indeed, the popularity of Google grew out of this community because Google at the time was seen as 'best in classs'. The results returned by this search engine were better than those of its rivals both in terms of the accuracy of the results delivered and the total number of sites and pages indexed. With its search algorithm and its page ranking facility Google delivered both the most and the best results. Even after its stock market floatation Google was basking in the rosy glow of how it was perceived by its user base. But does this view of Google still hold true?
Since its stockmarket debut Google (in common with many other large companies cf Microsoft) has been adding new and often quite different products to 'supplement' its core search engine business. Some have criticized this as an example of a large company straying from its core business. However, Brin and Page, Google's founders, have calculated that Google's engineers should spend apout 70% of their time on core products, 20% on relevant but tangential products and 10% of their time on the wildly fun things that might (but probably won't) lead to a product.
As a result a large number of tiny teams within Google have been working on all kinds of different products. However, many webmasters see this as Google not caring about their core business any more. In fact this is a view that many in the investment sphere now share and Google's share price has been rather volatile of late. Indeed, the view is held that at floatation Google was a fairly simple story: in effect they were a marketing company deploying online ads on top of the most popular search engine on the internet. Now, however, Google looks rather like a mess and comparisons are being made with Microsoft during the mid 1990s where they had a single core strength (Windows OS) and were churning-out a string of rather poor me-to applications.
For example Google Video has been trumped by YouTube and Google News lags far behind Yahoo! News. Google's instant messaging offering lags far behind AOL's Yahoo!'s and MSN's offerings. Even Google's slogan of 'Don't be evil' is looking faintly ridiculous as they use their status as a 500lb gorilla to stifle the growth of web-based developers and entrepreneurs.
Over the past few years Google has almost certainly offered the best search engine experience on the internet. This is significantly due to Google's search and indexing algorithm. The problem now is that Google actively regard themselves as the best and this has coloured their attitudes at all levels and represents one of the reason that they don't seem to have a clear strategy for the future and are currently dabbling aimlessly at a variety of non-core projects. This over-confidence has also led Google to have what might be viewed as an almost-religious faith in their mathematical algorithms which means that there is almost no human intervention as to what gets indexed and why. For Google the important thing to know about a website is how many incoming links are pointing to it and a website's overall rank (its Page Rank) is directly linked to the number of incoming links. This is great for old websites on popular topic, but if you're a new website in a specialist niche your indexing and rankings will suffer as a result.
This is why Ask (formerly Ask Jeeves) is employing a different track. They also use incoming links to rank web pages but they also cluster sites and pages by theme. Thus rank isn't absolute as it's measured with respect to a site's or a page's comparison to its peers — an approach that often returns better results than Google's. A cursory comparison certainly suggests that Ask's general web search is at least as good as Google's and it's searches for maps and images are currently superior. As a result Ask is gaining a small but dedicated following.
The potentially really big player in this arena, however, is Microsoft who are weighing-in with their MSN search engine. This is not as good for searching as Google, but has indexed more of the web (for example Google has indexed 86 pages from my site but MSN has 1100!). As Microsoft tweaks MSN's search abilities this product may well offer a very strong challenge to Google. After all, Microsoft has a tradition of entering a marketplace rather late with a fairly mediocre 'me too' product that gradually improves until it prevails. Moreover, MSN will be including an on-screen search box in its next operating system Vista that will be pre-configured to search MSN as a default.
Part 2
Google (as well as all the other search engines) are in a very peculiar position as the're delivering value on top of other peoples' content. The value being the delivery of that content to the user in a way that's useful to the user (which usually means the ranking of the results returned). To remain in top position the search engines must convince their users that they're delivering content that is 'the best of the web'.
Admittedly, Google comes for a fair amount of stick in articles such as this one; mostly this is because Google is the dominant player in the web indexing business and because of this a small change in the way that Google indexes or caches websites can make a web-based company succeed or fail. Admittedly, Google, along with all the other search engines, have a problem. The web is expanding month on month and though some of the new sites are providing good and useful content many of the others are spam sites, whose sole function is to promote other peoples' products. Others are 'scraper' sites, stealing content and presenting it as their own. Yet others are 'clone' sites. Sites that have Amazon or eBay marketing areas that are duplicates of other sites and use the same code and/or pages as them. There are also those websites that take pre-existing articles from article websites and publish these as new content.
Including multiple copies of such websites would both degrade the results of any search engine and would simply swamp any good information-rich website in a sea of junk. Over the past eight months Google have been busy tweaking their algorithms and delivering a system called Bigdaddy. What few messages that have been coming out of Google have been very 'up' about this, but the news from webmasters and a number of recent disasters indicate that Google may well have dropped the ball over these updates.
Part of the problem with all this is the difficulty in actually getting any information out of Google. All their systems are automated and there are pages where you can send questions to Google, but all you ever get back from these is a standard automated response. Google can also black-list websites because of content or practices that Google deem to be bad. But they don't tell you that this has happened. The only way to find out if you have one of these Google penalties is to use a form to tell Google that you think you've had a penalty, that you've corrected the problem and that you'll never, ever, do it again. There's an implicit assumption of guilt and that all webmasters are somehow 'bad'. The only other indication of a Google penalty is that the number of pages indexed in a site (as well as the site's overall rankings) drop dramatically.
Google also change their algorithms and indexing practices at will without making much of a general announcement about it. And it's this implicit environment of secrecy that annoys many webmasters. Especially as any updates to Google's indexing strategies may make a site fall in terms of rankings or may dramatically change the number of pages from a site that are indexed. A change in Google's algorithms may have the same effect as gaining a Google penalty, which obviously dramatically increases the stress levels for may web authors.
Despite their slogan of ''Don't be evil' Google come across as being a cold and uncaring company who have an almost religious belief in their indexing algorithm. Tweaking this algorithm is what they do and it sometimes appears to outsiders as if Google firmly believes that their algorithm can never be wrong.
I can understand why Google have done this. If they didn't then they'd be inundated by umpteen million e-mails every day. However, because of this they've painted themselves as being cold and aloof: uncaring of the needs of the very people who create the content from which they make their money.
The current upheaval started about seven months ago when Google began to roll-out their new Bigdaddy datacentres. These were, according to Matt Cutts 'the human voice of Google' were new data centre infrastructures rather than a major algorithm change in indexing. Bigdaddy is, in effect, a brand-new data center that Google uses to perform core search engine tasks like cataloging Web sites or serving up localized features. Google operates thousands of data centers across the globe.
However, as the Bigdaddy datacentres were rolled out during January and February many website designers and authors reported very dramatic reductions in the number of pages from their sites that were indexed. What seemed to have happened was that as Google rolled out some new technology to fix a few things this resulted in their breaking other things, But it wasn't until May 10th that Google's co-founder, Larry Page, addressed this issue saying the problems took the company by surprise and it was now investigating what is going on. "We have a team studying it now," he told an audience of journalists at Google's annual Press Day event. It looked as if in upgrading to Bigdaddy Google was erasing its existing data caches and replacing them completely with new indexes. During this process many web pages that were previously indexed simply disappeared from Google.
At about the same time Google CEO Eric Schmodt told the New York Times about a 'server crisis' where the machines holding their indexes were completely clogged.
Another problem was that real web pages were being replaced with 'supplemental results' often from six months or so previously. These supplemental results are pretty useless in terms of searching though they do artificially inflate the number of pages indexed from a given site.
Then on May 16th Matt Cutts offered a very long post on the timeline of Google's new indexing and Bigdaddy roll-out. This said that many of the supplemental results would be refreshed. Then came the scary stuff: If a site didn't have enough incoming links then it might be considered as having 'low confidence' and wouldn't get crawled as deeply. Even worse, depending on the number of links to your site Google may be crawling your site every day to see if there's interesting content but none of that content may be indexed (this is happening to me now). So Google is basically stealing my bandwidth without giving me anything for it.
Maybe Google doesn't want the good content of the web indexed. After all, if you're not getting any visitors for the search engines you're going to have to resort to advertising and one of the big sources of advertising out there is Google's AdWords program. The truth is that Google is no longer reflecting the internet as it is, but they're reflecting the Internet as Google thinks it should be. Moreover, other search engines are catching-up with Google and are indexing more of the web than Google are.
In their recent update it seems as if Google re only really counting the number of uni-directional (ie non-reciprocal) links coming into your site from related sites as a measure of a site's worth. However, if you do a search for your site's URL on Google and then perform a links search with the links:www.mysite.com you'll see that the links: search only returns a third of the total number of links coming into your site. So Google may well be under-estimating the number of sites linking to you and so your ranking and the number of pages indexed in your site will suffer as a result. Also if you're in a specialized niche then you may not have many inbound links, but those you do come from natural links built in your area of interest. However, as you have few incoming links most of your pages won't be indexed (as will be the case for many of the sites that link to you) so even if you have good links the non-indexing of these linking pages mean that both you and the sites linking to you aren't given the credit that you should.
The upshot of this is that a perfectly good, clean, website cannot now get all or its pages indexed if it doesn't have a sufficiently good score in terms of both in-bound and out-bound links. Too few inbound links and the site won't be fully indexed. As a result Google is effectively editorializing a site's content because Google doesn't think it's 'good enough'. In reality Google's users are being deprived of many good resources because of this. Users are only being exposed to a portion of an internet — a portion that Google defines for them. Does this really mean that Google is short of indexing space and that they can only keep going by 'pruning' the web? It's certainly looking more and more as if this is the case.
Added to this Google have had other debacles of late. One of the most notable were the reports in a number of SEO fora that Google had suddenly and very mysteriously added a few billion pages to their index. This might be a cause for celebration: had Google finally relented and indexed the pages that they'd dropped or excluded from their index? Analysis of my own site indicated no for Google still only indexed 85 pages (MSN was indexing 1100+) [as an aside, since I wrote this Google have added a further 40 pages to my site's index]. Rather, all those new pages were blatant spam and contained only pay-per-click (PPC) advertising as well as scraped conttent. Even worse, many of these pages were displaying well in the search indexes. A Google representative responded via forums to the issue by calling it a "bad data push," something that met with various groans throughout the SEO community.
What seems to have happened is that a Moldovan had the enterprising idea of using the way that Google handles subdomains (in reality these are sub-components of a website but Google handles them as if they were totally independent sites). This allowed he enterprising individual to craft scripts that would generate an endless number of subdomains as soon as the googlebot dropped by his site. As Google indexed these subdomians on an essentially 'no questions asked' basis he was able to deliver single pages containing keyword-rich scraped content, keyworded links, and PPC ads for those keywords. Spambots were then sent out to put GoogleBot on the scent via referral and comment spam to tens of thousands of blogs around the world. The spambots provided the broad setup, and it didn't take much to get the dominos to fall.
As a result this individual found himself with 5+ million indexed pages on Google. Word of this spread like wildfire in the SEO communities, though the general public seems essentially ignorant of Google's latest snafu. Google responded saying that only 'millions' of pages had been indexed and that they were working on tweaking their algorithms to remove the possibility of this happening again. Though close examination strongly indicates that the domains were actually being removed manually from Google's index.
What stuck in many webmasters' craw is not the fact that Google were caught out, but the fact that this scamster managed to get millions of pages into Google whilst Google's managing to ignore many thousands of websites with perfectly good content and is not including them in its index.
Then came the story at the end of June that Google had dropped 'amazon.com' from its entire search index (and its not the only big player to go missing from Google). If Google can't do the big players, then what hope have we, the minnows in the Internet pool?
Will I be sticking with Google? The answer is that for the moment I will. Google is generating some of my best income due to the AdSense program. However, bizarrely, most of the traffic that creates my AdSense income comes from non-Google search engines!
|
If you enjoyed this page and would like to get more tips, tricks and offers to help you make the most of your most of your web presence please sign up for my Weekly e-mail newsletter. Please note that your details will never be sold and shared with others. You are signing-u for my e-mail only. |
