Summary of the Google Penalization story
On Thursday August 9th all of the traffic coming to MasterNewMedia from Google had suddenly disappeared. From over 14,000 visitors to 4,000 and less is a pretty big change from one day to the next. All the worst when you (and a bunch of others) make a living on it, as slashing your daily revenue by three is certainly going to cause you some trouble.
While I have had apparently similar problems in the past, mostly due to server issues or to errors in the traffic reporting tools we used, after having done all the checks and verifications I knew this time it was something different.
I could tell because my webmaster Drazen had noticed that while traffic from search engine Google had dropped drastically from one hour to the next, visitors coming from their bookmarks or by simply typing the name of this site had remained constant.
Initially I thought that the site was possibly victim of some Google algorhitmic change or of one of the now-famous Google dances. And since if that was the case there is not really much one can do I took the following weekend as an opportunity to rest and wait for Google to do its things.
Later I realized that this decision was a mistake. Sitting back and waiting, when you are loosing hundreds of visitors per hour and most of your daily AdSense revenue is certainly not the best strategy. Sharing the info, asking the experts, researching other sites having had similar stories are the things to do right away.
The following Monday traffic and revenues were as bad as on the previous Friday. I was desperate and clueless about what was best to do. Never before, an outage of traffic so significant had lasted so long.
I decided then to come out of the cave and to review again all possible causes while making immediately the story public to possibly gather some attention and feedback from those who were in the same situation or that experienced similar adventures before.
I thought I initially had found the culprit inside an error reported by Google Webmaster Tools and due to a badly formed URL on Master New Media. Bad having fixed it nothing still changed.
What I had not paid attention to was a notice inside GWT that clearly said that while the errors it had reported for Master New Media did limit Google ability to fully index our site, these errors would have not impacted our Google search results or rank. In fact nonetheless visitors were still a small fraction of what we normally see, all of Master New Media pages were properly indexed by Google and the official PageRank was also firmly set at 7.
The thing I stupidly had completely missed over, showing how much I give for granted what granted is not, is that yes Master New Media was still fully indexed and page-ranked, but if you searched for any of the hundreds of keywords for which we ranked in the first page of Google search results, our pages were nowhere to be seen anymore.
That, supposedly meant, that since Google was still fully indexing the site, it had decided to penalize its rankings for some unknown reason.
The Google apparent penalization lasted for a full week slashing traffic and revenues to an historical low. The negative impact of the missed revenue was so strong that it basically forced me to give up at least two full-salary contributors and to seriously evaluate possible alternative strategies to loosen my too exclusive dependence on Google traffic and revenues.
Action-wise I chose to immediately get rid of all possible content components, and coding issues that may have possibly caused the apparent heavy penalization we were suffering from. These included a number of things that we had never considered as possible breaches of Google guidelines.
After 9 days, on Thursday August 18th Google traffic started trickling back. On the following Monday, August 22nd everything was happily back to normal.
During these nine terrible days, I wrote publicly about these Google-related issues and my discoveries, and published an open-letter to Matt Cutts asking basically Google for greater transparency about penalizations and issues causing them.
While I got lots of useful feedback from many of my readers, I had no response whatsoever from Google or Google officials. Not even through the dedicated webmaster messaging facility Google has recently introduced inside the GWT dashboard for the purpose of communicating with webmasters of sites that have problems or should be penalized.
What changes I did during the penalization week
As originally reported in my early report about the supposed Google penalization, these were the items that we identified and did something about as we believed that one or more of them were in breach of the Google guidelines. These included:
1) Hidden text
I had two hidden items on my site but had never considered them as possible penalization risks due to their non-SEO nature and scope. One was a unique four-digit hidden code number present in every single page of the site that we utilized for logistical reasons to reference individual articles inside our content management system.
The other was an H1 title present on the home pages of the four international versions of Master New Media I publish and which included the tagline of the site (Independent publishing news....). This was a leftover of a distant past where we had layout problems displaying the tagline as we needed on the home page and we then opted for hiding it as to bypass the issue.
Again neither one of these hidden texts had been placed on the pages to obtain better search engine ranking or to trick Google in giving us greater relevance than we deserved.
2) Duplicate pages - redirect via permanent 301
At the time this happened, we had multiple domains mapped to the contents of Master New Media including RobinGood.com, MasterNewMedia.com. This again was not done with any smart intention to trick the search engines but probably more out of inertia and superficiality than anything else.
From what I understand of the issue and have seen, having such other domains mapped to MasterNewMedia caused Google to index parts of our content as coming from ww.robingood.com/... and some other as coming from www.masternewmedia.org/... . This in turn may create the appearance of having tons of duplicate content and Google certainly does not like that too much (though I wonder how Google manages today the infinite replication of content that takes place because of RSS feed distribution and syndication - is that "bad" duplicate content too? How do you tell the difference?
So the proper way to manage and to be fully compliant with Google guidelines is to utilize a permanent "301 redirect". And this we did.
3) Spam comments
Italian SEO expert Giorgio Tave took up the issue in his forums and pointed to spam comments sitting on Master New Media articles as another likely cause of our apparent penalization.
I had indeed several "sex-related" spam comments appearing on some old articles that had somehow escaped our spam-filtering and approval system. These articles all dated back to 2005 or earlier and as such had been sitting there with those bad links for quite a long time. These spam comments were all immediately identified and cleaned up.
4) Text Links
On Master New Media we had been experimenting, in particular in the last 12 months, with a number of monetization alternatives to gradually increase alternative revenue streams to Google AdSense. Among these, text links were one such alternative. Text links are text-based ads that were "invented" to create a marketplace for buying and selling "authority" as Google measures the relevance of a site by the number of relevant incoming links. This is why, especially in very competitive markets companies choose to buy text links from higher ranked sites. The links coming from those paid for links increases their authority and credibility in Google's and other major search engine eyes.
I dropped altogether all text links, leaving only those that were in full respect of Google guidelines, by being clearly labeled as such and having no ability to pass PageRank to other sites. These included many Text Link Ads which provided possibly the largest alternative income stream to AdSense.
5) Issues reported in the Google Webmaster Tools dashboard
Right after the problems had started, I had realized that Master New Media had problems with Google indexing as reported by Google Webmaster Tools and by the Sitemap facility. Nonetheless some readers were drawn into believing from my own report that I had not been using Sitemaps until then, my webmaster had been using it on this site for quite a while, though, and here was the problem, he hadn't set up an automatic daily checkup of the same.
My specific problem was a badly formed URL in an article, and it was very easy to resolve.
In any case I have learned now that it is an absolute must to do a systematic daily check of the Sitemap and GWT reports and to be always on the lookout for possible technical issues preventing the site from being fully indexed.
Interesting also the comment that Derek provided on this topic:
"Google killed most of my traffic last month and I was stunned.
A quick check of Google's Webmaster tools showed DNS Lookup Timeouts across the board. I later learned that the firewall on my server had been updated, and for some reason, it saw Google's spider as something nasty, so it wouldn't allow it to index any of my sites on that server.
Content that was previously indexed was removed from Google.
After some tweaks to the firewall, Google's spider began its friendly visits again, and my pages are back. I don't know if this is personally helpful to you, but it may be to others if not."
Initial suggestions from readers
Just like Derek, many of you, took active part in this whole story and provided comments, suggestions, tips and criticism that was very useful in the end. Here are some of the most interesting and relevant comments received on the subject of Google penalization.
Steven Cox provided a specific recipe including:
"First, correct the issues that Google does not agree with.
Second, go to your webmaster tools and do a 'reinclusion request'. This can be found on the main page of your GWT.
Third, you have to admit you did things wrong before you can complete your request (I find this odd that you HAVE to admit guilt - almost like going to confession). Things could change in as little as 1 day to whenever.
Also, try to find any duplicate content on your site and just get rid of it. If you think it might be duplicate, just dump it.
Ok, that's what I've learned over time. It's not fun for the little guy - especially when you don't know what you're doing wrong."
Re the hidden text issue Daniel commented specifically:
"if you need to hide IDs in a page, there is a way to do it which should not cause any problems with google : use an input of type "hidden" (yes, it is a standard type of input that can be used to store variables in pages)."
Jim West wrote an email saying: "Same thing has happened to one of my sites (www.inkjet-direct.com) - I have 3 main income generating ones and 3 others less important. This site does not produce most of our traffic but nevertheless it helps and I want it back on asap. Like you I wracked my brains to figure out what I did wrong.
Previously, same thing happened to my main site www.proprint.co.uk - at that time Google emailed and said what I was doing wrong - hidden text - I had never read the guidelines and did not have a Google account. We fixed that site and they re-indexed it 30days (in Gaol) later.
When I examined the site that is now off - I did find a few pages with similar hidden text and have completely removed it or made it visible on the page. Here is hoping that this is the problem. If it is not I do not know where to look next - except perhaps to start from scratch with a very basic site."
What I think were the most likely causes
To be fully honest, to this day I have no certainty yet of what really caused this traffic disaster. To me it remains highly possible that had we done nothing to correct items that were not perfectly matching Google guidelines things may have gone back to how they were anyhow.
My inclination to believe this is due to the fact that if I assume Google not to be using "scare tactics" or sudden and unmotivated penalizations to intimidate small publishers (from deviating or even ignoring technical and editorial issues that may be damaging Google core search business), I would have been notified in one way or another of why this was happening and what was not OK with my site.
My story was publicly accessible, my name and emails are in Google hands, I am also a Premium AdSense publisher and therefore Google knows quite well where to find me. Google could have used the direct messaging tool inside GWT or just send me a standard email notification... but it didn't.
Even after my public exposure of the full story and its evolution, after an open letter to Matt Cutts expressing doubts and asking clarifications about the proper use of text links, I received no official sign or indication from any Google source of why I had been "penalized" and which was the cause for this.
So, if Google didn't contact me should I assume that either this was a very unfortunate accident, though it is hard to believe that a Google engineer flipped a penalization switch inadvertently, or that there was a glitch in the Google system that suddenly penalized only my site?
If I stop dreaming these fantasies for a moment, and I look more objectively at all the data and opinions collected so far, I could also consider that Google now spiders content of web sites and penalizes them automatically if triggered by certain components. One could think that since Google could not at this time attend to communicate directly with all infringers, it proactively penalizes those who trigger its checking system and then goes about verifying whether the site is actually breaching some guidelines or not. This may have the advantage of putting many webmasters on their toes as the story I recounted here is re-lived many hundred times across the web. The net effect is similar to the war on terror. There is greater awareness of the issue, and out of fear, webmasters adopt more conservative and Google-compliant publishing and SEO methods.
In this light and from this perspective, the factually innocuous hidden text code I had on every page could very well represent the item that triggered the Google filter and placed the site under temporary penalization.
But if this was true it is likely that this is something completely new that Google is doing, as those invisible number have been sitting there on my web pages for over three years now.
In the end, when asked what I think it was the most likely cause I still do not have a clear answer to provide. The sensation is that it is none of the things we blamed ourselves for, but I have yet no definite way to prove it.
Also consider this which I wrote in a previous article on the topic:
"...it is likely that Google keeps changing and refining its own approach and policies to improve the quality of its search results and in this effort, it appears that it may have become more restrictive and intolerant of issues and deviations from the official standards that it didn't bother about before. I say this because Master New Media has never done any active tweaking of its web pages to rank higher on Google search results nor has tried to trick Google or any other search engine to better exploit ranking positions and visibility. Our strengths on this front were all organic and due to lots of good in-depth writing work."
Text links - drop them or not?
I have already written a couple of articles specifically looking into the issue of text links. This is a difficult topic and nonetheless I had strong opinions myself about it, after you look deeper into it, it is not as clear cut and easy to judge upon easily.
My original position, as I wrote, was of perceived guilt for having used text links, as most of them are tacitly intended to pass on Google "value" (PageRank) to other sites that simply pay for having them displayed on higher ranking sites. This is evidently a way to "cheat" the Google ranking system which is heavily based on linking value.
On the other hand, by looking and listening at several articles both from the Google side as well as from some of its key critics, I realized that the distance between a text link ad and a traditional ad on a web page is not so big. In fact who is to tell which is which?
So, if you have not yet seen the presentation and video clips I showcased in my previous article on Google FUD I strongly suggest you do so, making sure you also watch closely this very, very interesting video interview with Michael Gray and Randy Fishkin recorded by Web Pro News at the recent Search Engine Strategies Conference in San Jose.
To place this video on your web site, copy/paste
the following code into your HTML file:
Also some of you contributed some very interesting comments.
Stefano wrote: "... about text links, as far as i know, your risk is those links don't pass pagerank anymore, but i never heard about such a penalization for selling links (of course, it means that it may be happened; and it also means you may be the first one). ..."
Cristoph Cemper wrote: "I would recommend to NEVER again combine 2 or more link networks to maximize your profits."
Me Me said:
"As little as 6-9 months ago, there was very little information on the validity of buying text links. Last December, we bought links on a guitar site because it was directly related to us. Next thing we know, all of our traffic stopped from Google. They didn't de-index us. They put is in what is know as a -950 penalty. Which means your rankings drop by approximately 950 places.
It took several weeks to figure out what the issue was. We have over 7,000 small instructors that count on us to help them find new students. Most of these people don't have web sites and definitely do not SEO them. During this time, all of these people suffered by a dramatic drop in revenue. It didn't just hurt us, it hurt real people in the real world who depend on us to pay the rent.
I had no moral issues with it because we were paying for relevant advertising on a relevant site. It's no different than suggesting that buying a billboard along a busy interstate is 'evil' because the billboard gets more traffic than your side street. ..."
Alex Choo of Wptextads, a company that provides a WordPress plugin that lets you sell text link ads directly to advertisers wrote: "...The rel=nofollow attribute is so important. Ethical link selling is fine, I think Google's ok with it too. What is wrong are those sites that sell links to drive up PR. Paid links ought to be clearly labeled as that."
Maybe you can appreciate better now how difficult it is to make a decision on text links that takes into account all of this factors.
At this stage my stance is to do away with all "spammy" links, meaning all those text links, that while having nothing to do with the content and topics I cover on my site tend to hide and make themselves as little visible as possible in order not to be detected by major search engines. Ads, whether in text format or not, need to be highly relevant to my content topics and must be editorially approved one by one. Under these circumstances, which reflect very much our existing policies for standard advertisers, text links could exist. They would be all clearly labeled, and made very visible to the readers. Whether these links will pass juice or not to other sites it is at this point a disputable variable to discuss.
Google appears to be in the best condition possible to evaluate circumstances in which links or sites should not pass value to others and it will act on them directly and without provoking any additional harm.
I am therefore re-evaluating all of my link partnerships and starting to test some alternative solutions.
Google policies and communication system
Among all issues, Google inability to promptly and transparently communicate penalization issues or other forms of penalty inflicted on web sites remains the one that creates the most negative consequences among the small independent publishers like me.
Here again, some of your contributions were rather illuminating, or in many cases comforting my very same initial negative perceptions on the issue:
Bizniz: "Hey Robin I too experienced a very similar issue with Google on one of my smaller web sites. Google was not helpful and totally indifferent to my numerous requests for assistance and information about how or where my website breached their policies."
Daniel McBrearty wrote: "... it would help a *lot* if there was some crystal clear list of *what* had upset them..."
Cristoph Cemper also summed it up pretty well:
"What bothers me the most is that the small guy is boxed in to using Google now. Everyone uses Google search - and if you don't play by their rules, you're screwed.
However, they don't tell you what the rules are.
Further, within the webmaster tools, it's now just one click to report a competitors site for selling links or spamming. Your competitors are incentivized to scour their competitors sites and report it to google - even if the information is incorrect! Is there a penalty to the reporting site if they report incorrect data to google? Are they penalized? Or does google just automatically and blindly believe them? Very scary.
Google has become a virtual utility (similar to water, or electricity) vs. an internet company. Thus, they have a larger responsibility to the population.
Imagine if the electric company cut off your power because of something you did, but they don't tell you what you're doing wrong. Further, they tell you that if you don't like it, stop using them and install solar panels on your house.
That's the way I feel with google. You cannot stop using them, but you don't know what they are going to throw at you next."
Antone Roundy provides a good and objective counterpoint to this: "...Google is not the government, but it's not a public utility either. It's a business that provides an extremely valuable service for free, which makes a lot of money for a lot of other businesses."
Though he convenes that:
"...Google would do well to better balance the need to avoid giving too much information to unethical individuals with the need to provide more information to ethical individuals who sometimes get penalized as a side effect of algorithm changes aimed at unethical individuals.
Perhaps when a site is penalized, details about why it was penalized could be hidden by default, but if the webmaster requests that information, a Google employee could take a quick look at their website, and if it looks legitimate, grant access to the details. That process should be easy to streamline so that it takes a low-pay employee less than a minute to process each request for access."
Finally, If I have to take this Google official announcement story from the end of last week as a reliable reference then my Google penalization story could have just really been a temporary glitch inside the major search engine.
"Update on penalty notifications
Wednesday, August 29, 2007 at 1:40 PM
Written by Ben D'Angelo, Search Quality Team
First, a brief recap: In late 2005, we started emailing webmasters to let them know that their site is violating our Webmaster Guidelines and that we have temporarily removed some of their pages from our index. A few months ago we put these emails on hold due to a number of spoofed messages being sent from outside Google, primarily to German webmasters. Then, in mid-July, we launched Message Center in our webmaster console, which allows us to send messages to verified site owners.
While Message Center is great for verified site owners, it doesn't allow us to notify webmasters who aren't registered in Google's Webmaster Tools. For this reason, we plan to resume sending emails in addition to the Message Center notifications. ..."
Source: Google Webmaster Central
I am a verified website owner, have the Message Center feature available inside my Google Webmaster Tools dashboard but I have never received any notification from Google. Does this mean that the supposed penalization Master New Media received a couple of weeks ago was just a glitch in the major search engine?
Morale and conclusions
"job" commented: "therefore, as an independant web publisher, I should take this as a lesson in putting all my eggs into one basket, and I should be proactively seeking new ways to supplement this lost google traffic, coz even though I might get relisted, I never know when this might happen again!"
I hope you dont take this as derogatory remarks, I didnt notice even a hint of you looking for alternative traffic, seems like its google or noogle ??"
Marco Ermini goes even heavier on it by writing: "I hope you'll have a thought not just about the technical issues you have experienced, but also about the poorness of a business model which is 100 percent based on another company - which whom, by the way, you have no agreement at all. They can trash you and all of your activity in 5 minutes with no hassle and no justifications.
It is a poor and not excusable management strategy in itself - that should be in the centre of your de-briefing session..."
Well.... I can't say these guys are not saying the truth. I do depend too much from Google search traffic which in turn directly feeds my Google AdSense revenue stream. So, if you stop Google traffic from reaching my site you also cut off the main and most effective source of advertising revenue I have.
It is necessary to cultivate and grow many other alternatives as there is certainly no lack of them.
Getting traffic from other search engines is a good thing. Making those other search engines carry a greater fraction of the whole web search traffic is the true matter and until Google has such a great share of it, it is very hard in practice not to depend on it. Not impossible but very challenging and time consuming.
The alternative monetization opportunities I will work on in the future include video advertising, merchandising, eBay and the new Amazon monetization opportunities as well as increased used of sponsorship and traditional ads.
To do all this it is also clear that a small publisher like me increasingly needs at least one person entirely devoted to the monetization strategy and to the advertising management needs that a small network of sites like this requires to stay afloat.
I hope the sharing of all this info can help others make better choices and avoid falling in the same traps I did.
Your further contributions and comments to this topic are also therefore very welcome.
Recent related resources:
Originally written by Robin Good for Master New Media and entitled: "Google Penalization: Dropping Out Of Google Search Results And Trying To Understand Its Causes"