August 15, 2007
Yes, I sang victory way too early. Nonetheless I had hoped to have come out of the terrible traffic loss that started last Friday, I instead discovered that Google has fully penalized Master New Media and dropped all of my articles out of Google search results.
Photo credit: Andres Rodriguez and Robin Good
I don't know if you have read my article of this past Monday, where I recounted the stress and apparent resolution of a major problem affecting the flow of web visitors to this very site. Starting last Friday in fact, all of the stats reporting to me the number of visitors to Master New Media crumbled to 1/4 or more of their normal level. Visitors went from about 15,000/day to 3-4,000/day. Even worst was the deeper slash in daily revenues which saw an analogous drop.
As I wrote in that article, this can prove to be one of the most difficult situations that an independent publisher may find itself in. Not only, but the bigger you have become, especially in terms of collaborators and staff, without the help of external financiers or other partners, the larger and more difficult to manage the consequent problems.
Fact is, that while on Monday I posted an apparently victorious article explaining the trouble I had run into and what I thought at the time to have been its cause and resolution, reality has settled in again with quite a different face.
The new discoveries I made afterwards with Drazen, my old-time trusted and untiring webmaster, left me really without words, as our whole micro publishing enterprise had fallen into a situation we had never planned for.
For some reason, yet unknown to us, Google has actually penalized all of my web pages, to the point that they could not be found anymore for any of the terms for which they were so highly ranked until a few days ago. This is not only disaster from a standard web publisher viewpoint, but it is triple tragedy because the very visitors coming from Google, and not out of their own bookmarks or of a friend recommending them, are the very ones that make my site sustainable.
Master New Media is based on a contextual advertising business model, in which text-based ads carrying advertising info for products and services that relate to the very content appearing in our articles. The moment you eliminate the majority of the visitors coming from search engines, you are wiping out your greatest potential to monetize your content via advertising. Read on and I will explain you why.
By looking at this often not immediately visible but very critical aspects of an independent publisher life, you can learn a lot more about the obstacles and challenges awaiting you ahead on this path.
This is why, like many other times before, I have chosen to fully share my experiences, errors and lost battles. To further help and support my very mission: helping the small guy grow smarter, bigger and more sustainable (that is = independent).
How? By being a new media explorer and an online experimenter ahead of those wanting to walk this same path, as well as by being an honest reporter who shares and reports his discoveries and errors all along the way.
This is why I do it.
Look at my traffic stats now:
Unique daily visitors for MasterNewMedia for August 2007 - stats by HitboxProfessional
Let me take you then where I left off this past Monday.
After three days of relatively zero traffic, from last Friday to Monday morning, and a million checks we did in every direction, I realized that we needed to look somewhere else. This is when I first remembered of my Google Webmaster Tools account and discovered that Google itself was reporting errors for my sites that I was not aware about.
The errors were in fact mostly relate with the format and syntax of the sitemap file one needs to periodically resubmit to Google, to get maximum indexing coverage from its crawlers.
I and Drazen thought we had found the culprit and that's when I wrote Sudden Traffic Loss? Google Doesn't Send You Anymore Visitors? Here Is What To Check, fully convinced that having cleared that things would have gone back to normal.
Was I wrong!
What I had not paid attention to was a notice inside GWT that clearly said that while the errors it had reported for Master New Media did limit Google ability to fully index our site, these errors would have not impacted our Google search results or rank. In fact nonetheless visitors were still a small fraction of what we normally see, all of Master New Media pages were properly indexed by Google and the official PageRank was also firmly set at 7.
The thing I stupidly had completely missed over, showing how much I give for granted what granted is not, is that yes Master New Media was still fully indexed and ranked, but if you searched for any of the hundreds of keywords for which we ranked in the first page of Google search results, our pages were nowhere to be seen anymore. You can try it yourself. Go to Google and search for "independent publishing" for which Master New Media has ranked first for a very long time now (several years). Master New Media is nowhere to be found in the first ten pages of results!
What does that mean?
Google, for some reason, has penalized Master New Media in the most significant way possible: it has dropped our pages from their search results.
Why?
Well, this is not something as easy to answer, but obviously I have spent most of my time yesterday trying to figure out this very issue and here it is what I have found out.
As you probably already know, Google will not come to you or send you an email saying "Hey Robin, look there are some things not Ok with your site, can you kindly fix them or we will have to ban you.". If their system senses and reports specific issues on a site, it looks like they just go ahead and penalize it. I don't know whether this is automatic or humanly-handled but fact is that you get no notifications or warning that this is about to happen.
Therefore the first and foremost lesson I have already learned from this is that, at least from now on, you and I need to be way more proactive in making sure that our sites are fully compliant with Google requirements and guidelines.
One other realization I made is that it is likely that Google keeps changing and refining its own approach and policies to improve the quality of its results and in this effort, it appears that it may have become more restrictive and intolerant of issues and deviations from the official standards that it didn't bother about before. I say this because Master New Media has never done any active tweaking of its web pages to rank higher on Google search results nor has tried to trick Google or any other search engine to better exploit ranking positions and visibility. Our strengths on this front were all organic and due to lots of good in-depth writing work.
But back to the main question: what could I have done to trigger Google into censoring all of my content pages?
We spent a lot of time looking at this, asking people and by reading comments and answers in the Google Webmaster Help forums.
In the end, the items we found that were clearly in contrast with Google guidelines were the following:
1) Hidden text - I had an H1 title on the home page of MasterNewMedia English edition as well as on the home pages of the other international editions (IT, ES, PT).
2) Hidden text - I had since the longest time a unique 5-digit numerical code hidden in every article page I publish. It is a kind of unique ID we have manually created for all of our article pages so that we can more easily reference them inside our content management system. We have had this for years, and it is obviously not something to trick anyone. But if Google monitors and penalizes pages via a program that does this automatically I doubt that it may be able to tell "good" from "bad" invisible text. So, the only solution here is not to try to explain to Google the why but rather to just avoid using any kind of solution that may jeopardize your compliance with their guidelines.
3) Non-compliant Text Links - I did have several text link ads appearing on many pages of Master New Media as well as on some of my other sites. These are those small text only ads often appearing at bottom of web pages or in other not very visible locations. (As I will write a whole article tomorrow about this very issue I reserve for that the going into more details about the ins and outs of these strange type of ads.) Google does not prohibit the use of "paid for" text link ads but it requires that you as a site publisher clearly label these ads as such and that a "no-follow" tag is assigned to them so that they cannot pass "Google juice" (read PageRank) to other sites. I was clearly not in compliance with these guidelines, as some did not display a clear detectable text label, nor they were all assigned a "no-follow" tag.
Without hesitation I decided to correct or take down all of these self-identified problems, no matter what the cost or loss in further revenue, and to immediately utilize Google official request form to be re-instated fully in its search results. While I have no guarantee that I did spot what Google didn't like and that they will re-instate all of my pages in their search results, this is my best effort to do so, and without further indications from Google itself, it is all I in fact can do.
There is also a lot more to say about text links, the universe that surrounds them, Google view and my own ethics, but as I have mentioned, I want to devote to a full article to this, as I think it is subject prey of lots of supposed-expert advice but for which I have had personally lots of difficulties in finding answers I could truly trust.
Obviously you should read the popular articles by Google's own Matt Cutts on this, as well as other relevant reports, but in my experience, they are not enough.
And while I don't know yet whether text links were the true case of my Google penalization, I have certainly learned lots about them and I think I have some very valuable insight to share with you about whether you should consider them your friend or foe.
Having cleared, corrected and dropped all of the above deviations from the Google official guidelines from Master New Media, I then went on to submitting a formal request to Google for having my pages fully re-instated in their search engine results.
As I have mentioned there is a dedicated form in your Google Webmaster Tools account that serves just this purpose.
In theory there is nothing else I can do right now. Just sit back and wait until a Google revisor sees my request, evaluates it and puts the site back in the search results.
According to what I have read in forums yesterday this can easily take several weeks if not months.
And this is where my heart crumbles. Without Google visitors, my revenues are 1/4 or less than our average monthly gross income. Without them I can't pay my office space, editors, writers, webmaster or server space, let alone my own monthly expenses and the taxes I am due at the end of the month.
It is a tough situation.
I don't know yet.
I consider still the situation in flux and will continue do so at least until the end of this week. I expect to discover new things as I go along and to hopefully learn from others interesting options on how to get out of this situation fast.
For now, I am doing my best to make sure that everything that could be possibly done to guarantee full Google compliance is done.
I have also already alerted most of my staff in a meeting yesterday, about how bad the situation is. Most of them have reacted with strong signals of support and desire to contribute even without being paid. "We have married your cause Robin," one said, "and we will not let you down now that you need us most.". That was an injection of love, thank you.
But evidently not everyone can be a martyr in this situation, and others, like me, will need to find a way to be able to pay their bills. So while I was greatly lifted by the positive moral support received I felt great respect for those who may need to take on other roads to compensate for this unexpected situation.
The next hours and the rest of the week are going to be devoted to brainstorm further the situation I am in, while evaluating all possible roads I have available.
If it was not for my smaller sister sites Kolabora and MasterViews, my final revenue at the end of August would be next to useless, but thanks to these two additional sites I publish I hope to be able to at least pay my rent and food until things do get better.
And obviously, this is so assuming these other sites also don't get banned or penalized by Google. I am not trying to be paranoid. Fact is that there is a number of posts and articles out there that suggests in fact, that if you are linked by sites that are banned or penalized then your site as well may be penalized itself. While the logic of this is all to be understood, I link first and foremost my own other sites! Therefore it goes without even thinking that all of my sites may risk being penalized if I don't do everything I can to put myself in safe waters.
That is why our next step is to take every single one of our sites and to systematically check and clear out any item that may further trigger Google sensitive probes.
Notwithstanding the apparent disaster I am in, and which I have no idea how long it is going to last, I have a natural drive to turn negative events into opportunities.
In this respect I am glad to have fallen prey, before others, to this situation, strongly hoping that my story and example could help many others avoid doing my same mistakes.
As my mission is one of helping small, online independent publishers find their own way, I am grateful, nonetheless the pain this carries, of the unique opportunity I have to share some of the true difficulties and risks that this new profession has in store.
If you have gone yourself through such a situation, and wish to share, even anonymously, your experience and lessons learned, please feel free to do so by either using the comments section here below or by writing to me directly at Robin dot Good at masternewmedia dot org.
I don't know whether Master New Media will survive this jolt, and if and how it will come out of it. But of one thing you can be certain: until someone comes here and locks my keyboard I am going to keep sharing what I learn and discover each single day.
Originally written by Robin Good for Master New Media and entitled "Penalized by Google: Master New Media Is Out Of Google Search Results"
| 2007-08-16 15:37:06 |
Yes, fair point Robin. Well, you can *sometimes* get to talk to them in other businesses. Ever dealt with big government departments or huge companies, as a small business? They can really screw you too.
But it would help a *lot* if there was some crystal clear list of *what* had upset them.
I guess the bigger the sword a company is wielding (and the G-sword is huge these days), the more delicate their touch needs to be. My guess is that they are also going through a learning curve.
| 2007-08-16 14:52:49 |
@Steven - I have already done all the things you have recommended - I have listed and reported the errors I thought were in breach of Google guidelines and I acknowledged myself to be guilty of them.
Problem is... I don't know if those were the issues Google penalized me for!
If they only would talk back to us. I am essentially their marketing and distribution agent, but still they treat me as a number.
@Daniel
Yes, maybe, but at least you can normally talk to them. Here I have to plead guilty without even knowing if what I did is wrong.
| 2007-08-16 14:51:08 |
A riveting account. Thx for sharing this honest indepth report with us. A good alternative to Text Link Ads is LinkExperts (www.linkexperts.com), a NYC based outfit that could help you. Give their CEO Seth a shout at seth@linkexperts.com.
(or call 212-213-6251. Buon Ferragosto.
| 2007-08-16 13:02:01 |
Robin,
I've been through similar experiences. Here's the scoop.
First, you are not removed from the index. You have been penalized.
A search in google for 'site:masternewmedia.org' shows you have over 17,000 pages indexed. When searching for "independent publishing", you now rank #504.
http://www.google.com/search?q=independent+publishing&num=20&hl=en&safe=off&start=500&sa=N
I know this doesn't help your traffic much, but you're still there.
Here's what we found was the best way to fix things.
First, correct the issues that Google does not agree with.
Second, go to your webmaster tools and do a 'reinclusion request'. This can be found on the main page of your WMT.
Third, you have to admit you did things wrong before you can complete your request (I find this odd that you HAVE to admit guilt - almost like going to confession). Things could change in as little as 1 day to whenever.
Also, try to find any duplicate content on your site and just get rid of it. If you think it might be duplicate, just dump it.
Ok, that's what I've learned over time. It's not fun for the little guy - especially when you don't know what you're doing wrong.
Keep the faith - your blog has been a tremendous help to me and I appreciate the work you do.
Sorry again for your pain. It sucks and is a troubling trend that seems to be developing.
Steven
| 2007-08-16 06:20:26 |
about the hidden text issues:
if you need to hide ids in a page, there is a way to do it which should not cause any problems with google : use an input of type "hidden" (yes, it is a standard type of input that can be used to store variables in pages).
what do you mean by "H1 in title tags"? that would not even be valid html, would it?
| 2007-08-16 05:58:25 |
Hi Robin,
first off, I am sorry for the pain you are obviously going through. I enjoy your blog. I do hope ityet turns out to be a techy problem, and nothing more sinister.
I have to say though, I am not completely in agreement with the "Big Brother" type comments. At first sight it looks like that, yes.
But in any business, if you were heavily dependent on a single supplier or client, that would be a worry - unless you had a special relationship with them.
IIRC, Google are about 50 of the global search engine providers. So there is another 50 out there that you could be using to spread the risk, no?
anyhow, good luck.
Daniel
| 2007-08-16 02:36:15 |
Thank you, thank you, thank you.
Your comments and feedback provide lots of moral and extra motivation not to give up.
@Hazel: I agree with you and it is indeed a topic that we must all face.
@Michael: Wow, that makes my heart smile. I am setting up a venue for those of you wanting to give a hand in this very difficult moment. You can already send any amount you deem appropriate via PayPal to ikonos dot sas at agora dot it.
@Mihai: thank you Mihai, that is so nice and kind of you. Truly appreciated my dear.
@Derek: thank you for your cool advice. Checked and though it is not our culprit, we still found something useful we were not aware of.
@Adam: yes, thanks for comforting me with this one. It is as you say, but we need really to be in Google main search results.
Robin
| 2007-08-16 00:53:23 |
Robin,
I have been following you and your writers for a very long time. You have the BEST new tech website on the internet!What really bothers me about this whole thing in addition to the suffering this is causing you and your staff, is what this implies...Google can be and may be BIG BROTHER!
Wow...they have the power to yank your livelihood right out from under you. That is disturbing.
I encourage us all to unite in some way to not let a company have that kind of power over our freedom.
I am just really pissed over this. I wish you a speedy recovery in Robin!
Hazel
| 2007-08-16 00:22:16 |
I'd happily pay a subscription upwards of $100 per year to ensure I keep recieving your news on new media. For me it's the bible on the subject.
Also consider running Ads in your RSS. I am sure you have thousands upon thousands pulling and reading RSS each day.
| 2007-08-15 11:23:25 |
Robin,
I'm really perplexed to hear this terrible news. Marjolein told me about it just a few moments ago
At this time of the year I have a lot of projects to work on, but if you need urgent technical support I am available to help you out with anything for free
I think the 4+ years spent working together on this and your other projects deserve now more than even concentrated efforts to restore your situation
Thank you for your great efforts invested on this publication and don't hesitate to immediately contact me if you need my help
Yours sincerely,
Mihai Bocsaru
| 2007-08-15 10:44:46 |
I hope you sort out your problems soon, this site has been the single best source of new technology and web tools, and I've learnt more here than any other site.
There are none better, and it deserves to survive, it's helped my own site improve drastically.
Good luck with Google...
Paul
The Ghost That Walks
| 2007-08-15 09:40:48 |
Hey Robin,
Google killed most of my traffic last month and I was stunned.
A quick check of Google's Webmaster tools showed DNS Lookup Timouts across the board.
I later learned that the firewall on my server had been updated, and for some reason, it saw Google's spider as something nasty, so it wouldn't allow it to index any of my sites on that server.
Content that was previously indexed was removed from Google.
After some tweaks to the firewall, Google's spider began its friendly visits again, and my pages are back.
I don't know if this is personally helpful to you, but it may be to others if not.
I'm very Web savvy, and this issue appeared out of nowhere and it took me several days to figure out.
Use Google's Webmaster tools regularly, check for errors, and do whatever Google tells you to do. :)
Derek Franklin
www.searchautomator.com
| 2007-08-15 09:02:08 |
I just wanted you to know the main Google search may have docked you search spots, but this article is on top in google blog search under "google" for now, so that should help some :)
I get my GoogleReader feeds from the google blog site and found this article and this source of independent news there.
GOOD LUCK,
Adam
 |
 |
 |
 |
 |
 |
|
|
|
|
|
|
|
|
|
|
|
 |
|
 |
Online Collaboration |
|
|
|
Independent Publishing |
|
|
|
Learning - Educational Technologies |
|
|
|
Content Delivery And Distribution |
|
|
|
Online Marketing |
|
|
|
Search Tools and Technologies |
|
||
|
News Archive | |
|
||
