Sign in
Google

Google News Help Feed Feed

Google Help > Google News > Discussions > Google News Publishers - Troubleshooting > All of a sudden, Google quits crawling my website! Help

Question: All of a sudden, Google quits crawling my website! Help Report abuse

edaily
Level 1
7/11/09
at www.eritreadaily.net, we are also facing the same problem, We have been long time participants in Google News but all of a sudden, since beginnnig of July 2009, our articles are not picked up by Google News Crawlers. Our site meets all the technical requirements. Please help, thanks.
Best regards,
Brin

Answers: 1 of 49 All answers

methode
Top Contributor
7/11/09
Ahh, sorry. I missed this thread somehow.

Do you have a news sitemap? If not, it would be cool if you could submit one in webmaster tools.
If you can't for some reason create one, let me know and i create a temporary one for you which you can submit.

Also, please try to mess a bit with crawl rate in webmaster tools. It might be that the Crawler doesn't visits your site as often as before. You could set the crawl rate a bit higher.

If these things won't help, you will have to wait for a Googler to look into your issue.

Have a nice day
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/12/09
Dear methode:
I appreciate your feedback greatly, thanks bud!
Yes, I do have a news sitemap that I submitted since November 2008  is and has been working perfect and even downloaded by Google yesterday. Will mess with the crawl rate again, ty.
 
You have a nice day too, my friend.
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/12/09
OK, if you have a news sitemap, then please check on the detailed sitemap page (sitemaps>> click [sitemap_name]) whether the Type and the Format are both listed as "News"; this is very important, so pay attention to it. Please report back what you see.
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/12/09
Hi methode, yes, both Type and Format listed as "News". Thanks
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/12/09
Hi methode:
I miss your feedback, bud! checkout my site: www.eritreadaily.net I would appreciate it if you would get some feedback from Google Folks in this matter. ty.
 
Nice day
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/12/09
Did you change anything in your site's layout, address, URL of articles or anything in the HTML code?
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/12/09
dear methode, never, never bro. Still hope you would be able to contact Googlee folks, please. Thanks.
Greatful for your prompt response.
 
Nice Day.
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/12/09
They don't work on weekends and wouldn't ruin their weekend by email but will point them to this thread Monday morning (EDT).
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/12/09
methode, you are great! Appreciate your efforts in this matter greatly. God bless you.
Have a nice weekend, my friend
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/12/09
OK, a few more things I would like from you to check out.
In webmaster tools>>diagnostics, is anything interesting?
Can you tell me where's your sitemap located?
In webmaster tools, is any error reported on the detailed sitemap page?

I wouldn't really want to bother the Google guys if it's something too simple and we can also fix.
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/12/09
mithode, diagnostic is unremarkable: no error reporting, sitemap downloaded by Google 10 July 2009, status OK. I checked and rechecke everything, every thing is AOK. Even my Google Analytic is functioning perfect. sitemap is located in the index directory: www.eritreadaily.net/. My friend, am not IT savvy as you are but I know I met all the technical requirements of Google. I hope that the location of the sitemap is right, ty. Some thing is blocking all of a sudden the crawling and picking up of my news items after long years of functioning. I am stupified!
Great thanks
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/12/09
Hi methode, here the location of my sitemap: www.eritreadaily.net/public_html/sitemap.xml
 
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/12/09
That is not a news sitemap but a web sitemap.
Again, do you have a News sitemap[1]?
References:
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/13/09
That is all I have been using and worked sofar. I would appreciate it if you would help with a News sitemap please. ty
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/13/09
k,


Download it and save it on your webserver. Then submit it in webmaster tools just like you'd submit a normal sitemap, but pay particular attention to submit it as "News". This can be selected after you pressed the "Add sitemap" button from a drop-down menu.

If this solves your issue momentarily, you will have to code a sitemap generator which is capable of generating a news sitemap.
Another publisher already hired me to create one, but that's only capable to create a news sitemap from an RSS feed, since you have no RSS feed, you will have to find another way.
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/13/09
By the way. You can download the above sitemap by right clicking on it and then clicking on the "Save target as", "Save link as" or similar menu item.
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/13/09
methode, great thanks. let me try it. will report back asap. ty
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/13/09
Hi methode, I already have a website map. Now, in order to add a News site map, do I have to delete the website or is it possible to add the News site map separately. Please help. ty
 
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/13/09
methode, I mean do I have to give up (delete) my website sitemap inorder to submit the News sitemap? Thanks
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/13/09
No, you should have the option of submitting a news sitemap.
See the help entry I linked to below
References:
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/13/09
methode, won't allow a second site map cause it prompts me to enter a website url but won't accept it cause it is already indexed, Hope you get in touch with Googlefolks about this issue tomorrow and await anxiously for your valuable feedback, ty
 
Nice evening
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/13/09
Hi methode: I was able to submit the News sitemap you created successfully this morning and Google status is OK. But the news items were not picked by Googlebot or may be the site not crawled, Thanks for creating the News citemap but I would appreciate it if you would get Google folks to look into this issue. Great thanks.
Nice day
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/13/09
methode, you checkout my sitemaps here: https://www.google.com/webmasters/tools/sitemap-list?hl=en&siteUrl=http%3A%2F%2Fwww.eritreadaily.net%2F
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/14/09
methode, you there! miss your feedback, bro! Don't let me down, need your professional and valuable feedbacks now than ever, ty.
Nice day
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/14/09
https://www.google.com/webmasters/tools/sitemap-list?hl=en&siteUrl=http%3A%2F%2Fwww.eritreadaily.net%2F
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/14/09
Am sorry, but I don't work for Google but for another corporation and sometimes I have to do that job as well.

Also, I can't check out your news sitemap on that location because it's protected by your login.

One more thing and then I report the thread to the Google engineers. Please follow the URL you posted above then click on the sitemap's filename you have recently added and which should be a news sitemap. A table will show up which several details of the news sitemap. Which is interesting for us is the "Type" and "Format" data. Please tell me these two values.
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/14/09
methode, the one site has the format: Web, the other which you created and is accepted by Goog has the forma: News
 
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/14/09
OK, and type? Click on the News  sitemap's name then see what it says for "Type"
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/14/09
Type too: website and News bro, ty
 
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/14/09
K, I report the site now. Please doublecheck the news sitemap's format and type. 

We will still have to figure out how to create for you an always up-to-date news sitemap.
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/14/09
methode, thanks bro. That would be great with the always up-to-date sitemap, God help you, buddy. Will check with you over the day. ty again for your help
 
Nice day
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/14/09
Hi methode, sorry to bother you again. Some one said this is the problem; <META HTTP-EQUIV="EXPIRES" CONTENT="-1"><META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE">  Would please help figure that out? How do I fix that. Thanks
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/14/09
As far as I know that causes no issue. Just think: it didn't cause issue till now, why would now?
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/14/09
methode, my friend, that is exactly how I responded too! Yes, that didn't cause issue for years, why now? Any way, were able to get response from Google folks yet? Thanks for hangign on. Hope to hear from you again. ty
 
Have a nice day
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/14/09
No, I let them know and much likely they are working on a solution or its on the to-do list
Please, do not expect a 2-minute-fix. They have loads of work.
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/14/09
methode, apology, don't mean to hustle them at all. Just curious if they at respond! You know, the site is crawled but the new items are not posted on Google News page, that is my worry! Once again great thanks and you have a wonderful day.
Do you think this answers the question? Sign in to vote. Report abuse
Inbal
Google Employee
7/15/09
methode (Top Contributor) + 2 other people say this answers the question:
Hi edaily,

Apologies for the confusion regarding your news content stopped being indexed in Google News.
The reason why we stopped crawling your content since July is that we periodically review news sources, particularly following user complaints, to ensure Google News offers a high quality experience for our users. When we reviewed your site we found that we can no longer include it in Google News at this time.

We currently only include articles from sources that could be considered organizations, generally characterized by multiple writers and editors, availability of organizational information, and accessible contact information. When we reviewed your site we weren't able to find this evidence of an organization.

If you have other information about your site that we should consider, please contact us via the link below.

Best Regards,
Inbal
References:
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/15/09
Eritrea Daily is a news website, we post news articles reported by various News Agencies like, AFP, AP, BBC, CNN, other websites like shabait.com, zhinua and international organizations like UN, AU, EU and more. Moreover we post our own op-ed. Basically, our website is a New trove for news about Eritrea in particular and horn of Africa in general and world affairs. Our contac address webmaster@eritreadaily.net, or mfilli@eritreadaily.net  telofon: 732-397-8849. Let me know if you need more info ty. We are hurt by the action you took. Hopefully you give the essence of our website due consideration and reverse your decision because it also hurts revenues over GoogleAdsense that we have been participating in for so long. Hope to hear from you soon in this matter, ty
PS-there are so many contat possibilities, I would appreciate it if you would direct me the right issue that pertains to my issue, ty
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/15/09
inbal, do you include articles from News troves? Eritrea Daily is a news trove: It posts news articles reported by various new agencies, organizations and others, ty
 
 
Regards,
edaily
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/15/09
Dear inbal, I would have hoped Google would warn me about the danger of not being included in its Google News and necessary remedy would have be taken before the action taken so abruptly. Moreover, wish I knew if News Trove such as my site would met the requirement of organization. ty hope to hear from you how long the exclusion would last, ty.
Best regards,
Brin for edaily
Do you think this answers the question? Sign in to vote. Report abuse
Inbal
Google Employee
7/15/09
1 person says this answers the question:
Dear Brin,

Please be assured that we tried to contact Eritrea Daily via email before removing it from Google News to offer the necessary remedy, however, alas, our emails bounced back, and due to some user complaints about no evidence for an organization behind the source we took this action, as we weren't able to find multiple writers and editors and accessible contact information on your site.
However, as you've provided us with this additional information about your site, we'll be re-adding it into our index for Google News.

Best regards,
Inbal
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/15/09
Additionally, You will still have to provide some form of contact information, list of editors and organizational info on your website.
Just think with your visitor's mind: they want to know more about the news source they read articles from, providing the missing information makes your site to be a more credible news source.

Have a nice day
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/15/09
Hi methode, finally good news. Just above your latest entry, Inbla posted an answer where he accepted the additional info about my site and decided "we'll be re-adding it into our index for Google News." I appreciate all your efforts in this matter and hopefully you will find a way to turn the temporary News sitemap you created for me into permanent one.
 
Have a great day my friend
Brin
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/15/09
1 person says this answers the question:
We had to see whether there's a problem which we can fix ourself or is something beyond us; that's why the thread grew this long. The Googlers have enough work already, no need to bother them when there's something we can also fix.
Yes, I saw that post, I responded to add a bit more info. Note that if the information is not present on your website someone could report it again and not necessarily Inbal will process the report thus your site might get removed again. Please, do provide the info and form of working contact on your website

As for creating a permanent news sitemap, it's not necessarily a need. If the system you were using till now worked, it will in the future as well. The temporary sitemap was created to test whether the crawler has issues identifying individual articles, nothing more. When I have time, i will create a tool like the one you are using to generate the web sitemap, though. 
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/16/09
methode, thanks again. There was already contact link, and Mission statement on our site. Any way I have now included the necessary infos on the site. Yea, you are right, never needed News sitemap for so long and I am about to remove the test sitemap. Appreciate all efforts and concerns, God bless you! Amen
 
Best regards,
Brin
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/16/09
Inbal great thanks for your decision to re-include my site in Google News. Appreciate it greatly. I know you got a ton of emails to respond to, but certainly hope that it won't be too long before our news items appear in Google News page again. ty
 
Best regards,
Brin
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/16/09
1 person says this answers the question:
Your site will be processed again in a matter of few days, max a week.
Do you think this answers the question? Sign in to vote. Report abuse
edaily
Level 1
7/16/09
Dear methode: Two thumbs up for your tireless efforts to help others and for boosting my moral. Great thanks and you have a great day.
 
Besst regards,
Brin
Do you think this answers the question? Sign in to vote. Report abuse
methode
Top Contributor
7/16/09
Your welcome.

If you have any more question, let me know; I'd be happy to answer them if I can or am allowed. 
Though, you should start a new thread for them as this has been already answered.
Do you think this answers the question? Sign in to vote. Report abuse

Post reply

Sign in to answer this question.

Subscribe