Google News now crawling with Googlebot

Thursday, August 25, 2011 2:00 PM

Posted by David Smydra, Google News Product Specialist

(Cross-posted on the Webmaster Central Blog)

Google News recently updated our infrastructure to crawl with Google’s primary user-agent, Googlebot. What does this mean? Very little to most publishers. Any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot.

Our Help Center provides detailed guidance on using the robots exclusion protocol for Google News, and publishers can contact the Google News Support Team if they have any questions, but we wanted to first clarify the following:

• Although you’ll now only see the Googlebot user-agent in your site’s logs, no need to worry: the appearance of Googlebot instead of Googlebot-News is independent of our inclusion policies. (You can always check whether your site is included in Google News by searching with the “site:” operator. For instance, enter “site:yournewssite.com” in the search field for Google News, and if you see results then we are currently indexing your news site.)
• Your analytics tool will still be able to differentiate user traffic coming to your website from Google Search and traffic coming from Google News, so you should see no changes there. The main difference is that you will no longer see occasional automated visits to your site from the Googlebot-news crawler.
• If you’re currently respecting webmaster guidelines for Googlebot, you will not need to make any code changes to your site. Sites that have implemented subscriptions using a metered model or who have implemented First Click Free will not experience any changes. For sites which require registration, payment or login prior to reading any full article, Google News will only be able to crawl and index the title and snippet that you show all users who visit your page. Our Webmaster Guidelines provide additional information about “cloaking” (i.e., showing a bot a different version than what users experience). Learn more about Google News and subscription publishers in this Help Center article.
• Rest assured, your Sitemap will still be crawled. This change does not affect how we crawl News Sitemaps. If you are a News publisher who hasn’t yet set up a News Sitemap and are interested in getting started, please follow this link.
• For any publishers that wish to opt out of Google News and stay in Google Search, you can simply disallow Googlebot-news and allow Googlebot. For more information on how to do this, consult our Help Center.
  

As with any website, from time to time we need to make updates to our infrastructure. At the same time, we want to continue to provide as much control as possible to news web sites. We hope we have answered any questions you might have about this update. If you have additional questions, please check out our Help Center.

Enhancements to Google News for Android tablets and iPads

Tuesday, August 9, 2011 10:00 PM

Posted by Arun Prasath, Tech Lead, Mobile Google News

Alongside working on improving the Google News design for smartphones, we have also been looking into enhancing our offering for tablet devices. Today, we are launching a few minor enhancements to Google News for Android tablets and iPads.

We have optimized columns in the home page and section pages so that they can be more easily viewed in portrait and landscape orientations. Some of the other updates include: a conveniently placed menu on the top for navigating across sections, support of finger swiping through the multimedia strip in expanded story boxes and a more friendly edition picker.


These improvements are now available in the U.S. edition. To view these changes, just type news.google.com on the browser of your favorite tablet.

Google News highlights unique content with Editors’ Picks

Thursday, August 4, 2011 1:11 PM

(Cross-posted on the Official Google Blog)

News organizations tell stories online in ways that bring together the best of traditional and digital journalism, whether that involves long-form investigative features, compelling photo slideshows or interactive maps and charts that add new levels of engagement to the day's news. To help connect you to the best works of news publishers, Google News is introducing a new section in the right-hand column of the U.S. edition. The section is called "Editors' Picks,” and it displays original content that publishers have selected as highlights from their publications. This is the latest addition to recent improvements we’ve made to the variety and presence of stories and multimedia on Google News.

An array of news organizations, including local, national and niche publishers, are now using Editors’ Picks to display their best, most engaging content. Because Google News relies on algorithms, Editors' Picks will always be just that—picks provided by publishers themselves, and not by Google. You can browse a set of publisher feeds that span national, specific and local interests—like The Atlantic, The Wall Street Journal, ProPublica, the Guardian and The Root, among many others—via the side-to-side arrows next to each publisher's logo. The feeds you see are chosen based on a variety of factors, including your news preferences. If you’re interested in using source preferences on Google News, Editors' Picks helps you do that with the slider that appears just below the articles.

You may have first noticed Editors’ Picks as an experiment last year. Based on the data from that experiment, we have been working with nearly two dozen publishers in recent months and have seen a positive response from readers and publishers alike: readers get the news they're interested in from the sources they trust, and publishers receive higher traffic to their websites. We encourage any news organizations that are interested to visit our Help Center to get started.

Posted by Yogita Mehta, Software Engineer, Google News Team

A Few Google News Design Changes

Thursday, July 21, 2011 2:49 PM

Posted by Jasson Schrock, User Experience Designer

Beginning today in the U.S. English edition of News, you may notice a few changes in the look and feel. This is part of a Google-wide initiative to improve your experience, and today we’re happy to extend this to Google News.

For starters, we’ve cleaned up the Google News homepage a bit. With fewer visual distractions and less clutter on the page, it should be faster to find the news important to you. The editions menu and “Personalize” button at the top should be easier to see. The “Personalize” button links to your recently consolidated personalization settings including your preferences for topics and favorite sources. Throughout the site, all the features and functionality are still there, but this updated design is aligned with the new consistent look across other recently updated Google sites like Gmail, Maps and Search.

Before

After

In the coming months, you’ll continue to see more improvements to the layout and design of the site, and we’ll also expand internationally.

Shareable Google News badges for your favorite topics

Thursday, July 14, 2011 4:00 PM

Posted by Natasha Mohanty, Engineer, Google News

(Cross-posted on the Official Google Blog)

On Google News, the average reader of political news has read 20 articles about politics in the last six months. Where do you stand?

Starting today, in the U.S. edition of Google News, you can see how voracious a news reader you are by earning Google News badges as you read articles about your favorite topics. The more you read, the higher level badge you’ll receive, starting with Bronze, then moving up the ladder to Silver, Gold, Platinum and finally, Ultimate.



We have more than 500 badges available, so no matter what kind of news you’re into, there’s a badge out there for you. Here’s a taste:

Your badges are private by default, but if you want, you can share your badges with your friends. Tell them about your news interests, display your expertise, start a conversation or just plain brag about how well-read you are. You can also add custom sections by hovering on a badge and clicking “add section” to read more about your favorite topics. To get started with badges, visit Google News from a signed-in account with web history enabled and then visit this page on our Help Center for instructions.

This is just the first step—the bronze release, if you will—of Google News badges. Once we see how badges are used and shared, we look forward to taking this feature to the next level.

In the spirit of continually trying to improve Google News, we have heard loud and clear from the many of you who asked us to separate our Sci/Tech section into two distinct sections. We are happy to report that we have now done this for all English editions, with more languages coming soon. We also combined some personalization settings from the “News for you” and News Settings menu into one handy sidebar at the top right corner of the home page, so you can easily tell us what you want to read on your Google News.



We hope you’ll badge up on Google News to keep track of what you’re reading, read more of what you love and share your passions with your friends.

Applauding the 2011 Knight News Challenge winners

Wednesday, June 22, 2011 11:49 AM

(Cross-posted on the Official Google Blog and the Google Public Policy Blog)

Over the past few months, we’ve announced $5 million in grants to be distributed by the John S. and James L. Knight Foundation and the International Press Institute—two non-profit organizations developing new approaches to journalism in the digital age—and we’re pleased to congratulate the first initiatives that have been selected as part of that funding.

Today at M.I.T., the Knight Foundation showcased 16 projects selected as the winners of the 2011 Knight News Challenge. Now in its fifth year, this media-innovation contest included $1 million in support from Google. As you’ll see in the full list of winners, these initiatives come from organizations large and small and are reminders that entrepreneurship can be sparked anywhere. Here are just a few examples of the creative ways the journalism community around the world is merging traditional skills with an online landscape:

• At the University of North Carolina at Chapel Hill, OpenBlock Rural will use its seed money to work with local governments and community newspapers across the state to collect, aggregate and publish data.


• In Virginia, the Miller Center Foundation’s State Decoded will serve as a platform to display state codes, court decisions and information from legislative tracking services to make government more understandable to the average citizen.


• The Chicago Tribune will collaborate with the Investigative Editors & Reporters organization and The Spokesman-Review on a set of open-source, web-based tools that make it easier for journalists to use and analyze data.


• Liverpool, U.K.-based ScraperWiki will bring its experiences with public data to journalism camps in 12 U.S. states.


• Chile’s El Mostrador will develop an editorial and crowdsourced database to bring greater transparency to potential conflicts of interest.


• Ushahidi will build off its past crisis efforts to improve information-verification across email, Twitter, web feeds and text messages.

Other winning proposals tell rich multimedia stories, bridge the gap between traditional and citizen media and further improve the utility of data to journalists. Our sister program in partnership with the International Press Institute is also well underway. The entries in that competition are now in and the winners will be announced later this summer. We look forward to seeing the impacts these initiatives have on digital journalism and hope they encourage continued experimentation and innovation at the grassroots level.

Posted by Jim Gerber, Director, Strategic Partnerships, News

Rededicating the Newseum’s Journalists Memorial

Wednesday, May 18, 2011 9:00 AM

Posted by Sean Carlson, Global Communications & Public Affairs

Outside 555 Pennsylvania Avenue in Washington, D.C., the front pages of newspapers from all 50 U.S. states mark the entrance to the Newseum. Inside the lobby, a gallery of Pulitzer Prize-winning photographs confronts visitors with moments of triumph and tragedy captured on camera. The museum’s corridors display exhibit after exhibit highlighting the role of journalism and journalists throughout history.

One of the Newseum’s most moving tributes is its Journalists Memorial, a wall of glass paneling imprinted with the names of more than 2,000 people around the world who have died while reporting the news. Nearby kiosks narrate their stories, an online database enables anybody with Internet access to learn more, and our new YouTube channel further remembers these fallen journalists through video.

Earlier this week, Krishna Bharat, founder and head of Google News, spoke at the memorial’s annual rededication ceremony. As you can watch in the video below, he began by reflecting on what motivated those being honored “to walk a path that was not paved with gold, but with danger.”

Over the course of his address, Krishna discussed the importance of a free press to society and of high-quality content to the web, observed the rising number of online journalists and bloggers coming under attack, and recounted incidents reported by the Committee to Protect Journalists. On a personal level, he also shared his memories as a boy in India and the influence of his grandfather in inspiring his appreciation for news.

“The journalists we remember and honor today chose lives that were full of meaning and purpose,” he concluded. “Let their stories not be forgotten. Let us repeat them. Let us re-tweet them. And let us print them on our pages so the world knows that silencing a journalist simply does not pay.”

To their families, friends, and colleagues, we extend our sympathy and respect.

Expanding Google News for more variety and multimedia

Monday, May 16, 2011 10:00 AM

Posted by Chase Hensel, Associate Product Manager, Google News

Every day, Google News crawls through thousands of news articles to present you with the most relevant and recent stories. For a long time, we’ve realized that bringing relevant news to the surface is only part of the puzzle—it should also be easy to scan for stories of interest and dig deeper when you find them.

The newly expandable stories on Google News in the U.S., released today, give you greater story diversity with less clutter.

Now you can easily see more content, see less of what you don’t use and have a more streamlined experience:

• Click-to-expand: Each story cluster is collapsed down to one headline with the exception of the top story. When something grabs you, click nearby anywhere but the title to expand the story box.
• Labeled diversity: For stories you’ve expanded, you’ll see genre labels for some of the additional articles that explain why they were chosen and how they add value. For example, you might see something labeled as an “Opinion” piece or an indication that an article is “In Depth.”
• Multimedia and more: Within each expanded story box, you’ll find a sliding bar of videos and photos, links to related sections and easier-to-use sharing options, so you can quickly digest the sights and sounds of a news story, dig into different types of publications and share what you find interesting with one click.
• Personalized top stories: The Top Stories section is expanded to six or more stories from three to give you more topic diversity. The first three stories remain unpersonalized and the same as before. The rest may be personalized based on your interests. To personalize your Google News experience you can click on “Edit” under “News for you.” You can choose the “Standard Edition” if you don’t want personalization.
• Less is more: The default view is now the popular “One Column” (formerly “Section”) view. We merged List View into Top Stories, as described above. You can still switch to “Two Column” view, which resembles classic Google News.

We hope you like these changes—please share your feedback and visit our Help Center to learn more.

Introducing “News near you” on Google News for mobile

Friday, May 13, 2011 10:15 AM

Posted by Navneet Singh, Product Manager - Google News

Google News for mobile lets you keep up with the latest news, wherever you are. Today we’re excited to announce a new feature in the U.S. English edition called “News near you” that surfaces news relevant to the city you’re in and surrounding areas.

Location-based news first became available in Google News in 2008, and today there’s a local section for just about any city, state or country in the world with coverage from thousands of sources. We do local news a bit differently, analyzing every word in every story to understand what location the news is about and where the source is located.

Now you can find local news on your smartphone. Here’s an example of a “News near you” mobile section automatically created for someone in Topeka, Kansas:


To use this feature, visit Google News from the browser of your Android smartphone or iPhone. If this is the first time you are visiting Google News on your phone since this feature became available, a pop-up will ask you if you want to share your location. If you say yes, news relevant to your location will appear in a new section called “News near you” which will be added at the bottom of the homepage. You can reorganize the sections later via the personalization page.


You can turn off the feature at any time either by hiding the section in your personalization settings or by adjusting your mobile browser settings. Please visit the Help Center for further details.

So, go to news.google.com from your smartphone and get the latest news from wherever you are.

Google News and the Coverage of Bin Laden

Friday, May 6, 2011 11:00 AM

Posted by Krishna Bharat, Founder and Head - Google News

Google News was born in the aftermath of the tragic events of September 11, 2001. An unprecedented act of terrorism on U.S. soil, by a foreign militant group led by Osama Bin Laden, changed the course of history. People around the world were trying to comprehend what had just happened, and its implications to public safety, foreign policy, financial markets, and their own lives. Much of that exploration happened online.

At Google we realized that our ability to display links to the freshest and most relevant news was limited by a fundamental problem: fresh news lacked hyperlinks. Google’s ranking depended on links from other authors on the web. Fresh news, by definition, was too fresh to accumulate such links. A new importance signal was needed.

I realized that if Google could compute how many news sources were covering the underlying story at a given point in time, we could then estimate how important the story was. Thus, “Storyrank” was invented. This insight led to a ranking that combined the editorial wisdom of many editors on the web in real time. In addition to making search better it led to Google News - a display of stories in the news ranked automatically by an algorithm. This also allowed us to group news articles by story, thus providing visual structure and giving users access to diverse perspectives from around the world in one place.

After 10 years Mr. Bin Laden is in the news again. The story of the killing of Bin Laden has taken the online world by storm. This time, relevant coverage from around the world is just a click away, in an automatically compiled Google News cluster with more than 80,000 sources.

We have certainly come a long way in the last decade. Indeed, Google News now has over 70 editions in over 30 languages, and sends over 1 billion clicks a month to news publishers worldwide. Additionally, 1 out of 6 web searches on Google includes a set of news results, which are computed with the help of Storyrank. This helps bring coverage of the most important news story matching the query to the top of the ranking.

In the last 10 years there has been a lot of learning, iteration, and innovation in our team. And most importantly, we have acquired a loyal audience of news enthusiasts, who appreciate diversity and the ability to access multiple points of view on a story. To our users we would like to say “Thank You!”

We wanted to share with you some of the news coverage of the death of Bin Laden. Here is a sample of 100 links to news articles from representative sources worldwide:

ABC News - Abril - Agenzia Giornalistica Italia - ANSA.it - Associated Press - Atlanta Journal Constitution - Baltimore Sun - BBC News - Billboard - Bloomberg - Boston Globe - Boston Herald - BusinessWeek - CBC.ca - CBS News - CBSSports - Chicago Sun-Times - Chicago Tribune - Christian Science Monitor - CNET - CNN - Computerworld - Corriere della Sera - Dallas Morning News - derStandard.at - Detroit Free Press - E! Online - El Pais (Colombia) - El Paí­s (España) - El Universal (Venezuela) - ESPN - Forbes - Fox News - Globe and Mail - Ha'aretz - Hindustan Times - Huffington Post - InformationWeek - Jerusalem Post - Jewish Telegraphic Agency - Kansas City Star - La Repubblica - La Stampa - Le Point - Los Angeles Times - MarketWatch - MLB.com - MSNBC - MTV - National Geographic - National Post - NDTV - New York Daily News - New York Times - New Yorker - Newsday - Newsweek - NFL News - NPR - NZZ Online - O Globo - PC Magazine - PCWorld - People Magazine - Philadelphia Inquirer - Politico - Reuters - RollingStone - Salt Lake Tribune - San Francisco Chronicle - San Jose Mercury News - Seattle Post Intelligencer - SI.com - Slate Magazine - Spiegel Online - Sydney Morning Herald - Telegraph.co.uk - The Atlantic - The Economist - The Guardian - The Hindu - TIME - Times of India - Toronto Sun - U.S. News & World Report - Us Magazine - USA Today - Vancouver Sun - Vanity Fair - Voice of America - Wall Street Journal - Washington Post - WELT ONLINE - Wired News - Yahoo! Sports - ZDNet - العربية ن - الجزيرة - 朝日新聞 - 読売新聞

For those you who enjoy digging into data, here is a much larger list of over 150,000 links to news articles mentioning Osama Bin Laden over the last 5 days (May 1-5, 2011).

One of the many lessons I learned from 9/11 is that the world is highly connected. We live in a global society crisscrossed by virtual and physical dependencies, where knowledge is power and ignorance has consequences. This is a world where knowing what is happening to people in other parts of world, and understanding their circumstances and beliefs, matters more than ever -- because their actions will ultimately affect our lives. Tools such as Google News, which bring order to information and make search smarter can help us cope with the complexity of news and understand the big picture.

Further, as the wave of revolutions in North Africa demonstrates, online information does not merely reflect world events -- it can even cause them. These are indeed exciting times for those of us who work in the news space and get to witness the impact of journalism on society first hand!