Can Wikipedia Ever Be a Definitive Medical Text?

How to create a credible, accessible, crowd-sourced resource for health information

Every time I panic unreasonably over some minor bodily abnormality—which is often—I take to the Internet. I’m far from the only one—72 percent of Internet users have looked online for health information in the past year, according to Pew Research. And though as a responsible health editor, I should of course say that if you really think something’s wrong, you should go to the doctor, I know that even if you do go to the doctor, chances are you’ll Google whatever she tells you anyway.

Wikipedia being the sixth-largest site on the whole wide Internet, these people searching for medical information online are often going to end up there. Whether or not they should be doing it, they are. I am. Patients are, and so are doctors. Which is why efforts to improve the quality of Wikipedia’s medical information are important—if you can’t lead people away from the fountain of crowd-sourced knowledge, you can at least try to unmuddy the waters.

A new study published in The Journal of the American Osteopathic Association says the waters are still pretty muddy. The researchers looked at the Wikipedia pages for the 10 most costly medical conditions (in terms of public and private expenditures) and compared them with peer-reviewed sources, finding them wanting.

“Most Wikipedia articles representing the 10 most costly medical conditions in the United States contain many errors when checked against standard peer-reviewed sources,” the study reads. Unfortunately, the question of Wikipedia’s accuracy is a little more complicated than that.

“The story started a couple years ago,” says study lead author Dr. Robert Hasty, an osteopathic physician and an associate dean at the Campbell University School of Osteopathic Medicine. “We were sitting in an educational session with the residents and I saw one of them using Wikipedia to look up information. I said: ‘Is there any evidence that’s a good resource?’”

There is, some. A study published in Nature in 2005 found that Wikipedia was about as accurate as the Encyclopedia Britannica, after experts on 42 different subjects reviewed articles from each source. But, when you consider the broad scope of Wikipedia, 42 is a pretty small sample size. And there was a bit of a kerfuffle over the study’s accuracy, with Encyclopedia Britannica responding that “everything about the journal’s investigation…was wrong and misleading,” and Nature responding to the response saying, “We reject those accusations, and are confident our comparison was fair.”

Looking specifically at medical information, research published in BMJ found that many peer-reviewed health science studies themselves cite Wikipedia as a reference. Studies on specific areas of medicine have come down on different sides of the debate: One says that Wikipedia is “appropriate for use by nursing students,” another says the “depth of discussion” is not sufficient for it to be a “reliable source of information for medical students searching for gastroenterology and hepatology articles.” Good for information about kidneys and mental disorders, not great for ear, nose, and throat conditions in children.

None of that really helps answer the question “Is Wikipedia a good resource?” and, though this new study attempts to generalize broadly on the quality of medical information, it doesn’t really answer that question either. The “many errors” the study says it found is a vague measure, and Hasty tells me that “a lot of the [errors] I saw were small little details. We didn’t power the study to look at what’s important.”

To determine the accuracy of the articles, the researchers had medical interns and residents find all the assertions an article made, then search for peer-reviewed sources from the last five years to check those assertions against. Peer-reviewed sources being, Hasty says, the “gold standard” of medical information. A couple problems, which Hasty acknowledges, are that the reviewers didn’t always agree about the number of assertions an article contained, and they could use pretty much any peer-reviewed source they found.

“One of the challenges is what they define as a peer-reviewed reference is questionable quality sometimes,” he says. “In the future, probably experts in that field would be better to review the articles.”

The doctors of Wikiproject Medicine, an effort within Wikipedia to improve the quality of medical information by getting physicians and health care experts to edit the articles, suggest a different gold standard source.

“We try to use secondary sources like reviews, meta-analyses, and major textbooks,” says Dr. James Heilman, a clinical instructor in the University of British Columbia’s department of emergency medicine, and one of the most active medical editors on Wikipedia. His reasoning is that, for these secondary sources, someone has typically taken the time to review all of the peer-reviewed literature, which is often contradictory, and come to a conclusion based on what’s out there.

This study was not popular on the Wikiproject Medicine Talk page, with one commenter calling it “utterly meaningless and [not] worth the electrons it's printed on.”

Heilman, along with Dr. Samir Grover at the University of Toronto, is designing a study that he says would be a better way to test Wikipedia’s accuracy: have medical students take a standardized test using either medical textbooks or Wikipedia. Or, he says, you could look at whether an article’s statements are supported by the references it provides for those statements—not just any peer-reviewed source you find.

“Just because a reference is peer-reviewed doesn’t mean it’s a high-quality reference,” he says.

Wikipedia has its own peer review process before articles can be classified as “good” or “featured.” Heilman, who has participated in that process before, says “less than 1 percent” of Wikipedia’s medical articles have passed.

So both sides acknowledge: There are errors in Wikipedia’s health articles. And that’s a problem, because people use them.

“Amongst my colleagues, [using Wikipedia] is a little bit like flatulence,” Hasty says. “Everybody does it but nobody wants to admit to it…Whatever we can do to encourage one of the most popular sites to be more accurate is very good. I think that’s one of the positive things about this article.”

Both Hasty and Heilman emphasize that even as Wikipedia’s quality improves, no one making a serious medical decision—patient or doctor—should ever use just one source.

“It takes judgment, no matter what happens,” Hasty says. “That’s another reason we should use professionals who do devote their lives to ensuring patients get the best recommendations.”

Keeping that in mind, Wikiproject Medicine’s goal of having those same professionals editing Wikipedia pages seems like the best option. In the face of constantly-evolving medical literature, and primary—and even sometimes secondary—sources that contradict one another, professional judgement may be the goldest standard we have.

Julie Beck is a senior editor at The Atlantic, where she oversees the Family section, and is the creator of “The Friendship Files.”