Analysis by the Complexity Science Hub shows the misinformation watchdog has a high level of reliability across countries and provides stable ratings for news sources
[Vienna, 30.01.2025]—A recent study evaluating the NewsGuard database, a leading media reliability rating service, has found no evidence supporting the allegation that NewsGuard is biased against conservative news outlets. Actually, the results suggest it’s unlikely that NewsGuard has an inherent bias in how it selects or rates right-leaning sources in the US, where trustworthiness is especially low.
“It seems unlikely that NewsGuard has an inherent bias against conservative sources, both in selecting and giving them lower ratings. Instead, the US media system is flooded with right-wing sources that tend to not adhere to professional editorial practices,” says first author Jula Lühring, from the Complexity Science Hub (CSH).
Stable Trustworthiness Over Time
Lühring and her colleagues analyzed NewsGuard’s trustworthiness ratings for more than 11,000 news sources in nine countries: United States, Great Britain, Italy, Canada, France, Germany, Austria, Australia, and New Zealand. The results showed that the ratings have remained consistent since 2022, with particularly stable coverage in the US, France, Italy, Germany, and Canada.
“NewsGuard selects the majority of news sites using web traffic data,” explains Lühring, also a PhD student at the University of Vienna. “We manually checked sources in the US, UK, and Germany, and found that the database misses almost no news sites with substantial traffic. The sites it does miss are not systematically biased towards any political ideology.”
Lowest Trustworthiness in the US
US-based news sources, however, consistently received lower trustworthiness scores compared to those from other nations–particularly for right-leaning sources, according to Lühring. “Smaller (hyper-)partisan sources tend to lack editorial practices and transparency measures. Since these are key criteria for NewsGuard, judging based on NewsGuard criteria results in an objectively lower overall trustworthiness,” explains the researcher.
“Our findings add to previous findings by Lin et al. (“High level of correspondence across different news domain quality rating sets”, published in 2023 in PNAS Nexus), who showed that trustworthiness ratings of news sources by NewsGuard align with ratings of other fact-checking endeavors,” adds co-author Jana Lasser, a professor at the University of Graz and associate faculty member at CSH.
Changes in trustworthiness scores over time: Flags and dotted lines indicate when countries were added and dashed lines with green and red arrows highlight the top five major score changes. The height of the bars describes the number of sources added vs. changed, with colors indicating the proportions. Also shown are the average trustworthiness of the sources in green text boxes.
Timely Analysis Amid Political Tensions
“The evaluation is particularly timely given the upcoming change in the US government. The incoming Trump administration regulators and far-right Republicans in Congress are not only trying to hamper the work of misinformation researchers in the US, but also accusing NewsGuard to systematically censor conservative news sites,” points out co-author Hannah Metzler, a resident scientist at CSH.
Lasser also notes that misinformation and disinformation have become a global concern in recent years. “Tools to measure the trustworthiness of sources–like the NewsGuard database–are therefore crucial to quantify the spread of untrustworthy information in online environments.”
Navigating the Challenges of Misinformation Research
While the study confirmed NewsGuard’s general reliability, the researchers also caution about the limitations of binary trustworthiness labels. These limitations, they argue, could affect the validity of some misinformation research, and they offer recommendations for more nuanced approaches when using the NewsGuard database.
“We found that using a binary ‘trustworthy’ vs. ‘not trustworthy’ classification is prone to changes in the database over time, potentially leading to large variations in the measured prevalence of untrustworthy information. Therefore, if possible, the continuous point score that provides a trustworthiness assessment on a more fine-grained scale should be used,” explains Lasser.
About the Study
The paper “Best practices for source-based research on misinformation andnews trustworthiness using NewsGuard,” by Jula Lühring, Hannah Metzler, Ruggero Lazzaroni, Apeksha Shetty, and Jana Lasser, was published in the Journal of Quantitative Description: Digital Media. DOI: 10.51685/jqd.2025.003
About CSH
The Complexity Science Hub (CSH) is Europe’s research center for the study of complex systems. We derive meaning from data from a range of disciplines—economics, medicine, ecology, and the social sciences—as a basis for actionable solutions for a better world. Established in 2015, we have grown to over 70 researchers, driven by the increasing demand to gain a genuine understanding of the networks that underlie society, from healthcare to supply chains. Through our complexity science approaches linking physics, mathematics, and computational modeling with data and network science, we develop the capacity to address today’s and tomorrow’s challenges.
