Full paper
Calibrating Small Language Models for Claim Check-Worthiness Detection
By Pratuat Amatya, V Venktesh, Vinay Setty
The problem
Fact-checking organizations face a major challenge in identifying which online statements are worth verifying. Verifying information manually is a time and resource demanding practice. But when it comes to automating this process, Large Language Models (LLMs), that are big in scale, excel at detecting checkworthiness of claims, running them against millions of daily news articles and other types of political content which is too expensive. On the other hand, smaller AI models are affordable, but they often lack accuracy and suffer from prediction biases.
The solution
To solve this issue, Factiverse researchers created NN-PPI (Nearest Neighbor Prediction-Powered Inference), a lightweight post-hoc layer that corrects smaller AI models without training them again. So, when a small AI model evaluates an information claim, NN-PPI finds a few semantically similar claims from a small database of statements annotated by humans. Moreover, NN-PPI calculates the AI's typical prediction error of the similar claims and adjusts the final score.
The results
In tests using political debate and multi-domain news datasets, NN-PPI improved the accuracy of small AI models by 12% to over 30%. This enables low-cost, compact models to perform as well as giant, expensive LLMs. Ultimately, NN-PPI and the approach associated with it, makes automated fact-checking cheaper and quicker.
.avif)
