Zaxiu88888/issue-rag, a retrieval-augmented question answering system, has been making waves on GitHub with its ability to audit the accuracy and hallucination rates of Large Language Models (LLMs). But what exactly is zaxiu88888/issue-rag, and what can it do for users?
Background And Purpose
Zaxiu88888/issue-rag is an open-source tool designed to measure the accuracy of LLMs in answering questions related to a specific project’s issue tracker. The project’s primary purpose is to evaluate how well LLMs can retrieve relevant information from a project’s issue tracker and provide accurate answers. This is particularly useful for developers and maintainers who need to ensure that their LLMs are providing reliable and trustworthy information.
How It Works
The system uses a retrieval-augmented generation approach, where it first retrieves relevant information from the issue tracker and then generates an answer based on that information. The system is designed to work with any project’s issue tracker, but it has been tested on the vitejs/vite project, which has a large and active issue tracker. The evaluation set consists of 95 hand-reviewed items, with each question being a real issue title and each reference answer being the maintainer’s reply that closed the issue.
Key Performance Numbers
The results show that retrieval-augmented generation can significantly improve the accuracy of LLMs. Under the in-project judge, retrieval lifts accuracy from 0.221 to 0.700, a gain of 3.17×. An independent stricter grader measures a gain from 0.305 to 0.711, a 2.3× increase. A blind human review of 30 answers measured a gain from 0.167 to 0.867, a 5.2× increase. However, the system also found that retrieval can raise the fabrication rate, which is the rate at which the LLM generates incorrect information, from 1.1% to 20%.
Surprising Findings
One surprising finding is that the four retrieval configurations are mutually indistinguishable, with a spread of 0.632 to 0.700, which is smaller than the grader’s own noise. This means that the choice of retrieval configuration does not significantly impact the accuracy of the system. Another surprising finding is that the judge gives vague advice more credit than a person does (+0.100) and good retrieval answers less (−0.133). This means that the machine-measured gain is a conservative estimate, not an inflated one.
Who Is It For?
Zaxiu88888/issue-rag is designed for developers and maintainers who want to ensure that their LLMs are providing accurate and reliable information. The project provides a hand-reviewed evaluation set and a results table that can be used to evaluate the performance of different LLMs and retrieval configurations. The system is also useful for researchers who want to study the behavior of LLMs and the impact of retrieval-augmented generation on their accuracy. With its open-source design and easy-to-use interface, zaxiu88888/issue-rag is a valuable tool for anyone working with LLMs and issue trackers.
Sources
Photo: Jake Walker / Unsplash (Unsplash License)
