[HN Gopher] Show HN: Using LLama2 to Correct OCR Errors
___________________________________________________________________
Show HN: Using LLama2 to Correct OCR Errors
I've been disappointed by the very poor quality of results that I
generally get when trying to run OCR on older scanned documents,
especially ones that are typewritten or otherwise have unusual or
irregular typography. I recently had the idea of using Llama2 to
use common sense reasoning and subject level expertise to correct
transcription errors in a "smart" way-- basically doing what a
human proofreader who is familiar with the topic might do. I came
up with the linked script that takes a PDF as input, runs Tesseract
on it to get an initial text extraction, and then feeds this
sentence-by-sentence to Llama2, first to correct mistakes, and then
again on the corrected text to format it as markdown where
possible. This was surprisingly easier than I initially expected
thanks to the very nice tooling now available in libraries such as
llama-cpp-python, langchain, and pytesseract. But the big issue I
was encountering was that Llama2 wasn't just correcting the text it
was given-- it was also hallucinating a LOT of totally new
sentences that didn't appear in the original text at all (some of
these new sentences used words which never appeared elsewhere in
the original text). I figured this would be pretty simple to
filter out using fuzzy string matching-- basically check all the
sentences in the LLM corrected text and filter out sentences that
are very different from any sentences in the original OCRed text.
To my surprise, this approach worked very poorly. In fact, lots of
other similar tweaks, including using bag-of-words and the spacy
NLP library in various ways (spacy worked very poorly in everything
I tried). Finally I realized that I had a good solution staring me
in the face: Llama2. I realized I could get sentence level vector
embeddings straight from Llama2 using langchain. So I did that,
getting embeddings for each sentence in the raw OCRed text and the
LLM corrected text, and then computed the cosine similarity of each
sentence in the LLM corrected text against all sentences in the raw
OCRed text. If no sentences match in the raw OCRed text, then that
sentence has a good chance of being hallucinated. In order to save
the user from having to experiment with various thresholds, I saved
the computed embeddings to an SQLite database so they only had to
be computed once, and then tried several thresholds, comparing the
length of the filtered LLM corrected text to the raw OCRed text; if
things worked right, these texts should be roughly the same length.
So as soon as the filtered length dips below the raw OCRed text
length, it backtracks and uses the previous threshold as the final
selected threshold. Anyway, if you have some very old scanned
documents laying around, you might try them out and see how well it
works for you. Do note that it's extremely slow, but you can leave
it overnight and maybe the next day you'll have your finished text,
which is better than nothing! I feel like this could be useful for
sites like the Internet Archive-- I've found their OCR results to
be extremely poor for older documents. I'm very open to any ideas
or suggestions you might have. I threw this together in a couple
days and know that it can certainly be improved in various ways.
One idea that I thought might be fun would be to make this work
with a Ray cluster, sending a different page of the document to
each of the workers in the cluster to do it all at the same time.
Author : eigenvalue
Score : 6 points
Date : 2023-08-02 19:53 UTC (3 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| version_five wrote:
| Very cool. I think this is an interesting benchmarking task for a
| language model (as well as the practical uses). I tried the same
| thing some time ago, just on a random snippet of tesseract OCR. I
| had a Vicuna model (I forget which) that failed miserably, and
| chat GPT did it flawlessly. I did not have any hallucination
| problem with chatGPT.
|
| It sounds from your writeup then like llama2 (which one) doesn't
| work well enough without some guardrails but it's possible to
| make it work? How would you rate the performance overall?
| eigenvalue wrote:
| I'd say that it does work pretty well. It could simply be that
| I'm sampling too much from the LLM which is causing it to
| hallucinate more than it should, hence why I needed to spend so
| much time filtering out the hallucinations. But I think the
| risk of "made up" stuff in what's supposed to be an accurate
| representation of a scanned document is big enough that you
| might always want to do something like that for quality control
| purposes, just to be on the safe side.
|
| I used the Llama2 13B Chat model ggml weights from TheBloke on
| Huggingface.
| version_five wrote:
| You should be able to build a text->image->corrupt with
| noise->tesseract pipeline to generate some synthetic
| supervised fine tuning examples (or just try corruption the
| text, I'm assuming putting it through tesseract makes more
| authentic corruption)
___________________________________________________________________
(page generated 2023-08-02 23:02 UTC)