One of the minor irritations in my life is that OCR does not consistently work for biological sequences (nucleic acids and proteins).
People think of OCR as a solved problem because AIs are very good at it, but the mistakes AIs make in human language are usually forgivable: maybe an O instead of a 0, or a hyphen instead of an en-dash.
Biological sequences have peculiar challenges: they can be thousands of characters or more; they are often repetitive and high entropy compared to human language; they are fairly often embedded in images, whether that is a scanned PDF, or a rasterized figure in a paper. Most importantly, a single typo can break the whole sequence.
Patents
Patents have some of the most useful biological sequences, and some of the worst OCR problems. Google Patents includes the reference/original PDF of the patent (the USPTO certifies PDFs), and an OCR'd html page, which can have errors. Most of the time, the PDF contains extractable text, and no OCR is necessary, but even then, the sequences often live in images.
I got an AI to examine a sample of 100 patent PDFs, with 25 patents drawn from each of four publication eras (before 2000, 2000–2009, 2010–2019, 2020 onward). 11/100 contained no embedded or extractable text on any page (i.e., all images). Surprisingly, all were published in the last five years (e.g., US11584788B2).
I tried to find some illustrative examples of failed OCR: in WO2017040932A1, the Q in EIVLTQSP is rendered as O for SEQ ID NO: 22; in US11965030B2, Gln is rendered as GIn in two occurrences, SEQ ID NOs: 15 and 19.
Patent WO2017040932A1 PDF (top) vs HTML (bottom). There are multiple OCR errors underlined in pink. This is a grainy image, but readable by a human.
In the process of this search, I was a bit surprised to find that in most cases of impossible DNA or protein sequences, the PDF itself contained the typo.
Patent US8110549 has an impossible codon (underlined in pink).
I tried to do a more comprehensive evaluation of the prevalence of such errors, but it would have taken a ton of manual review. I think it probably makes more sense to wait a year and get a smart AI to do it.
OCR
There are two main kinds of OCR: specialized OCR tools like hoary mainstay tesseract, which works well but is tuned for human language, and AI OCR tools (just asking ChatGPT, Claude, Gemini), which are smarter but have the tendencies of a human reader, and so can get confused or distracted.
Broadly, the most common failure modes I have seen for these are:
- specialized: mistaking Gln for GIn (not enough intelligence to understand that this is a sequence of amino acids)
- AI: miscounting or losing track of the number of repeated characters (homopolymers)
Benchmark
Last year, after failing to extract some not-especially challenging protein sequences from several PDFs—forcing me to spend hours manually typing them in—I got annoyed and decided to make a benchmark.
I chose six sequences. These were an attempt at a representative sample, and were not chosen to be especially challenging. I created a bio_ocr_minibenchmark repo to let me test out methods and see what works best.
The six sequences in the benchmark are not particularly challenging.
These are all monospace fonts, have a constrained alphabet, and clear, uncorrupted images. Yet I have tried every OCR system I could find over the last year and so far nothing has gotten all of them correct. Almost everyone is surprised this is still an issue!
There has been progress in just the past few weeks. Opus 5 and GPT-6 Astra are the best I have tested so far (Fable refuses any biological work, so I can't evaluate it). Specifically, Opus 5 gets 5/6 correct, failing on image 6, which is just some Gs and Cs I typed into Google Docs! It loses count and adds some extra Gs for no reason.
The benchmark includes specialized OCR tools and AI OCR tools. There has been steady improvement over time.
Conclusion
We do not yet have a robust way to extract biological sequences from images. This is surprising, and a pain.
I think we are still maybe a year or more from a fully trustworthy OCR system you don't have to double-check, which is crazy to say given the current capabilities of AI generally.