lasal.ai /OCR case study

A case study · 1914 to 2026

Machines that learned to read

A century of OCR as one river: from reading machines older than computers to 0.8B-parameter models that turn a page into JSON. Every breakthrough deleted one hand-built step. Follow the current; stop wherever a demo invites you in.

32breakthroughs
112years
6eras
10live visualisers

Highlighted words explain themselves: hover, focus or tap. The icon says what kind of idea it is: learningseeingsequencesarchitecturedatameasuringdocuments

00

1914 to 1989

Before learning

OCR is older than computers. For sixty years the trick was to make text easy for the machine (special fonts, magnetic ink, fixed forms) rather than make the machine good at text. Then came the first feature hierarchies that could learn.

1914OptophonePrinted letters turned into tones for blind readers.
1929Tauschek's reading machineA photo-sensor behind mechanical templates: a match when light passes through.
1951GISMORead typewritten letters aloud; led to the first OCR company in 1952.
1954Reader's DigestThe first business to install an OCR reader.
1958E-13B MICRMagnetic-ink digits for cheques. Still on every cheque.
1968OCR-A and OCR-BTypefaces designed for machines. OCR-B still runs along passport lines.
1976Kurzweil Reading MachineOmni-font OCR plus text-to-speech: any printed font, not one.

Scroll sideways · make the text easy for the machine, not the machine good at text

1980 · Kunihiko Fukushima · Biological Cybernetics

Neocognitron: feature cells, then shift-tolerant cells

stroke heresame C-cell firesA
1 cellper pattern, wherever it sits
Before
Pattern recognisers broke when the same shape appeared in a different position.
The breakthrough
Modelled on the visual cortex. S-cells detect local features with shared weights; C-cells pool nearby S-cells so a feature can move and the response stays. The pair repeats, each layer seeing larger patterns.
Result
Each final cell answered to one pattern wherever it appeared. The blueprint of every CNN, minus a way to train it end to end.
doi:10.1007/BF00344251
1989 · LeCun, Boser, Denker et al. (Bell Labs) · Neural Computation

Train the convolutional hierarchy with backprop

5x5 shared kernelfeature map
5.0%error on real zip codes
Before
Digit recognisers relied on hand-engineered preprocessing and features.
The breakthrough
Feed the raw image straight in. Constrain the network with local receptive fields and strided 5x5 kernels whose weights are shared, so 64,660 connections need only 9,760 free parameters. Train every layer with Backpropagation on 7,291 real zip-code digits.
Result
5.0% error on 2,007 test digits, running on a DSP at over 10 digits a second. The earliest real-world network trained end to end.
doi:10.1162/neco.1989.1.4.541
1985 to 1994 · Ray Smith (HP Labs Bristol) · open-sourced 2005, Google from 2006

Tesseract: spline baselines and an adaptive classifier

a quadratic spline per lineno de-skewing of the page
Top 31995 UNLV accuracy test
Before
OCR engines assumed straight, de-skewed lines and a fixed set of character models.
The breakthrough
Outlines of connected components are gathered into blobs, blobs into lines, and each line gets a quadratic spline baseline, so curved and skewed text needs no straightening. Words go through two passes: confident words from pass one train an Adaptive classifier on the fly, and pass two re-reads the unsure ones.
Result
Top three in the 1995 UNLV accuracy test. The adaptive classifier cuts errors 30 to 60% on long documents. The default free OCR engine for over a decade.
Live visualiser Tesseract Inside Out The real Tesseract 5 running in your browser, with its LSTM rebuilt in plain JavaScript so every gate can be inspected. 1,461,007 weightsclassic engine vs LSTMfine-tuning: CER 0.040 to 0.002 on clean lines An Overview of the Tesseract OCR Engine (ICDAR 2007)
01

1998

Recognising a symbol

One centred character, one label. The problem convolutional networks solved first, and the benchmark that made the field measurable.

1998 · LeCun, Bottou, Bengio, Haffner · Proceedings of the IEEE

LeNet-5: let the network learn its own features

7convolve, subsample, repeat
0.95%error on MNIST
Before
Recognisers were hand-designed feature extractors feeding a trainable classifier.
The breakthrough
5x5 convolutions with shared weights alternate with subsampling, about 60k parameters, all trained end to end by gradient descent. The forgotten half of the paper: a full cheque reader combining a segmenter, LeNet-5 and a language model, trained jointly.
Result
0.95% error on MNIST (0.8% with distortions). Deployed in NCR cheque readers, reading millions of US cheques.
doi:10.1109/5.726791
1998 to 2019 · 60,000 train / 10,000 test, 28x28 · the benchmark

MNIST: from 12% error to 0.13%, then solved

linear12%k-NN1.1%LeNet-50.95%SVM0.68%GPU nets0.23%ensemble0.13%test error, log scale
0.13%13 errors in 10,000
Before
No shared, clean benchmark to compare recognisers on.
The breakthrough
Linear classifier 12.0%, k-NN 1.1%, LeNet-5 0.95%, SVM 0.68%, a GPU committee of deep nets 0.23% (2012), and ensembles reach 0.13%: 13 mistakes in 10,000, mostly ambiguous or mislabelled digits.
Result
Past the estimated 0.2% human error rate in 2012. But MNIST hands the model a pre-cropped, pre-centred symbol: no detection, no sequence, no layout. Real text is none of those things.
Live visualiser MNIST Visualiser Draw a digit or a letter and watch a small CNN read it, layer by layer, feature map by feature map. 98.68% on MNIST test91.93% on EMNIST letters3-D embedding explorer MNIST database
02

2006 to 2022

Reading real text

Find text anywhere in an image, then read whole lines without cutting them into characters. The longest era, and the one every production OCR pipeline is still built from.

2006 · Graves, Fernandez, Gomez, Schmidhuber · ICML

CTC: train a sequence model without knowing the alignment

-CAT- C C - A A A - T -merge repeats, drop blanks: CAT
0frame-level alignment labels needed
Before
Training a sequence model needed a label for every frame, usually from an HMM.
The breakthrough
Add a blank symbol. The network emits one label per frame; collapse merges repeats and drops blanks, so '-CC-AAA-T-' becomes CAT. The loss sums the probability of every path that collapses to the target, computed with forward-backward dynamic programming.
Result
A BiLSTM with CTC beat HMM baselines on speech with no pre-segmentation. Still the decoding step of the fastest OCR recognisers in 2026.
doi:10.1145/1143844.1143891
2014 / 2016 · Jaderberg et al. (Oxford) / Gupta et al. · NIPS DL Workshop / CVPR

Synthetic data: render millions of labelled words

readfontreadborderreadcolourreadwarpreadblendthe rendered word is the label
9Mlabelled words, zero human annotation
Before
Real scene-text datasets held only a few thousand labelled words.
The breakthrough
Generate the labels instead. Synth90k renders words with random fonts, borders, colours and projective warps, blended onto natural images; the rendered word is the label. SynthText pastes text onto real scenes along predicted surfaces.
Result
Recognisers trained only on Synthetic data data reached state of the art on real benchmarks. MJSynth plus SynthText became the default training set for years.
arXiv 1406.2227
2015 · Shi, Bai, Yao (HUST) · TPAMI 2017

CRNN: read a whole word as one sequence, no character boxes

STATE-SS-TT-A--TTE-STATEBiLSTM, then CTC
8.3Mparameters, any word length
Before
Text was read character by character, or classified against a fixed word list.
The breakthrough
Convolutions turn the line into a feature map; each column becomes one time step. A BiLSTM reads the columns with context from both sides, and CTC turns per-column guesses into the string. Trained end to end from images and transcripts.
Result
8.3M parameters, any length, no segmentation. The reference architecture for line recognition: every later recogniser swaps one block.
Live visualiser CRNN Line Reader Write a whole line and see it read without being cut into characters: convolutions, slices, BiLSTM and CTC, aligned column by column. about 274k weightstrained on synthetic lines onlyhover one slice, see it everywhere arXiv 1507.05717
2018 · Ray Smith and contributors (Google) · Tesseract 4.0

Tesseract 4: read whole lines with an LSTM

reversed LSTMforward LSTM-EE--n--c--yy--c--l--o-
100+languages, Apache-2.0
Before
Tesseract 3 segmented and classified characters one by one; broken or joined glyphs caused errors.
The breakthrough
CRNN's idea brought to the most widely used open-source engine. Convolutions, a summarising LSTM that collapses the height, forward and reverse LSTMs along the line, then a softmax trained with CTC and a dictionary-guided Beam search.
Result
Default engine from 4.0, 100+ languages out of the box, still maintained in 2026 and the baseline newer engines are measured against.
Live visualiser Tesseract Inside Out The real Tesseract 5 running in your browser, with its LSTM rebuilt in plain JavaScript so every gate can be inspected. 1,461,007 weightsclassic engine vs LSTMfine-tuning: CER 0.040 to 0.002 on clean lines Tesseract documentation
2017 · Zhou, Yao et al. (Megvii) · CVPR

EAST: predict word geometry at every pixel, in one network

every pixel: score, 4 distances, angle
13.2 fpsone network, one merge step
Before
Text detection was a slow chain of proposals, filters and word splitting.
The breakthrough
A single fully convolutional network predicts, at every pixel, a text score plus the distances to the four box edges and an angle. One Non-maximum suppression step merges the overlapping boxes.
Result
F-score 0.78 on ICDAR 2015 at 13.2 frames a second. Weak spot: very long or curved text, which the receptive field cannot see end to end.
arXiv 1704.03155
2015 · Jaderberg, Simonyan, Zisserman, Kavukcuoglu (DeepMind) · NeurIPS

Spatial Transformer: a layer that learns to un-warp its input

texta learned warp straightens the input
0warp labels needed
Before
CNNs tolerated small shifts but not rotation, scale or warping.
The breakthrough
A drop-in, differentiable module. A localisation network predicts transform parameters, a grid generator maps the output grid back into the input, and a sampler reads it bilinearly. Gradients flow through all three.
Result
Trained with class labels only, it learns to crop, scale and straighten by itself.
arXiv 1506.02025
2018 · Shi, Yang, Wang, Lyu, Yao, Bai · IEEE TPAMI

ASTER: flatten curved text before reading it

CURVEDCURVEDthin-plate spline, then attention
20control points bend the word flat
Before
Recognisers read along a straight line, so curved and slanted text failed.
The breakthrough
A spatial transformer with a thin-plate spline places control points along the top and bottom of the text and flattens it. An Attention decoder then reads the straightened word.
Result
Large gains on curved and perspective benchmarks, and state of the art on regular text at the time.
doi:10.1109/TPAMI.2018.2848939
2019 · Baek, Lee, Han, Yun, Lee (NAVER Clova) · CVPR

CRAFT: detect characters and the links between them

CRAFTcharacter regionsand the links between them
2heatmaps: region and affinity
Before
Word boxes failed on curved text, and real datasets had no character labels.
The breakthrough
Predict two heatmaps: where each character's centre is, and where neighbouring characters join. Character labels for real images are bootstrapped from a model trained on synthetic data (Weak supervision).
Result
State of the art on six benchmarks, including curved Total-Text and CTW-1500.
arXiv 1904.01941
2019 · Baek, Kim, Lee et al. (NAVER Clova) · ICCV

What is wrong with scene text recognition? One protocol, four swappable stages

TPSnonetransformResNetVGGfeaturesBiLSTMnonesequenceAttnCTCpredictevery combination, one protocol
4stages, every combination tested
Before
Papers used different training data and test subsets, so results were not comparable.
The breakthrough
Split every recogniser into four stages (transformation, feature extraction, sequence modelling, prediction) and test every combination under one data and evaluation protocol.
Result
Much reported progress came from data and evaluation differences, not models. The open code became the standard baseline.
arXiv 1904.01906
2020 · Liao, Wan, Yao, Chen, Bai · AAAI

DBNet: learn the binarisation threshold end to end

probabilitylearned thresholdboxesP is all that runs at inference
62 fpsF-measure 82.8 on MSRA-TD500
Before
Segmentation detectors used a fixed, hand-tuned threshold outside training.
The breakthrough
Predict a text probability map and a threshold map, and combine them with a steep, differentiable step (differentiable binarisation), so the threshold is learned. At inference only the probability map is needed.
Result
State of the art on five benchmarks, 62 fps with a ResNet-18. The detector inside PaddleOCR to this day.
arXiv 1911.08947
2017 · Vaswani, Shazeer, Parmar et al. (Google) · NeurIPS

Transformer: attention alone, no recurrence

TheThetotaltotalisisduedueononreceiptreceiptevery token attends to every token
0recurrent steps
Before
Sequence models used recurrence, so training was serial and long-range context was hard to learn.
The breakthrough
Every Token looks at every other token through scaled dot-product attention, in parallel, with many heads at once. An Encoder builds context; a Decoder writes the output one token at a time while attending back to it.
Result
A new state of the art in translation at a fraction of the training cost. It became the architecture of every modern language and vision-language model.
arXiv 1706.03762
2020 · Dosovitskiy, Beyer, Kolesnikov et al. (Google) · ICLR 2021

ViT: an image is worth 16x16 words

image16 patch tokens, one sequence
88.55%on ImageNet
Before
Vision models were built from convolutions; attention was bolted onto CNNs.
The breakthrough
Cut the image into 16x16 patches, flatten each into a vector, add a position, and feed the sequence to a plain transformer Encoder.
Result
With enough pretraining data it matched or beat the best CNNs. With too little it trailed them: scale beats built-in bias.
arXiv 2010.11929
2021 · Li, Lv, Cui et al. (Microsoft) · AAAI 2023

TrOCR: a pretrained image transformer meets a text transformer

InvoiceTotalpatches in, word pieces out
2.89CER on handwriting
Before
Recognisers paired a CNN with an RNN and often needed a separate language model.
The breakthrough
A ViT encoder reads the line image as patches; a pretrained text transformer Decoder writes word pieces while cross-attending to them. Both halves start from pretrained weights.
Result
F1 96.58 on printed receipts and CER 2.89 on IAM handwriting with no external language model.
Live visualiser Transformer OCR Microsoft's TrOCR running on your device, with the encoder's attention, the decoder's cross-attention and every key and value laid out. 61.6M weights in the browser576 image patchesattention vs CTC, side by side arXiv 2109.10282
2022 · Du, Chen, Jia et al. (Fudan, Baidu) · IJCAI

SVTR: one visual model, no sequence model, still CTC

S V T Rlocal + global mixing, CTC in one shot
1 shotdecoding, no autoregression
Before
Accurate recognisers needed a slow sequence or language model on top of vision.
The breakthrough
Replace conv plus LSTM with a single vision transformer that mixes local and global patch context, then decode with CTC in one shot.
Result
Matches language-aware models on English, beats them on Chinese, and runs faster. PaddleOCR's recogniser descends from it.
arXiv 2205.00159
03

2017 to 2025

Understanding documents

The unit changes from a line to a whole page, and the output from a string to structure: LaTeX, HTML tables, Markdown, JSON. Position on the page starts to carry meaning, then models skip the OCR engine altogether.

2017 · Deng, Kanervisto, Ling, Rush (Harvard) · ICML

im2latex: generate markup from pixels with attention

x² + √yx^{2}+\sqrt{y}one LaTeX token at a time, attending anywhere
75%of formulas re-render exactly
Before
Maths OCR relied on hand-built grammars, and CTC assumed text runs strictly left to right.
The breakthrough
A CNN grid, a row encoder across it, and an Attention decoder that emits one LaTeX token at a time, looking wherever on the image it needs.
Result
Over 75% of test formulas re-render to the exact source image, against 15.6% for the classical system and 7.6% for a CTC model.
arXiv 1609.04938
2019 · Zhong, ShafieiBavani, Jimeno Yepes (IBM) · ECCV 2020

PubTabNet: tables as HTML trees

tabletrtrtdtdtdcompared as trees: TEDS
+9.7TEDS points
Before
Table recognition lacked large datasets, and its metrics ignored the table's tree structure.
The breakthrough
568k table images paired with HTML. An encoder with two decoders: one writes the structure tags, one fills each cell. A new metric, TEDS, compares tables as trees.
Result
9.7 points better than the previous best. TEDS became the standard table metric and a third of today's headline document benchmark.
arXiv 1911.10683
2022 · Nassar, Livathinos, Lysak, Staar (IBM Zurich) · CVPR

TableFormer: structure plus a box for every cell

structure tags plus a box per cell
96.75structure TEDS
Before
Image-to-HTML table models needed their own OCR decoder and gave no box per cell.
The breakthrough
A transformer decodes the structure tags while a second head predicts a bounding box for each cell, so cell text can be taken straight from the PDF or an OCR engine.
Result
Structure TEDS 96.75 on PubTabNet. The table model inside IBM's open-source Docling.
arXiv 2203.01017
2020 · Xu, Li, Cui et al. (Microsoft Research Asia) · KDD

LayoutLM: put 2-D position inside the language model

INVOICE(20,30)No 1042(150,30)Item(20,70)9.50(150,70)Total(110,108)19.00(170,108)'Total' bottom right means something
+8.6F1 on form understanding
Before
Language models read documents as flat text and ignored where words sit.
The breakthrough
Every word Embedding gets its box coordinates added. Pretrain with masked words on millions of scanned pages, so 'Total' at the bottom right of a page means something different from 'Total' in a paragraph.
Result
The first joint text-and-layout pretraining. Form understanding jumped from 70.7 to 79.3 F1.
arXiv 1912.13318
2022 · Kim, Hong, Yim et al. (NAVER) · ECCV

Donut: pixels in, JSON out, no OCR engine

{  "number": "1042",  "date": "2026-11-20",  "total": 19.00}no OCR engine in between
0OCR engines in the loop
Before
Document understanding depended on an OCR engine whose errors propagated.
The breakthrough
A Swin Encoder reads the page and a text Decoder writes the answer directly as structured tokens. Pretrained on a synthetic document generator (SynthDoG), so new languages need no real labels.
Result
State-of-the-art speed and accuracy on classification, receipt extraction and document VQA, about twice as fast as the OCR-based LayoutLMv2.
arXiv 2111.15664
2023 · Blecher, Cucurull, Scialom, Stojnic (Meta AI) · arXiv

Nougat: write a scientific page as Markdown and LaTeX

# Results- accuracy rose- cost fell$$E = mc^2$$headings, lists and equations from pixels
1 imageto a full Markdown page
Before
OCR recovered characters from papers but lost equations and structure.
The breakthrough
Donut's recipe trained on arXiv papers paired with their source, so the model writes headings, lists and equations from the image alone.
Result
Beat existing tools at converting papers, especially equations. Works on scans and books with no text layer.
arXiv 2308.13418
2024 · Wei, Liu, Chen et al. (StepFun, Megvii et al.) · arXiv

GOT-OCR 2.0: one model for every kind of character

Aa∑x♪▦⌬▁▃▆one 580M modeltext, maths, music, tables, molecules, charts
580Mparameters, one model
Before
OCR pipelines were brittle and general vision-language models were huge and poor at dense text.
The breakthrough
A 580M encoder-decoder trained on text, formulas, tables, sheet music, molecules and charts. Point at a region with a box or a colour and it reads just that.
Result
State-of-the-art page-level OCR in English and Chinese at 580M parameters. Later used as a teacher to distil PaddleOCR's detector.
arXiv 2409.01704
2025 · Poznanski, Soldaini, Lo et al. (Allen AI) · arXiv

olmOCR: anchor on the PDF's text, then reward passing unit tests

PDF text anchor✓table cell right✓order kept✓no invented text✓maths rendersreward = unit tests passed
< $200per million pages
Before
Frontier models extracted PDF text well but cost thousands per million pages.
The breakthrough
Give a 7B VLM the page image plus the PDF's own text layer as an anchor. Version 2 renders synthetic HTML pages so the ground truth is exact, then trains with Reinforcement learning on unit tests ('is this table cell right?').
Result
Under $200 per million pages, about 1/32 of the batch price of a frontier API. Version 2 among the best open models at release.
arXiv 2502.18443
2025 · Wei, Sun, Li (DeepSeek-AI) · arXiv

DeepSeek-OCR: a page image as compressed context

40 text tokens4 vision tokensabout 97% recovered under 10x
10xtext compressed into vision tokens
Before
A page of text costs thousands of language-model tokens; context is expensive.
The breakthrough
Turn the question around: how few vision tokens can carry a page of text? A compressing encoder squeezes the page; a small Mixture of experts decoder expands it back.
Result
About 97% precision when there are under 10 text tokens per vision token, about 60% at 20.
arXiv 2510.18234
04

2020 to 2026

Inside PaddleOCR, and what came after

PaddleOCR is the best map of the field because it absorbed most of the papers above: DB for detection, CRNN then SVTR for recognition, CTC for decoding, GOT as a teacher. Then a small vision-language model, then models that beat it.

2020 · Du, Li, Guo et al. (Baidu) · arXiv

PP-OCR: a 3.5M-parameter OCR system from 19 strategies

detectDBclassifyanglerecogniseCRNNdistil, prune, quantise, shrink3.5M parameters in total
3.5Mparameters, whole system
Before
Accurate OCR was large and GPU-bound; phones and CPUs needed tiny models.
The breakthrough
Three stages (DB detection, a direction classifier, CRNN recognition), each shrunk with a list of strategies: light backbones, Distillation, pruning, quantisation, and lots of data.
Result
3.5M parameters for 6,622 Chinese characters, trained on 17.9M recognition images.
arXiv 2009.09941
2025 to 2026 · Baidu PaddleOCR team · PP-OCRv5 / v6

PP-OCRv5/v6: train with two heads, ship one

backboneCTC: shipsattention: training onlytwo heads teach, one fast head runs
5.2xfaster on CPU (Baidu-reported)
Before
Attention decoders read better but are slow; CTC is fast but learns weaker context.
The breakthrough
An attention branch shares the backbone during training and teaches it sequence context, then is thrown away. Only the fast CTC head ships. The detector is distilled from GOT-OCR 2.0.
Result
1.5M to 34.5M parameters, 50 languages in one model; the medium tier is 5.2x faster than v5 on CPU. CTC also gives a per-line Confidence score that generative models do not.
Live visualiser PaddleOCR Viz PP-OCRv6 in the browser through all 21 stages of its forward pass, then PaddleOCR-VL's layout-first document parsing. dense page in about 4 s18,708-character dictionarydetection map you can scrub arXiv 2507.05595
2025 to 2026 · Cui, Sun et al. (Baidu) · arXiv

PaddleOCR-VL: layout first, then a 0.9B VLM per region

1234titletexttableformulalayout + reading order0.9B VLM per cropthe VLM never sees a whole page
96.33OmniDocBench v1.6
Before
Full-page models decode slowly and lose reading order or hallucinate on long pages.
The breakthrough
Split the job. A small Layout analysis model fixes regions and reading order; each crop goes to a 0.9B VLM that writes text, HTML tables, LaTeX or chart data. Version 1.6 added a hard-region data engine plus SFT and GRPO.
Result
96.33 on OmniDocBench v1.6 at 0.9B parameters, 109+ languages, Apache-2.0 (Baidu-reported).
Live visualiser PaddleOCR Viz PP-OCRv6 in the browser through all 21 stages of its forward pass, then PaddleOCR-VL's layout-first document parsing. dense page in about 4 s18,708-character dictionarydetection map you can scrub arXiv 2510.14528
2026 · Xia, Li, Chen et al. (ATH-MaaS) · arXiv

OvisOCR3: distil five specialist teachers into one 0.8B model

parseOCRfieldsfigurestablesfive 4B teachers, one 0.8B student
96.74from a 0.8B model
Before
Specialist models win single tasks, but running five models in production is costly.
The breakthrough
Train five 4B teachers (parsing, OCR, key fields, figures, cross-page tables), then On-policy distillation into one 0.8B student: the student writes, the teachers score its tokens. Finish with Reinforcement learning using a reward per domain.
Result
96.74 on OmniDocBench v1.6, first on OCRBench v2, best on four KIE benchmarks (self-reported). The architecture is a standard small VLM; the whole breakthrough is the training recipe.
arXiv 2610.33066

Where the leaders stood in November 2026

OmniDocBench v1.6: 1,651 pages; overall = mean of text, table TEDS and formula scores. * from the model card or paper, not yet on the board.

Youtu-Parsing-Omni 5B96.96 *
TeleOCR 1.2B96.91
OvisOCR3 0.8B96.74 *
OvisOCR2 0.8B96.47
PaddleOCR-VL-1.6 0.9B96.34
MinerU2.5-Pro 1.2B95.75
Gemini 3 Pro frontier92.91
GPT-5.2 frontier86.59

Axis starts at 85. Small specialists sit above frontier general models; no invoices or receipts are in this benchmark.

05

built in 2026 · built by lasal.ai

From reading to understanding

The papers explain how machines learned to read. These builds ask the next question: once a model can read, how do you turn a document into data you can trust? Every one of them runs live; open it and poke at the internals.

the old way · OCR plus a hand-written rulebook · Legacy IDP

The rule pyramid: accurate on what it has seen, brittle on what it has not

new layouts crack the base
43 to 59%on layouts it never saw
Before
Extraction meant OCR followed by rules: find 'Invoice No', look right, take the next token.
The breakthrough
TrOCR reads the page; 36 rules written across 23 versions turn lines into 11 fields. Each new supplier layout adds a rule on top of the last, so the Rulebook becomes a pyramid of patches.
Result
On layouts it was tuned on, accuracy climbs from 65.5% to 94.9%. On unseen layouts it stays between 43% and 59%, however many rules are added.
Live visualiser Legacy IDP How document extraction used to be built: OCR, then a growing hand-written rulebook. Watch the rule pyramid crack when new layouts arrive. 36 rules, 23 versionsseen layouts: 65.5% to 94.9%unseen layouts: stuck at 43 to 59%
inside the giants · Qwen 27B vs Gemma 26B mixture of experts · LLM-OCR Vis

What a large model actually does when it reads

dense QwenGemma MoEsame document, two ways of reading
8 / 128experts per token
Before
Large models read documents well, but nobody sees how.
The breakthrough
Both models read the same document while every layer is captured: the vision encoder, Attention from answer to page, the dense model's DeltaNet memory, and which of 128 experts fire for each token.
Result
Twelve prepared prompts and a side-by-side view of how a dense model and a mixture of experts form the same answer.
Live visualiser LLM-OCR Vis Two open large models read the same document: a dense Qwen with DeltaNet memory against a Gemma mixture of experts. Their internals, captured on a GPU. 64 vs 30 layers8 of 128 experts per token70 to 1,120 image tokens
inside the workhorse · Qwen3-VL-2B · Qwen3-VL Viz

Where each token looks on the page

token
28language layers, traced
Before
The small models that can run a whole document pipeline on one GPU were black boxes.
The breakthrough
Follow the image through 24 vision blocks, three DeepStack taps that inject vision features into the language layers, and a 2x2 merger, then watch each generated Token attend to its source on the page.
Result
A layer-by-layer explorer of a 2B vision-language model writing an extraction. Still being built.
Live visualiser · work in progress Qwen3-VL Viz Inside Qwen3-VL-2B: where every generated token looks on the page, through 24 vision blocks, DeepStack taps and 28 language layers. DeepStack taps at blocks 5, 11, 172x2 patch mergerwork in progress
the new way · Qwen3-VL-8B as a five-pass pipeline · SOTA IDP

One open model, five passes, checked in code

classifypartiesextractcheckcorrectplain codeQwen3-VL-8B, five small steps193 / 198 on unseen documents
193 / 198on unseen documents
Before
Rulebooks could not generalise; a single prompt to a model was good but not reliable enough.
The breakthrough
Classify the document, identify the parties, extract the fields, check them with plain code (totals, dates, formats), then let the model correct only what failed. Each pass is a small, inspectable step for a VLM.
Result
193 of 198 fields right on documents it had never seen, against 109 for the rulebook and 183 for one prompt to the same model.
Live visualiser SOTA IDP The modern replacement: one open 8B vision-language model run as a five-pass pipeline (classify, identify parties, extract, check in code, correct). 193 / 198 fields on unseen documentslegacy rules: 109 / 198every pass inspectable

Bleeding edge · an open question

what next? · an AI agent edits weights directly · Weight-Space Programming

Could you program a model by editing its weights?

3 edits, 0.013% of weightsno training run
0.013%of weights changed
Before
Changing a model's behaviour means collecting data and running fine-tuning. Every fix is another training run.
The breakthrough
Following Takehi et al. (2026), an AI agent inspects Qwen3-VL-2B and writes small, targeted weight edits by hand, keeps the ones that help on a dev set and checks them on a held-out test set. No training run, no new data.
Result
Three edits touching 0.013% of the weights raised dev-set fields from 355 to 382 of 495. On the held-out test set the gain is only 153 to 156 of 198, so the honest answer for now is: maybe. If it holds up, fixing a model could look more like editing code than retraining it.
Live visualiser · password protected Weight-Space Programming An AI agent edits a 2B vision-language model's weights directly, with no training run, to make it extract fields better. 3 edits, 0.013% of weightsdev fields 355 to 382 / 495plus a game played by tiny RL networks

The arc

Every breakthrough deleted one hand-built step

The pattern repeats for a century: something engineers designed by hand becomes something the network learns. The last row is different in kind. The architecture has largely converged (a vision encoder plus a small language decoder), and the progress is now in how the training data and rewards are built. Hover a row to light its stops; click to jump there.

When
Papers
Hand-built step removed
Replaced by
1980 to 98Neocognitron, LeCun 1989, LeNet-5Hand-designed featuresConvolutions learned by backprop
2006CTCFrame-level alignment labelsA loss summed over all alignments
1995 / 2018Tesseract, Tesseract 4Closed engines; per-character classifiersOpen-source OCR; an LSTM line reader
2014Synth90k / SynthTextExpensive human labelsMillions of rendered words
2015CRNNCharacter segmentationCNN + BiLSTM + CTC on whole lines
2017 to 20EAST, CRAFT, DBNetMulti-stage detection, fixed thresholdsDense maps with a learned threshold
2015 to 20STN, Transformer, ViTFixed geometry, recurrence, convolutionsLearned warps; attention for everything
2017 to 22im2latex, PubTabNet, TableFormerMaths grammars, table heuristicsImage-to-markup and structure decoders
2020LayoutLMLayout rules and templates2-D position inside the language model
2022 to 24Donut, Nougat, GOTThe OCR engine itselfImage-to-sequence generation
2025 to 26olmOCR 2, PaddleOCR-VL, OvisOCR3Supervised-only trainingData engines, distillation and RL

What it means in practice

Use each generation for what it is best at

The grounding and audit layer

Pipeline OCR

PP-OCRv6: DB + SVTR + CTC

Per-word boxes, per-line confidence, no invented text, CPU speed.

The structure layer

Document VLM

PaddleOCR-VL, OvisOCR3, TeleOCR

Full pages to Markdown or JSON, with tables, formulas and reading order, on one GPU.

The judgement layer

Frontier model

General-purpose large models

Reasoning over messy documents. Lower raw parsing scores, higher cost.

Guardrails for generative OCR

  • Compare character-level agreement between generative output and a pipeline OCR.
  • Require grounding coordinates for every extracted field.
  • Validate with business rules: totals, checksums, date formats.
  • Route low-agreement pages to a person. Handwriting is still the weak spot for every model.

Public benchmarks pick the candidates. Only your own documents, scored field by field, pick the winner.

The lab

Ten visualisers, one story

Each stop above links to a working, in-browser visualiser. Here they all are, in reading order.

01MNIST Visualiser Draw a digit or a letter and watch a small CNN read it, layer by layer, feature map by feature map.02CRNN Line Reader Write a whole line and see it read without being cut into characters: convolutions, slices, BiLSTM and CTC, aligned column by column.03Tesseract Inside Out The real Tesseract 5 running in your browser, with its LSTM rebuilt in plain JavaScript so every gate can be inspected.04Transformer OCR Microsoft's TrOCR running on your device, with the encoder's attention, the decoder's cross-attention and every key and value laid out.05PaddleOCR Viz PP-OCRv6 in the browser through all 21 stages of its forward pass, then PaddleOCR-VL's layout-first document parsing.06Legacy IDP How document extraction used to be built: OCR, then a growing hand-written rulebook. Watch the rule pyramid crack when new layouts arrive.07LLM-OCR Vis Two open large models read the same document: a dense Qwen with DeltaNet memory against a Gemma mixture of experts. Their internals, captured on a GPU.08Qwen3-VL Viz Inside Qwen3-VL-2B: where every generated token looks on the page, through 24 vision blocks, DeepStack taps and 28 language layers.09SOTA IDP The modern replacement: one open 8B vision-language model run as a five-pass pipeline (classify, identify parties, extract, check in code, correct).10password protectedWeight-Space Programming An AI agent edits a 2B vision-language model's weights directly, with no training run, to make it extract fields better.

Reading list

The papers behind it

  1. 1980Neocognitron
  2. 1989LeCun, zip codes
  3. 1998LeNet-5
  4. 2006CTC
  5. 2014Synth90k
  6. 2015CRNN
  7. 2015Spatial Transformer
  8. 2016SynthText
  9. 2017Transformer
  10. 2017im2latex
  11. 2017EAST
  12. 2018ASTER
  13. 2018Tesseract 4 LSTM
  14. 2019CRAFT
  15. 2019Baek et al.
  16. 2019PubLayNet
  17. 2019PubTabNet
  18. 2020DBNet
  19. 2020LayoutLM
  20. 2020ViT
  21. 2020PP-OCR
  22. 2021TrOCR
  23. 2022SVTR
  24. 2022TableFormer
  25. 2022DocLayNet
  26. 2022Donut
  27. 2023Nougat
  28. 2024GOT-OCR 2.0
  29. 2025olmOCR
  30. 2025olmOCR 2
  31. 2025DeepSeek-OCR
  32. 2025PP-OCRv5
  33. 2025PaddleOCR-VL
  34. 2026OvisOCR3

Figures are as reported in each paper. Most 2025 and 2026 benchmark numbers are self-reported by the model authors. All diagrams on this page are original drawings, not reproductions of the papers' figures.