[HN Gopher] DeepSeek OCR
___________________________________________________________________
DeepSeek OCR
Author : pierre
Score : 842 points
Date : 2025-10-20 06:26 UTC (16 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| farseer wrote:
| How good is this compared to most commercial OCR software?
| ozim wrote:
| Any vision model is better than commercial OCR software.
| Etheryte wrote:
| I'm not really sure if that's an accurate summary of the
| state of the art, [0] is a better overview. In short, SOTA
| multi-modal LLMs are the best option for handwriting, nearly
| anything is good at printed text, for printed media,
| specialty models from hyperscalers are slightly better than
| multi-modal LLMs.
|
| [0] https://research.aimultiple.com/ocr-accuracy/
| ozim wrote:
| I see it confirms what I wrote state of art is "not using
| tessaract anymore" and I think bunch of commercial
| solutions are stuck with tessaract.
| ares623 wrote:
| I assume Tesseract has the advantage of being able to
| give a confidence score?
| dragonwriter wrote:
| Since "commercial OCR software" includes VLM-based commercial
| offerings, that's clearly not correct.
| empressplay wrote:
| This could be great for extracting text from old magazines;
| traditional OCR gives you a bit of a mess you have to clean up,
| but this looks like it can properly identify columns and track
| the flow accurately (and extract images!) It appears it can
| convert magazine layouts to markdown too
| piker wrote:
| This looks really cool for prototyping and playing around.
|
| It seems to me though if one is building a modern application
| that needs to get image segmentation and/or text recognition
| right there are better APIs available than natural language? It
| seems like a lot of effort to make a production-scale CV
| application to weigh it down with all of an LLM's shortcomings.
| Not a field I'm familiar with but I would assume that this
| doesn't produce state of the art results--that would change the
| analysis.
| randomNumber7 wrote:
| Imagine you build an image segmentation model for a e.g.
| specific industrial application.
|
| With this LLM approach you can at least create your training
| data from the raw images with natural language.
| piker wrote:
| That does make sense
| CheeseFromLidl wrote:
| As a hobby photographer, I organise everything for speedy
| retrieval but this would be amazing to search my collection.
| krackers wrote:
| The paper is more interesting than just another VLM for OCR, they
| start talking about compression and stuff. E.g. there is this
| quote
|
| >Our work represents an initial exploration into the boundaries
| of vision-text compression, investigating how many vision tokens
| are required to decode text tokens. The preliminary results are
| encouraging: DeepSeek-OCR achieves near-lossless OCR compression
| at approximately 10x ratios, while 20x compression still retains
| 60% accuracy.
|
| (I guess you could say a picture token is worth 10 textual
| tokens...)
|
| Could someone explain to a noob what the information-theoretic
| intuition is here? Why does this work, is it that text tokens are
| still too "granular"/repetitive and don't come close to the ideal
| entropy coding? Or is switching to vision tokens escaping the
| limitation of working "one word-ish at a time", allowing you to
| get closer to entropy (similar to the way that arithmetic
| encoding does compared to huffman codes)?
|
| And then they start talking about handling long-context by
| literally(?) downscaling images, forming a correspondence between
| information loss in the textual domain and the image domain.
| looobay wrote:
| LLMs are compute heavy with quadratic scaling (in compute) per
| tokens. They are trying to compress text tokens into visual
| tokens with their VLM.
|
| Maybe they would render texts to an image before tokenizing to
| reduce the compute cost.
| krackers wrote:
| But naively wouldn't you expect the representation of a piece
| of text in terms of vision tokens to be roughly the same
| number of bits (or more) than the representation as textual
| token? You're changing representation sure, but that by
| itself doesn't give you any compute advantages unless there
| is some sparsity/compressability you can take advantage of in
| the domain you transform to right?
|
| So I guess my question is where is the juice being squeezed
| from, why does the vision token representation end up being
| more efficient than text tokens.
| looobay wrote:
| Vision tokens are a good compression medium because with
| one vision token you have one vector of N elements, but
| with textual tokens you have M vectors of N elements,
| because one vision token represent multiple pixels (and
| possibly multiple words). This is why its a good
| compression medium for compute.
|
| It will never be as precise as textual tokens but it can be
| really good as they show in the paper.
| krackers wrote:
| >with one vision token you have one vector of N elements,
| but with textual tokens you have M vectors of N elements
|
| Each vision token represents a 16x16 patch, but to fully
| cover a word you might need multiple vision tokens. So
| assuming that the embedding size of the vision token and
| text token is the same `d` (which I think has to be the
| case for multimodal models), then wouldn't the fair
| comparison be `x * d` elements for a sentence in terms of
| vision tokens, and `y * d` for the same sentence in terms
| of text tokens? I don't see how you could see a priori
| that x << y (especially by a factor of 10 as quoted in
| the paper).
|
| That said, if I do experimentally try this by shrinking
| this very comment down to the smallest font size I can
| read it at, then seeing how many 16x16 tokens it takes,
| you can fit more text than I expected in each "vision
| token". So I can maybe buy that x is at least not greater
| than y. But it can't be as simple as "each vision token
| can cover more text", since that only enables better
| compression if the encoder can actually uncover some sort
| of redundancy within each token. (And presumably the type
| of redundancy it uncovers probably isn't something that
| "classical" compression techniques can exploit, otherwise
| it seems like it would have been tried by now?).
| looobay wrote:
| You should read the 6th page of the paper (and page 5 for
| architecture breakdown), they show that they are
| compressing the vision tokens with convolution to keep a
| strong semantic understanding and keep a small amount of
| tokens.
|
| But I think it's still experimentall.
| imjonse wrote:
| I wonder if text written using chinese characters is more
| compatible with such vision centric compression than latin
| text.
| f33d5173 wrote:
| Vision is how humans see text. So text must have built in
| adaptations to protect from visual noise. For example, two
| words that look similar must never appear in similar
| contexts, or else they would be conflated. Hence we can
| safely reduce such words to the same token. Or something
| like that.
| ffsm8 wrote:
| Is that really factual/true?
|
| Lots of words have multiple meanings and can mean
| different things even if used in the same
| sentence/context just from the interpretation of the
| person reading it.
|
| Heck, it'd argue that most (not all) dayjob conflicts are
| down to such differences in interpretation
| /miscommunications
| fxtentacle wrote:
| That also works purely on text and it's the trick I used
| in my German speech recognition engine (
| https://arxiv.org/abs/2206.12693 ).
|
| "I'm studying at Oxford Univ" has basically no loss in
| meaning even though "University" was truncated to less
| than half its characters.
| UltraSane wrote:
| This is like how many CLIs accept the shortest unique
| version of commands.
| numpad0 wrote:
| just a hunch but like, from something to do with Unicode?
| psb217 wrote:
| The trick is that the vision tokens are continuous valued
| vectors, while the text tokens are elements from a small
| discrete set (which are converted into continuous valued
| vectors by a lookup table). So, vision tokens can convey
| significantly more bits per token than text tokens. This
| allows them to pack the content of multiple text tokens
| into a single vision token.
| HarHarVeryFunny wrote:
| A text token generally represents a portion of a single
| word, while a vision token represents a portion of the
| entire page, which may include multiple words. This is
| where the "compression factor" comes from.
|
| The number of bits to represent a text or vision token is
| the same, since they are both represented as embeddings of
| a fixed number of dimensions defined by the Transformer
| (maybe a few thousand for a large SOTA model).
|
| Whether a vision token actually contains enough information
| to accurately extract (OCR) all the text data from that
| portion of the image is going to depend on how many pixels
| that vision token represents and how many words were
| present in that area of the image. It's just like
| considering images of the same page of text at different
| resolutions - a 1024x1024 image vs a 64x64 one, etc. As the
| resolution decreases so will OCR accuracy. At some point
| the resolution is insufficient and the words become a
| blurry mess and OCR accuracy suffers.
|
| This is what DeepSeek are reporting - OCR accuracy if you
| try to use a single vision token to represent, say, 10 text
| tokens, vs 20 text tokens. The vision token may have enough
| resolution to represent 10 tokens well, but not enough for
| 20.
| runeblaze wrote:
| each text token is often subword unit, but in VLMs the visual
| tokens are in semantic space. Semantic space obviously
| compresses much more than subword slices.
|
| disclaimer: not expert, on top of my head
| miki123211 wrote:
| Text tokens are quantized and represent subword units, vision
| tokens only exist in the embedding space.
|
| The way text tokenization works in LLMs is that you have a
| "lookup table" of (small) token ids to (large) vector
| embeddings. To pass text to the LLM, you split it at token
| boundaries, convert strings to token ids, and then construct
| the "context", a matrix where each row is a vector taken from
| that lookup table.
|
| Transmitting text token sequences can be relatively efficient,
| you just transmit the token IDs themselves[1]. They're small
| integers (~100k possible token ids is typical for large
| models). Transmitting the actual embeddings matrix would be far
| less efficient, as embeddings often consist of thousands of
| floating point numbers.
|
| Images are encoded differently. After some basic preprocessing,
| image data is passed straight to a neural- network-based image
| encoder. That encoder encodes the image into vectors, which are
| then appended to the context. There are no token ids, there's
| no lookup table, we go straight from image data to token
| embeddings.
|
| This means transmitting image tokens cannot be done as
| efficiently, as you'd have to transmit the embeddings
| themselves. Even though an image is encoded in fewer tokens,
| the most efficient representation of those tokens takes more
| bytes.
|
| You can think of a text token as an integer between 0 and n,
| which we know how to map to a vector. This means you have `n`
| possible choices of tokens. In contrast, an image token is an
| array of m floating point numbers (the vector itself), each of
| which can take on many possible values. This means the "token
| space" of vision tokens is actually much larger.
|
| There's also the issue of patterns. Text tokens correspond
| directly to a contiguous span of UTF-8 bytes, and most
| tokenizers won't create tokens that span word boundaries. This
| means they can't encode global patterns efficiently. You can't
| have a "Hamlet's monologue" or "the text that follows is in
| Spanish" token.
| rco8786 wrote:
| Great explanation, thanks. I was surprised to hear that
| models still only work with ~100k tokens, but after giving it
| some thought it makes sense. There's only so many
| words/subword units that get used in any given language. The
| entropy comes from all the billions of different ways those
| subwords can be _ordered_.
| freeqaz wrote:
| There is also a tradeoff between different vocabulary sizes
| (how many entries exist in the token -> embedding lookup
| table) that inform the current shape of tokenizers and
| LLMs. (Below is my semi-armchair stance, but you can read
| more in depth here[0][1].)
|
| If you tokenized at the character level ('a' -> embedding)
| then your vocabulary size would be small, but you'd have
| more tokens required to represent most content. (And
| context scales non-linearly, iirc, like n^3) This would
| also be a bit more 'fuzzy' in terms of teaching the LLM to
| understand what a specific token should 'mean'. The letter
| 'a' appears in a _lot_ of different words, and it's more
| ambiguous for the LLM.
|
| On the flip side: What if you had one entry in the
| tokenizer's vocabulary for each word that existed? Well,
| it'd be far more than the ~100k entries used by popular
| LLMs, and that has some computational tradeoffs like when
| you calculate the probability of each 'next' token via
| softmax, you'd have to run that for each token, as well as
| increasing the size of certain layers within the LLM (more
| memory + compute required for each token, basically).
|
| Additionally, you run into a new problem: 'Rare Tokens'.
| Basically, if you have infinite tokens, you'll run into
| specific tokens that only appear a handful of times in the
| training data and the model is never able to fully imbue
| the tokens with enough meaning for them to _help_ the model
| during inference. (A specific example being somebody's
| username on the internet.)
|
| Fun fact: These rare tokens, often called 'Glitch
| Tokens'[2], have been used for all sorts of shenanigans[3]
| as humans learn to break these models. (This is my interest
| in this as somebody who works in AI security)
|
| As LLMs have improved, models have pushed towards the
| largest vocabulary they can get away with without hurting
| performance. This is about where my knowledge on the
| subject ends, but there have been many analyses done to try
| to compute the optimal vocabulary size. (See the links
| below)
|
| One area that I have been spending a lot of time thinking
| about is what Tokenization looks like if we start trying to
| represent 'higher order' concepts without using human
| vocabulary for them. One example being: Tokenizing on LLVM
| bytecode (to represent code more 'densely' than UTF-8) or
| directly against the final layers of state in a small LLM
| (trying to use a small LLM to 'grok' the meaning and hoist
| it into a more dense, almost compressed latent space that
| the large LLM can understand).
|
| It would be cool if Claude Code, when it's talking to the
| big, non-local model, was able to make an MCP call to a
| model running on your laptop to say 'hey, go through all of
| the code and give me the general vibe of each file, then
| append those tokens to the conversation'. It'd be a lot
| fewer tokens than just directly uploading all of the code,
| and it _feels_ like it would be better than uploading
| chunks of code based on regex like it does today...
|
| This immediately makes the model's inner state (even more)
| opaque to outside analysis though. e.g., like why using
| gRPC as the protocol for your JavaScript front-end sucks:
| Humans can't debug it anymore without other tooling. JSON
| is verbose as hell, but it's simple and I can debug my REST
| API with just network inspector. I don't need access to the
| underlying Protobuf files to understand what each byte
| means in my gRPC messages. That's a nice property to have
| when reviewing my ChatGPT logs too :P
|
| Exciting times!
|
| 0: https://www.rohan-paul.com/p/tutorial-balancing-
| vocabulary-s...
|
| 1: https://arxiv.org/html/2407.13623v1
|
| 2: https://en.wikipedia.org/wiki/Glitch_token
|
| 3: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidg
| oldm...
| rco8786 wrote:
| Again, super interesting thanks!
|
| > One area that I have been spending a lot of time
| thinking about is what Tokenization looks like if we
| start trying to represent 'higher order' concepts without
| using human vocabulary for them. One example being:
| Tokenizing on LLVM bytecode (to represent code more
| 'densely' than UTF-8)
|
| I've had similar ideas in the past. High level languages
| that humans write are designed for humans. What does an
| "LLM native" programming language look like? And, to your
| point about protobufs vs JSON, how does a human debug it
| when the LLM gets stuck?
|
| > It would be cool if Claude Code, when it's talking to
| the big, non-local model, was able to make an MCP call to
| a model running on your laptop to say 'hey, go through
| all of the code and give me the general vibe of each
| file, then append those tokens to the conversation'. It'd
| be a lot fewer tokens than just directly uploading all of
| the code, and it _feels_ like it would be better than
| uploading chunks of code based on regex like it does
| today...
|
| That's basically the strategy for Claude's new "Skills"
| feature, just in a more dynamic/AI driven way. Claude
| will do semantic search through YAML frontmatter to
| determine what skill might be useful in a given context,
| then load that entire skill file into context to execute
| it. Your idea here is similar, use a small local model to
| summarize each file (basically dynamically generate that
| YAML front matter), feed those into the larger model's
| context, and then it can choose which file(s) it cares
| about based on that.
| jerf wrote:
| Textual language is really, really amazing if you sit down
| and think about what it does versus the resources it
| consumes to do it.
|
| It's a common pasttime for programmers to claim that our
| textual programming languages are just terrible and need to
| be replaced somehow with something visual, but I think this
| very often comes from a place of not understanding just how
| amazing textual languages are. Not they couldn't possibly
| be improved by something in at least some domains, and
| there are after all some successful niches for visual
| languages, but I think if you set out to wholesale replace
| textual languages without an understanding of and
| appreciation for the impressive nature of the competition
| they offer you're setting yourself up to fail.
| jph00 wrote:
| Actually there are VAEs which use a codebook approach to
| creating discrete tokens instead of float vectors. There has
| been some success in that direction in diffusion models for
| instance.
| ttul wrote:
| This is a great summary. If you think about it a bit, text is
| an expanded representation of concepts meant for display on a
| two-dimensional surface that can then be read back by human
| eyes; our brains convert the two-dimensional information into
| concepts again.
|
| So to me it's not a surprise that you can transform the two-
| dimensional representation of the same information into
| concepts again without losing much.
|
| The paper talks about using this approach to generate large
| amounts of LLM training data rapidly. That's intriguing. It
| suggests that one of the best ways of training models on a
| wide variety of input data with very long context is to
| provide it with an image representation instead of text
| tokens.
| miki123211 wrote:
| Text is actually one-dimensional, _writing_ is two-
| dimensional.
|
| To a pure LLM, characters 15 and 16 at line 1 are
| considered adjacent, but there's no relationship between
| character 15 of line 1 and character 15 of line 2.
|
| A vision model (which considers text as squiggles, not UTF8
| codepoints), such a relationship does exist.
| isaacfung wrote:
| Some models use vector quantized variational autoencoders to
| discretize images into sequences of discrete symbols from a
| fixed codebook.
|
| https://grok.com/share/bGVnYWN5LWNvcHk%3D_572b4955-6265-4210.
| ..
| lubesGordi wrote:
| So in terms of OCR, does the neural network 'map' the words
| into an embedding directly, or is it getting a bunch of words
| like "Hamlet's monologue" and mapping that to an embedding?
| Basically what I'm asking is if the neural network image
| encoder is essentially doing OCR 'internally' when it is
| coming up with the embedding (if that makes any sense).
| storus wrote:
| That's not really true, the latest autoregressive image
| models create a codebook of patches that are then encoded as
| tokens and image is assembled out of them.
| HarHarVeryFunny wrote:
| I don't know if there is any common practice among multi-modal
| input "LLM"s as to how they encode image inputs - convert them
| into "vision tokens", but it's basically going to come down to
| splitting the image into a grid of regions and encoding those.
|
| I'm not sure there's any information theoretic intuition to be
| had with DeepSeek's experiments - it seems to be more about
| what's the lowest resolution image resolution/grid you can get
| away with and still capture enough image detail to be able to
| accurately perform OCR on it.
|
| It'd be cool if Karpathy would extend his NanoChat to be multi-
| modal to spread the knowledge of how this is typically done.
| ssivark wrote:
| Surely the appropriate ratio depends on the resolution of each
| character, relative to the size of the vision token patch? That
| is the only way the number of text tokens needed to describe
| the output of OCR can be independent of the resolution of the
| image (as it should).
| bugglebeetle wrote:
| Looks great, but looking at the benchmark, can't help but think
| about how crazy good dots-ocr is as a model. Too bad they're not
| as open as the Deepseek team because its so crazy good and would
| love to know how it was trained.
| rfoo wrote:
| If you look you'd notice that it's the same Haoran Wei behind
| DeepSeek-OCR and GOT-OCR2.0 :p
| bugglebeetle wrote:
| Oh you're right! Good catch!
| bethekind wrote:
| Did we read the same graph? DeepSeek Gundam 200 dpi appeared to
| get similar perf as dots-ocr, but with less tokens needed. The
| x axis is inverted, descending with distance from the origin.
| k_sze wrote:
| It's interesting how they use "Gundam" in their variant names. I
| gather that Gundam-M and Gundam are their most powerful ones.
| daemonologist wrote:
| I think maybe to distinguish their dynamic resolution approach
| from the t-shirt sizes, which have a fixed input. (Although I
| don't know why "Gundam")
| brightUiso wrote:
| Please a bit of education, what does it do?
| yoran wrote:
| How does an LLM approach to OCR compare to say Azure AI Document
| Intelligence (https://learn.microsoft.com/en-us/azure/ai-
| services/document...) or Google's Vision API
| (https://cloud.google.com/vision?hl=en)?
| sandblast wrote:
| Not sure why you're being downvoted, I'm also curious.
| ozgune wrote:
| OmniAI has a benchmark that companies LLMs to cloud OCR
| services.
|
| https://getomni.ai/blog/ocr-benchmark (Feb 2025)
|
| Please note that LLMs progressed at a rapid pace since Feb. We
| see much better results with the Qwen3-VL family, particularly
| Qwen3-VL-235B-A22B-Instruct for our use-case.
| CaptainOfCoit wrote:
| Magistral-Small-2509 is pretty neat as well for its size, has
| reasoning + multimodality, which helps in some cases where
| context isn't immediately clear, or there are few missing
| spots.
| cheema33 wrote:
| Omni OCR team says that according to their own benchmark, the
| best OCR is the Omni OCR. I am quite surprised.
| numpad0 wrote:
| Classical OCR still probably make undesirable su6stitutions in
| CJK from there being far too many of similar ones, even some
| absurd ones that are only distinguishable under microscope or
| by looking at binary representations. LLMs are better
| constrained to valid sequences of characters, and so they would
| be more accurate.
|
| Or at least that kind of thing would motivate them to re-
| implement OCR with LLM.
| fluoridation wrote:
| Huh... Would it work to have some kind of error checking
| model that corrected common OCR errors? That seems like it
| should be relatively easy.
| colonCapitalDee wrote:
| It's harder then it first seems. The root problem is that
| for text like "hallo", correcting to "hello" may be fixing
| an error or introducing an error. In general, the more
| aggressive your error correction, the more errors you
| inadvertently introduce. You can try and make a judgement
| based on context ("hallo, how are you?"), which certainly
| helps, but it's only a mitigation. Light error correction
| is common and effective, but you can't push it to a full
| solution. The only way to fully solve this problem is to
| look at the entire document at once so you have maximum
| context available, and this is what non-traditional OCR
| attempts to do.
| fluoridation wrote:
| Okay, but there way more common errors that should be
| easy to fix. "He11o", "Emest Herningway", incorrect
| diacritics like the other person mentioned, etc.
| make3 wrote:
| aren't all of these multimodal LLM approaches, just open vs
| closed ones
| daemonologist wrote:
| My base expectation is that the proprietary OCR models will
| continue to win on real-world documents, and my guess is that
| this is because they have access to a lot of good private
| training data. These public models are trained on arxiv and
| e-books and stuff, which doesn't necessarily translate to
| typical business documents.
|
| As mentioned though, the LLMs are usually better at avoiding
| character substitutions, but worse at consistency across the
| entire page. (Just like a non-OCR LLM, they can and will go
| completely off the rails.)
| stopyellingatme wrote:
| Not sure about the others but we use Azure AI Document
| Intelligence and its working well for our resume parsing
| system. Took a good bit of tuning but we havent had to touch it
| for almost a year now.
| CloseChoice wrote:
| It's deepseek so one can expect an open-source license but for
| anyone (like me) who wants to see that explicitly, since it's not
| obvious in the GitHub repo: https://huggingface.co/deepseek-
| ai/DeepSeek-OCR/blob/main/LI...
|
| TLDR: It's MIT licensed
| AndroTux wrote:
| > since it's not obvious in the GitHub repo
|
| Literally says MIT license on the right sidebar and in the
| readme tab and in the file called LICENSE
| maxloh wrote:
| Model weights are MIT too: https://huggingface.co/deepseek-
| ai/DeepSeek-OCR/blob/main/LI...
| x______________ wrote:
| >Xian Tian Xia Zhi You Er You
|
| How is this an example of a prompt?
|
| Google translated this to "Worry about the world first" while
| Bing says "Worry before the worries of the world."
|
| Can anyone shed some light on this saying or why it's in the
| article?
| fspeech wrote:
| Google is closer. This is from a famous essay expressing tbe
| author's desire to bear the burden for the world. Essay is Yue
| Yang Lou Ji by Fan Zhong Yan in year 1046
| https://zh.wikisource.org/zh-hans/%E5%B2%B3%E9%99%BD%E6%A8%9...
| SequoiaHope wrote:
| Ask a language model - ChatGPT says it's a line from a famous
| poem "Memorial to Yueyang Tower" which expresses the Confucian
| ideal of selfless concern for people and society.
| raincole wrote:
| It's a very famous (classical) Chinese phrase.
|
| Both translations don't catch the meaning well though. It
| means: "worry before the rest of the world (notice that they
| have something to) worry." The next part is Hou Tian Xia Zhi Le
| Er Le ("be happy only after the rest of the world is happy.")
|
| I don't know why it's a prompt example.
| jdthedisciple wrote:
| Sibling comment has the second part as
|
| Hou Tian Xia Zhi Le Er Le
|
| which one is correct?
| raincole wrote:
| Traditional vs Simplified Chinese.
|
| There are two (modern) "spellings" of written Chinese.
| Basically colour vs color.
| Y_Y wrote:
| It depends on who you think is the rightful successor to
| the Qing dynasty
| gudzpoz wrote:
| This clause is usually used together with the next sentence in
| the original poem:
|
| > Xian Tian Xia Zhi You Er You ,Hou Tian Xia Zhi Le Er Le
|
| > (put the world's worries before yours, and put your happiness
| after the world's) > edit: this translation is wrong, and
| raincole has a definitely better translation
|
| Since the model is a language model, they probably use this to
| demonstrate the model's language capabilities - the model
| should be able to complete the whole sentence pair. The paper
| also mentions this:
|
| > To ensure the model's language capabilities, we introduced
| 10% of in-house text-only pretrain data.
|
| So I believe it is just a text-only demonstration.
| jdthedisciple wrote:
| Sibling comment has the second part as
|
| Hou Tian Xia Zhi Le Er Le
|
| Which one is correct?
| numpad0 wrote:
| a) Hou Tian Xia Zhi Le Er Le b) Hou Tian Xia Zhi Le
| Er Le c) Hou Tian Xia Zhi Le Er Le
|
| a) is clearly Simplified Chinese from a sibling comment, b)
| is Traditional copied from your comment, and c) is as I
| just typed in my own language. Unicode Hanzi/Kanji are a
| mess and there are characters same or different, in
| appearance or in binary, depending on intended variants,
| languages, fonts, systems, keyboard, distance between Earth
| and Alpha Centauri, etc.
| jdthedisciple wrote:
| Fascinating! That's exactly why I asked, so thank you.
|
| Do people usually recognize all variants as valid and
| legible? Or does any particular set of letters/symbols
| prevail in practice?
| hank2000 wrote:
| Very location dependent. But when you learn to write the
| characters you understand the variants differently. They
| look like random strokes to an untrained eye. But they're
| not. I'm not sure if that makes sense.
|
| Take a lowercase a in English for example. This font
| writes it differently than a child. Or in cursive. Or
| probably than you would write it. But you recognize all
| of them and don't really think about it.
| numpad0 wrote:
| Traditional kinds are usually recognizable, but I'd be
| unsure or straight up wrong about most Simplified
| versions. Overall proportions and small details often
| feel "wrong" for both as well due to cultures converging
| at different points.
| ellisd wrote:
| The paper makes no mention of Anna's Archive. I wouldn't be
| surprised if DeepSeek took advantage of Anna's offer granting OCR
| researchers access to their 7.5 million (350 TB) Chinese non-
| fiction collection ... which is bigger than Library Genesis.
|
| https://annas-archive.org/blog/duxiu-exclusive.html
| throawayonthe wrote:
| hahaha also immediately thought of this, wonder when the ocr'd
| dataset would be getting released
| singularfutur wrote:
| Yes it means they will never release their dataset :(
| ikamm wrote:
| Why do they need to grant access for people to use copies of
| books they don't own?
| JohnLocke4 wrote:
| Not to rationalize it, but it appears that they're
| gatekeeping the dataset to get access to the OCR-scans from
| the people they choose to share it with. This is to improve
| their existing service by making the content of books (and
| not just their title/tags) searchable.
|
| As per the blog post: >What does Anna's Archive get out of
| it? Full-text search of the books for its users.
| ikamm wrote:
| Fair enough, it just seems like they're painting an even
| bigger target on their backs by restricting access to
| copyrighted material they don't own the rights to
| dev1ycan wrote:
| Oh great so now Anna's archive will get taken down as well by
| another trash LLM provider abusing repositories that students
| and researchers use, META torrenting 70TB from library genesis
| wasn't enough
| c0balt wrote:
| It appears this is an active offer from Anna's archive, so
| presumably they can handle the load and are able to satisfy
| the request safely.
| sigmoid10 wrote:
| Seems like they are doing fine:
|
| https://open-slum.org
| bluecoconut wrote:
| Previous paper from DeepSeek has mentioned Anna's Archive.
|
| > We cleaned 860K English and 180K Chinese e-books from Anna's
| Archive (Anna's Archive, 2024) alongside millions of K-12
| education exam questions. https://arxiv.org/abs/2403.05525
| DeepSeek-VL paper
| mrasong wrote:
| Kinda reminds me of PaddleOCR.
|
| Would be awesome if DeepSeek OCR could be integrated into a
| mobile app someday. That'd make OCR way more convenient!
| pzo wrote:
| iOS already have on device both text detector and document
| scanner in apple Vision API. Hard to say how good are they
| compared to LLM based solutions. Similarly google had MLKit
| with OCR working on devices also for many years.
| pietz wrote:
| My impression is that OCR is basically solved at this point.
|
| The OmniAI benchmark that's also referenced here wasn't updated
| with new models since February 2025. I assume that's because
| general purpose LLMs have gotten better at OCR than their own OCR
| product.
|
| I've been able to solve a broad range of OCR tasks by simply
| sending each page as an image to Gemini 2.5 Flash Lite and asking
| it nicely to extract the content in Markdown under some
| additional formatting instructions. That will cost you around
| $0.20 for 1000 pages in batch mode and the results have been
| great.
|
| I'd be interested to hear where OCR still struggles today.
| kbumsik wrote:
| > My impression is that OCR is basically solved at this point.
|
| Not really in practice to me. Especially they still struggle
| with Table format detection.
| coulix wrote:
| This.
|
| Any complex parent table span cell relationship still has low
| accuracy.
|
| Try the reverse, take a complex picture table and ask
| Chatgpt5, claude Opus 3.1, Gemini Pro 2.5 to produce a HTML
| table.
|
| They will fail.
| bobsmooth wrote:
| Maybe I misunderstood the assignment but it seems to work
| for me.
|
| https://chatgpt.com/share/68f5f9ba-d448-8005-86d2-c3fbae028
| b...
|
| Edit: Just caught a mistake, transcribed one of the prices
| incorrectly.
| kbumsik wrote:
| Right, I wouldn't use full table detection to VLM model
| because they tend to mistake with numbers in table...
| pietz wrote:
| Maybe my imagination is limited or our documents aren't
| complex enough, but are we talking about realistic written
| documents? I'm sure you can take a screenshot of a very
| complex spreadsheet and it fails, but in that case you
| already have the data in structured form anyway, no?
| kbumsik wrote:
| > realistic written documents?
|
| Just get a DEF 14A (Annual meeting) filing of a company
| from SEC EDGAR.
|
| I have seen so many mistakes when looking at the result
| closely.
|
| Here is a DEF 14A filing from Salseforce. You can print
| it to a PDF and then try converting.
|
| https://www.sec.gov/Archives/edgar/data/1108524/000110852
| 425...
| grosswait wrote:
| Historical filings are still a problem, but hasn't the
| SEC required filing in an XML format since the end of
| 2024?
| richardlblair wrote:
| It's not really about SEC filings, though. While we folks
| on HN would never think of hard copies of invoices, but
| much of the world still operates this way.
|
| As mentioned above I have about 200 construction
| invoices. They are all formatted in a way that doesn't
| make sense. Most fail both OCR and OpenAI
| KoolKat23 wrote:
| OpenAI has unusuably low image DPI. Try Gemini.
| daemonologist wrote:
| Now if someone mails or faxes you that spreadsheet?
| You're screwed.
|
| Spreadsheets are not the biggest problem though, as they
| have a reliable 2-dimensional grid - at worst some cells
| will be combined. The form layouts and n-dimensional
| table structures you can find on medical and insurance
| documents are truly unhinged. I've seen documents that
| _I_ struggled to interpret.
| KoolKat23 wrote:
| To be fair, this is problematic for humans too. My old
| insurer outright rejected things like that stating it's
| not legible.
|
| (I imagine it also had the benefit of reducing
| fraud/errors).
|
| In this day and age, it's probably easier/better to
| change the process around that as there's little excuse
| for such shit quality input. I understand this isn't
| always possible though.
| richardlblair wrote:
| I had mentioned this when the new QWEN model dropped - I have
| a stack of construction invoices that fail through both OCR
| and OpenAI.
|
| It's a hard (and very interesting) problem space.
| carschno wrote:
| Technically not OCR, but HTR (hand-written text/transcript
| recognition) is still difficult. LLMs have increased accuracy,
| but their mistakes are very hard to identify because they just
| 'hallucinate' text they cannot digitize.
| sramam wrote:
| Interesting - have you tried sending the image and
| 'hallucinated' text together to a review LLM to fix mistakes?
|
| I don't have a use case of 100s or 1000s of hand-written
| notes have to be transcribed. I have only done this with
| whiteboard discussion snapshots and it has worked really
| well.
| lazide wrote:
| Often, the review LLM will also say everything is okay when
| it's made up too.
| mormegil wrote:
| This. I am reading old vital records in my family genealogy
| quest, and as those are sometimes really difficult to read, I
| turned to LLMs, hearing they are great in OCR. It's been...
| terrible. The LLM will transcribe the record without
| problems, the output seems completely correct, a typical text
| of a vital record. Just... the transcribed text has nothing
| to do with my specific record. On the other hand,
| transkribus.eu has been fairly usable for old vital record
| transcription - even though the transcribed text is far from
| perfect, many letters and words are recognized incorrectly,
| it helps me a lot with the more difficult records.
| pietz wrote:
| We ran a small experiment internally on this and it looked
| like Gemini is better at handwriting recognition than I am.
| After seeing what it parsed, I was like "oh yeah, that's
| right". I do agree that instead of saying "Sorry, I can't
| read that" it just made up something.
| CraigRood wrote:
| I have a thought that whilst LLM providers can say "Sorry"
| - there is little incentive and it will expose the reality
| that they are not very accurate, nor can be properly
| measured. That said, there clearly are use cases where if
| the LLM can't a certain level of confidence it should refer
| to the user, rather than guessing.
| Rudybega wrote:
| This is actively being worked on my pretty much every
| major provider. It was the subject of that recent OpenAI
| paper on hallucinations. It's mostly caused by benchmarks
| that reward correct answers, but don't penalize bad
| answers more than simply not answering.
|
| E.g.
|
| Most current benchmarks have a scoring scheme of
|
| 1 - Correct Answer 0 - No answer or incorrect answer
|
| But what they need is something more like
|
| 1 - Correct Answer 0.25 - No answer 0 - Incorrect answer
|
| You need benchmarks (particularly those used in training)
| to incentivize the models to acknowledge when they're
| uncertain.
| raincole wrote:
| If you can accept that the machine just make up what it doesn't
| recognize instead of saying "I don't know," then yes it's
| solved.
|
| (I'm not being snarky. It's acceptable in some cases.)
| jakewins wrote:
| But this was very much the case with existing OCR software as
| well? I guess the LLMs will end up making up plausible
| looking text instead of text riddled with errors, which makes
| it much harder to catch the mistakes, in fairness
| rkagerer wrote:
| Good libraries gave results with embedded confidence levels
| for each unit recognized.
| wahnfrieden wrote:
| Existing ocr doesn't skip over entire (legible) paragraphs
| or hallucinate entire sentences
| Davidzheng wrote:
| rarely happens to me using LLMs to transcribe pdfs
| criddell wrote:
| I usually run the image(s) through more than one
| converter then compare the results. They all have
| problems, but the parts they agree on are usually
| correct.
| KoolKat23 wrote:
| This must be some older/smaller model.
| red75prime wrote:
| Just checked it with Gemini 2.5 Flash. Instructing it to mark
| low-confidence words seems to work OK(ish).
| wahnfrieden wrote:
| Do any LLM OCRs give bounding boxes anyway? Per character and
| per block.
| kelvinjps10 wrote:
| Gemini does but it's not as good as Google vision, and the
| format it's different Here it's the documentation
| https://cloud.google.com/vertex-ai/generative-
| ai/docs/boundi...
|
| Also Simon Willison Made a blog post that might be helpful
| https://simonwillison.net/2024/Aug/26/gemini-bounding-box-
| vi...
|
| I hope that this capability improves so I can use only
| Gemini API.
| KoolKat23 wrote:
| These days it does just that, it'll say null or whatever if
| you give it the option. When it does make it up, it tends to
| be limitation of the image qualify ( max dpi).
|
| Blotchy text and specific typeface make 6's look like 8's,
| even to the non-discerning eye, a human would think it's an
| 8, zoom in and see it's a 6.
|
| Google's image quality on uploads is still streets ahead of
| openai for instance btw.
| peter-m80 wrote:
| No way it's solved. try to make OCR over a magazine with
| creative layouts. Not possible. I have a collection of vintage
| computer magazines and from time to time I try to OCR them
| whith the state of the art mechanisms. All of them requiere a
| lot of human intervention
| jmkni wrote:
| do you have an example of a particularly tricky one?
| ekianjo wrote:
| Just try old ads you will see how hard it gets
| pietz wrote:
| Could you provide an example that fails? I'm interested in
| this.
| constantinum wrote:
| I use LLMWhisperer[1] for OCR'ing old magazine ads. It
| preserves the layout and context. Example >
| https://postimg.cc/ts3vT7kG
|
| https://pg.llmwhisperer.unstract.com/
| darkwater wrote:
| So, the mug with inspirational text says "Bountiful Potential"?
| cahaya wrote:
| Lot's of OCR/ LLM's (Even Gemini Pro 2.5) still struggle
| converting complex tables to markdown or HTML: Tables with
| multiple headers and merged cells that get mixed up, multiple
| columns with tick boxes get mixed up, multi page tables that
| are not understood correctly. Also Llamaindex fails miserably
| on those things.
|
| Curious to hear which OCR/ LLM excels with these specific
| issues? Example complex table:
| https://cdn.aviation.bot/complex-tables.zip
|
| I can only parse this table correctly by first parsing the
| table headers manually into HTML as example output. However, it
| still mixes up tick boxes. Full table examples:
| https://www.easa.europa.eu/en/icao-compliance-checklist
| pietz wrote:
| I threw in the first image/table into Gemini 2.5 Pro letting
| it choose the output format and it looks like it extracted
| the data just fine. It decided to represent the checkboxes as
| "checked" and "unchecked" because I didn't specify
| preferences.
| CaptainOfCoit wrote:
| > Lot's of OCR/ LLM's (Even Gemini Pro 2.5) still struggle
| converting complex tables to markdown or HTML:
|
| But that's something else, that's no longer just OCR
| ("Optical Character Recognition"). If the goal suddenly
| changes from "Can take letters in images and make into
| digital text" to "Can replicate anything seen on a screen",
| the problem-space gets too big.
|
| For those images you have, I'd use something like Magistral +
| Structured Outputs instead, first pass figure out what's the
| right structure to parse into, second pass to actually fetch
| and structure the data.
| eeixlk wrote:
| htceaad t nofdnsy lyruuieo sieerrr t owcope?
| kmacdough wrote:
| > But that's something else, that's no longer just OCR
| ("Optical Character Recognition").
|
| Lines often blur for technologies under such rapid
| evolution. Not sure it's helpful to nitpick the verbal
| semantics.
|
| It is a fair question whether the OCR-inspired approach is
| the correct approach for more complex structured documents.
| But saying it's "not OCR" do doesn't seem meaningful from a
| technical perspective.
| kmacdough wrote:
| > But that's something else, that's no longer just OCR
| ("Optical Character Recognition").
|
| Lines often blur for technologies under such rapid
| evolution. Not sure it's helpful to nitpick the verbal
| semantics.
|
| It is a fair question whether the OCR-inspired approach is
| the correct approach for more complex structured documents
| where wider context may be important. But saying it's "not
| OCR" doesn't seem meaningful from a technical perspective.
| It's an extension of the same goal to convert images of
| documents into the most accurate and useful digitized form
| with the least manual intervention.
| CaptainOfCoit wrote:
| Personally I think it's a meaningful distinction between
| "Can extract text" VS "Can extract text and structure".
| It is true that some OCR systems can handle trying to
| replicate the structure, but still today I think that's
| the exception, not the norm.
|
| Not to mention it's helpful to separate the two because
| there is such a big difference in the difficulty of the
| tasks.
| vintermann wrote:
| OCR of printed text may be one thing, but handwriting OCR
| (a.k.a HTR) is very, very far from solved. It's actually hard
| to find a practical task general historical HTR is good enough
| to do usefully, even for state of the art models.
| robotswantdata wrote:
| VLLMs suck at complex layouts and there is a high risk of
| hallucination. Never use alone for contracts or health data.
| sbinnee wrote:
| Maybe for English. Other languages are very much not solved.
| llm_nerd wrote:
| Complex documents is where OCR struggles _mightily_. If you
| have a simple document with paragraphs of text, sure OCR is
| pretty solved. If you have a complex layout with figures and
| graphs and supporting images and asides and captions and so on
| (basically any paper, or even trade documents), it absolutely
| falls apart.
|
| And GP LLMs are heinous at OCR. If you are having success with
| FL, your documents must be incredibly simple.
|
| There has been enormous advances in OCR over the past 6 months,
| so the SoTa is a moving, rapidly advancing target.
| burpsnard wrote:
| I've only used tesseract, 'recreationally', but i tried
| generating images of random chars to see what
| resolution/contrast/noise was minimally recognisable; shocked
| at how bad it was. heavily relies on language models of
| character sequences, pretty useless On 'line noise'
| baobun wrote:
| Chinese, especially handwritten.
| Davidzheng wrote:
| I think it'll be good to have an end-to-end pdf to latex
| converter for old math papers. For commutative diagrams almost
| all models still struggle. especially very complicated
| commutative diagrams.
| simlevesque wrote:
| > That will cost you around $0.20 for 1000 pages in batch mode
| and the results have been great.
|
| Can you explain more about your setup ? I have a quarter
| million pages I want to OCR.
| constantinum wrote:
| Why PDF parsing is Hell[1]:
|
| Fixed layout and lack of semantic structure in PDFs.
|
| Non-linear text flow due to columns, sidebars, or images.
|
| Position-based text without contextual or relational markers.
|
| Absence of standard structure tags (like in HTML).
|
| Scanned or image-based PDFs requiring OCR.
|
| Preprocessing needs for scanned PDFs (noise, rotation, skew).
|
| Extracting tables from unstructured or visually complex
| layouts.
|
| Multi-column and fancy layouts breaking semantic text order.
|
| Background images and watermarks interfering with text
| extraction.
|
| Handwritten text recognition challenges.
|
| [1] https://unstract.com/blog/pdf-hell-and-practical-rag-
| applica...
| KoolKat23 wrote:
| I agree, Gemini 2.5 models are excellent.
|
| The fuss around old fashioned OCR seemed strange to me
| initially considering the above, but I selfishly forgot to
| consider addressing compute/offline requirements.
|
| It would also be nice for there to be a good competitor.
| cle wrote:
| That will not work with many of the world's most important
| documents because of information density. For example, dense
| tables or tables with lots of row/col spans, or complex forms
| with checkboxess, complex real-world formatting and features
| like strikethroughs, etc.
|
| To solve this generally you need to chunk not by page, but by
| semantic chunks that don't exceed the information density
| threshold of the model, given the task.
|
| This is not a trivial problem at all. And sometimes there is no
| naive way to chunk documents so that every element can fit
| within the information density limit. A really simple example
| is a table that spans hundreds pages. Solving that generally is
| an open problem.
| Gazoche wrote:
| There is no "solved" in computer vision, there is only "good
| enough" and what constitutes "good enough" depends on your
| problem domain.
|
| Take an OCR model with 99.9% character-wise accuracy. Sounds
| pretty good, right? Well if your use case is, say, digitalizing
| old printed novels, then yeah it's probably good enough.
|
| But what if your documents are personal records with millions
| of names, to insert in some administrative database? Now 1 out
| of 1000 persons will have their name misspelled. Ooops.
| 6gvONxR4sf7o wrote:
| OCR for printed documents is super robust, but handwriting, low
| res, and aligned recognition (not just image to "hello world"
| but also having "h is here in space e is here in space...) are
| all still well behind "basically solved."
| kelvinjps10 wrote:
| Google vision it's still better than Gemini at OCR, for example
| at getting bounding boxes.
| blindriver wrote:
| I attempted OCR using all of the open source models available
| about 3 months ago, including Llama 4. These were pngs of text
| using a regular font. Most produced garbage except Llama 4, and
| even then it was only about 90% accurate. Using OpenAI or
| Gemini produced much better results but the open source models
| were really bad.
| themanmaran wrote:
| > OmniAI benchmark that's also referenced here wasn't updated
| with new models since February 2025. I assume that's because
| general purpose LLMs have gotten better at OCR than their own
| OCR product.
|
| Benchmark author here. No, just pivoted away from OCR API as a
| product! Still use our API internally but have been lazy about
| updating benchmarks.
|
| Gemini is definitely the best model for OCR. But it has a
| really high rate of "recitation" errors. Where it will
| determine the output token is too close to its training data
| and cut it off. Something like 10% of the time from our
| testing. Also it has this hilarious hallucination when you have
| a blank page in the document mix and it just makes up new info.
|
| OpenAI is OK. GPT5 wasn't any better than 4o or 4.1. Main
| issues were: dropping content like headers/footers, loses it's
| mind on sideways pages, and will frequently refuse to read
| things like ID documents, health care forms, or things it
| judges to have too much PII.
| singularity2001 wrote:
| Instead of downloading a specific OCR model how would one fare
| just downloading the currently best multi-modal foundation model?
| And what would that be at less than 30 GB?
| prats226 wrote:
| Then you can just download finetuned version of same multi-
| modal foundation model that's trained on documents?
| ammar_x wrote:
| Language support is not mentioned in the repo. But from the
| paper, it offers extensive multilingual support (nearly 100
| languages) which is good, but I need to test it to see how it
| compares to Gemini and Mistral OCR.
| zacmps wrote:
| I suspect the number of langauges it can do with reasonable
| accuracy is actually much smaller, probably <15.
| 2big2fail_47 wrote:
| I find it interesting that there's all these independent AI-OCR
| Projects but still no commercial offering. Is it still too
| inaccurate, too complex or simply too expensive?
| Annatar01 wrote:
| I dont know, but maybe existing commercial OCR is still on top,
| and also using ML. Recently tried a free trial for OCR/reading
| Sutterlin and it was a weird feeling being so outclassed in
| reading.
| Eisenstein wrote:
| It is because the AI is not actually doing OCR. It is giving an
| interpretation of what the text in an image is by ingesting
| vision tokens and mapping them onto text tokens.
|
| So you either have to be fine with a lot of uncertainty as to
| the accuracy of that interpretation or you have to wait for an
| LLM that can do it in a completely reproducible way every time.
| rsolva wrote:
| Mistral offers their OCR commercially through their API and in
| their Chat services, at least.
|
| https://mistral.ai/news/mistral-ocr
| simlevesque wrote:
| https://cloud.google.com/document-ai
| daemonologist wrote:
| There are commercial OCR offerings from the big cloud providers
| (plus, like, Adobe). In my experience they generally outperform
| anything open-weights, although there's been a lot of
| improvement in VLMs in the past year or two.
| aleinin wrote:
| One that I've seen recently is https://reducto.ai It appears to
| be an OCR wrapper.
| prats226 wrote:
| https://docstrange.nanonets.com/ as well, wrapper on top of 7B
| version of https://huggingface.co/nanonets/Nanonets-OCR2-3B
| tinyhouse wrote:
| OCR is not a great name for these models. While they can do
| traditional OCR such as digitize and scanned PDF for example,
| they do so much more.
| intalentive wrote:
| >they do so much more I'm not familiar. What else are they good
| for?
| tinyhouse wrote:
| They can take something like an image of a graph and provide
| a description of it. From my understanding, these are
| multimodal models with reasoning capabilities.
| breadislove wrote:
| For everyone wondering how good this and other benchmarks are:
|
| - the OmniAI benchmark is bad
|
| - Instead check OmniDocBench[1] out
|
| - Mistral OCR is far far behind most Open Source OCR models and
| even further behind then Gemini
|
| - End to End OCR is still extremely tricky
|
| - composed pipelines work better (layout detection -> reading
| order -> OCR every element)
|
| - complex table parsing is still extremely difficult
|
| [1]: https://github.com/opendatalab/OmniDocBench
| hakunin wrote:
| Wish someone benchmarked Apple Vision Framework against these
| others. It's built into most Apple devices, but people don't
| know you can actually harness it to do fast, good quality OCR
| for you (and go a few extra steps to produce searchable pdfs,
| which is my typical use case). I'm very curious where it would
| fall in the benchmarks.
| wahnfrieden wrote:
| It is unusable trash for languages with any vertical writing
| such as Japanese. It simply doesn't work.
| thekid314 wrote:
| Yeah, and fails quickly at anything handwritten.
| hakunin wrote:
| I mostly OCR English, so Japanese (as mentioned by
| parent) wouldn't be an issue for me, but I do care about
| handwriting. See, these insights are super helpful. If
| only there was, say, a benchmark to show these.
|
| My main question really is: what are practical OCR tools
| that I can string together on my MacBook Pro M1 Max w/
| 64GB Ram to maximize OCR quality for lots of mail and
| schoolwork coming into my house, all mostly in English.
|
| I use ScanSnap Manager with its built in OCR tools, but
| that's probably super outdated by now. Apple Vision does
| way better job than that. I heard people say also that
| Apple Vision is better than Tesseract. But is there
| something better still that's also practical to run in a
| scripted environment on my machine?
| wahnfrieden wrote:
| LiveText too? It has a newer engine
| hakunin wrote:
| This is the second comment of yours about LiveText (this
| is the older one
| https://news.ycombinator.com/item?id=43192141) -- I found
| that one by complete coincidence because I'm trying to
| provide a Ruby API for these frameworks. However, I can't
| find much info on LiveText? What framework is it part of?
| Do you have any links or any additional info? I found a
| source where they say it's specifically for screen and
| camera capturing.
| wahnfrieden wrote:
| https://developer.apple.com/documentation/visionkit/image
| ana... VisionKit. Swift-only (as with many new APIs) so
| lots of people stuck on ObjC bridges simply ignore it.
|
| It does not provide bounding boxes but you can get text.
| hakunin wrote:
| That's great, I'm going to give this a shot. If you have
| any more resources please do share. I don't mind Swift-
| only, because I'm writing little shims with `@_cdecl` for
| the bridge (don't have much experience here, but hoping
| this is going to work, leaning on AI for support).
| CaptainOfCoit wrote:
| Yeah, if it was cross-platform maybe more people would be
| curious about it, but something that can only run on ~10% of
| the hardware people have doesn't make it very attractive to
| even begin to spend time on Apple-exclusive stuff.
| ch1234 wrote:
| But you can have an apple device deployed in your stack to
| handle the OCR, right? I get on-device is a hardware
| limitation for many, but if you have an apple device in
| your stack, can't you leverage this?
| CaptainOfCoit wrote:
| Yeah, but handling macOS is a infrastructure-capacity
| sucks, Apple really doesn't want you to so tooling is
| almost none existing. I've setup CI/CD stacks before that
| needed macOS builders and it's always the most cumbersome
| machines to manage as infrastructure.
| coder543 wrote:
| AWS literally lets you deploy Macs as EC2 instances,
| which I believe includes all of AWS's usual EBS storage
| and disk imaging features.
| CaptainOfCoit wrote:
| Alright, so now the easy thing is done, now how do you
| actually manage them, keep them running and do
| introspection without resorting to SSH or even remote
| desktop?
| coder543 wrote:
| How do you manage _any_ EC2 instance "without resorting
| to SSH"? Even for Linux EC2 instances, the right answer
| is often tools like Ansible, which do still use SSH under
| the hood.
| CaptainOfCoit wrote:
| You usually provision them via images, that they then
| either install from or boot from directly. Not to mention
| there are countless of infrastructure software to run
| that works for at least Linux, sometimes Windows and
| seldom even macOS.
| coder543 wrote:
| I specifically mentioned the imaging capability of EBS
| for Mac, which you dismissed as the easy part. Now you're
| claiming that is the main thing? Well, good news!
|
| And yes, Ansible (among other tools) can be used to
| manage macOS.
|
| This discussion doesn't seem productive. You have a
| preconceived view point, and you're not actually
| considering the problem or even doing 5 seconds of
| googling.
|
| Managing a Mac fleet on AWS isn't a real problem. If
| Apple's OCR framework were significantly above the
| competition, it could easily be used. I would like to see
| benchmarks of it, as the other person was also asking
| for.
| hakunin wrote:
| 10% of hardware is an insanely vast amount, no?
| CaptainOfCoit wrote:
| Well, it's 90% less than what everyone else uses, so even
| if the total number is big, relatively it has a small
| user-base.
| hakunin wrote:
| I don't think 10% of anything would be considered
| relatively small even if we talk about 10 items:
| literally there's only 10 items and this 1 has the rare
| quality of being among 10. Let alone billions of devices.
| Unless you want to reduce it to tautology, and instead of
| answering "why it's not benchmarked" just go for "10 is
| smaller than 90, so I'm right".
|
| My point is, I don't think any comparative benchmark
| would ever exclude something based on "oh it's just 10%,
| who cares." I think the issue is more that Apple Vision
| Framework is not well known as an OCR option, but maybe
| it's starting to change.
|
| And another part of the irony is that Apple's framework
| probably gets way more real world usage in practice than
| most of the tools in that benchmark.
| CaptainOfCoit wrote:
| The initial wish was that more people cared about Apple
| Vision Framework, I'm merely claiming that since most
| people don't actually have Apple hardware, they're
| avoiding Apple technology as it commonly only runs on
| Apple hardware.
|
| So I'm not saying it should be excluded because it's can
| only used by relatively few people, but I was trying to
| communicate that I kind of get why not so many people
| care about it and why it gets forgotten, since most
| people wouldn't be able to run it even if they wanted to.
|
| Instead, something like DeepSeek OCR could be deployed on
| any of the three major OSes (assuming there is
| implementations of the architecture available), so of
| course it gets a lot more attention and will be included
| in way more benchmarks.
| hakunin wrote:
| I get what you're saying, I'm just disagreeing with your
| thought process. By that logic benchmarks would also not
| include the LLMs that they did, since most people
| wouldn't be able to run those either (it takes expensive
| hardware). In fact, more people would probably be able to
| run Vision framework than those LLMs, for cheaper (Vision
| is even on iPhones). I'm more inclined to agree if you
| say "maybe people just don't like Apple". :)
| graeme wrote:
| Interesting. How do you harness it for that purpose? I've
| found apple ocr to be very good.
| hakunin wrote:
| The short answer is a tool like OwlOCR (which also has CLI
| support). The long answer is that there are tools on github
| (I created the stars list:
| https://github.com/stars/maxim/lists/apple-vision-
| framework/) that try to use the framework for various
| things. I'm also trying to build an ffi-based Ruby gem that
| provides convenient access in Ruby to the framework's
| functionality.
| cheema33 wrote:
| > the OmniAI benchmark is bad
|
| According to Omni OCR benchmark, Omni OCR is the best OCR. I am
| sure you all will find no issues with these findings.
| loaderchips wrote:
| Great work guys, how about we replace the global encoder with a
| Mamba (state-space) vision backbone to eliminate the O(n2)
| attention bottleneck, enabling linear-complexity encoding of
| high-resolution documents. Pair this with a non-autoregressive
| (Non-AR) decoder--such as Mask-Predict or iterative refinement--
| that generates all output tokens in parallel instead of
| sequentially. Together, this creates a fully parallelizable
| vision-to-text pipeline, The combination addresses both major
| bottlenecks in DeepSeek-OCR.
| loaderchips wrote:
| not sure why i m getting downvoted. Would love to have a
| technical discussion on the validity of my suggestions.
| neves wrote:
| I see that the project uses conda for development. Is it still a
| good tool now that pip also install binaries?
| modeless wrote:
| No. Everyone should be using uv instead.
| foofoo12 wrote:
| How does it compare to Tesseract? https://github.com/tesseract-
| ocr/tesseract
|
| I use ocrmypdf (which uses Tesseract). Runs locally and is
| absolutely fantastic. https://ocrmypdf.readthedocs.io/en/latest/
| utopiah wrote:
| Indeed, seems the default benchmark is LLM/VLM based
| alternatives as if they somehow "solved" the problem but IMHO
| even if it goes from (totally made up numbers) 80% with
| tesseract to 95% with this or Qwen or whatever but it takes
| 100x harddisk with containers or a CUDA stack, dedicated
| hardware, e.g. GPU with 16GB or VRAM, etc then it's such a
| trade of it should be considered.
| tidbeck wrote:
| How does this compare to https://huggingface.co/ibm-
| granite/granite-docling-258M in performance and how they work?
| bugglebeetle wrote:
| The granite Dockling models are unfortunately quite far below
| SOTA. dots-ocr and PaddleOCR were best here.
| hank2000 wrote:
| Have yall seen tensorlake? I'm curious how this compares to a
| model custom built for the problem. My guess is it can be as
| good. But can it be as efficient?
|
| disclaimer: I do not work for tensorlake--but i know the folks
| behind it.
| modeless wrote:
| Hmm, at first I was thinking "why OCR?", but maybe the reason is
| to ingest more types of training data for LLM improvement, e.g.
| scanned academic papers? I imagine all the frontier labs have a
| solution for this due to the value of academic papers as a data
| source.
|
| Edit: Oh I see the paper abstract says this explicitly: "In
| production, DeepSeek-OCR can generate training data for LLMs/VLMs
| at a scale of 200k+ pages per day (a single A100-40G)". This is
| just part of the training data ingestion pipeline for their real
| models. Explains why the architecture is not using all of their
| latest tricks: it's already good enough for their use case and
| it's not the main focus.
| rsp1984 wrote:
| Can someone ELI5 to me (someone who doesn't have the time to keep
| up with all the latest research) what this is and why it's a big
| deal?
|
| It's very hard to guess from the github and paper. For example,
| there is OCR in the title but the abstract and readme.md talk
| about context compression for LLMs, which I find confusing.
| Somebody care to explain the link and provide some high-level
| context?
| intalentive wrote:
| Suppose you have an image with 1000 words in it, and suppose
| for simplicity that every word is 1 token. Then the image is
| "worth" 1000 tokens.
|
| But under the hood, the image will have to be transformed into
| features / embeddings before it can be decoded into text.
| Suppose that the image gets processed into 100 "image tokens",
| which are subsequently decoded into 1000 "text tokens".
|
| Now forget that we are even talking about images or OCR. If you
| look at just the decoding process, you find that we were able
| to compress the output into a 10x smaller representation.
|
| The implication for LLMs is that we don't need 1000 tokens and
| 1000 token embeddings to produce the 1001st token, if we can
| figure out how to compress them into a 10x smaller latent
| representation first.
| dumpsterkid wrote:
| I haven't fired this up yet to try but I've been evaluating &
| working with quite a few different VLMs from the small granite,
| qwen etc models up to the larger VLMs available to see if we can
| fully replace traditional OCR in our system but I've been
| disappointed so far - our system takes documents from customers
| and supplies them back normalized documents (i.e rasterized
| multi-page bitmaps) marked up as they've requested - however in
| our use case we need accurate coordinates of data down to the
| letter/word level and from my experience the positional outputs
| from these VLMs are either wildly inconsistent, completely
| hallucinated, or so vague that it doesn't allow us to target
| anything with any kind of accuracy or granularity.
|
| our solution so far has been to stick to using tesseract with
| good clean-up routines and then augmenting/fixing-up the output
| using the VLM OCR text where we don't have structured source
| document data available
|
| it could be that we just have a very niche use-case and it
| doesn't matter to most people, I'm sure if you just want a text
| dump or restructured markdown/html representation of documents
| these VLMs work well but the number of articles & comments I've
| seen claiming that these models have 'solved' OCR just seems
| counter to our experiences
| sampton wrote:
| You can train a cnn to find bounding boxes of text first. Then
| run VLM on each box.
| kamranjon wrote:
| Have you tried moondream yet[1]? The moondream 3 preview
| model[2], according to the blogpost[3] appears to outperform
| many frontier models on VLM tasks and does so with a relatively
| small footprint.
|
| [1] https://moondream.ai/
|
| [2] https://huggingface.co/moondream/moondream3-preview
|
| [3] https://moondream.ai/blog/moondream-3-preview
| jmpeax wrote:
| Your customers don't have any handwritten text?
| shepardrtc wrote:
| How does this fair with the Vidore benchmark?
|
| https://huggingface.co/spaces/vidore/vidore-leaderboard
| schopra909 wrote:
| It's not clear to me what the bottleneck for OCR to "100%" work
| with LLMS is.
|
| In my work we do a lot of stuff with image understanding and
| captioning (not OCR). There object identification and description
| works great, since all the models are using a CLIP like visual
| backbone. But it falls apart when you ask about nuances like
| left/right or counting (reasoning kind of improves the latter but
| it's too expensive to matter IMO).
|
| For our tasks, it's clear that there's more fundamental research
| that needs to be done on vision understanding to push past CLIP.
| That would really improve LLMs for our usecases.
|
| Curious if there's something similar going on for OCR in the
| vision encoder that's fundamentally holding it back.
| edtechdev wrote:
| I tried this out on huggingface, and it has the same issue as
| every other multimodal AI OCR option (including MinerU, olmOCR,
| Gemini, ChatGPT, ...). It ignores pictures, charts, and other
| visual elements in a document, even though the models are pretty
| good at describing images and charts by themselves. What this
| means is that you can't use these tools yet to create fully
| accessible alternatives to PDFs.
| mediaman wrote:
| I have a lot of success asking models such as Gemini to OCR the
| text, and then to describe any images on the document,
| including charts. I have it format the sections with XML-ish
| tags. This also works for tables.
| allanren wrote:
| It says the conversation can reduce size with large compression,
| which basic make the image blur but still contaim import
| information.
|
| This is indeed amazing. It's actually how human try to understand
| and remember things. BY VISUAL! And when memory fade out, the
| image are getting blurred.
|
| Not sure if those close source multimodal models are already
| using this method.
| dlowe24 wrote:
| The only model that I found so far that extract table data with
| OCR is dots.ocr. Models that came after it have not done a good
| job. Interesting on testing this new model.
| joshstrange wrote:
| > [2025/x/x] We release DeepSeek-OCR, a model to investigate the
| role of vision encoders from an LLM-centric viewpoint.
|
| So close but it should be 2025/X/XX as "X" = 10 in Roman Numerals
| /s
|
| Jokes aside, this is really neat and I'm looking forward to
| getting this running. For most OCR-type stuff I just use AWS
| Textract since I need it so rarely and that service does a decent
| job. I really like how well this model seems to extract
| images/figures as well from the original document.
| simonw wrote:
| I figured out how to get this running on the NVIDIA Spark (ARM64,
| which makes PyTorch a little bit trickier than usual) by running
| Claude Code as root in a new Docker container and having it
| figure it out. Notes here:
| https://simonwillison.net/2025/Oct/20/deepseek-ocr-claude-co...
|
| Here's a result I got
| https://github.com/simonw/research/blob/main/deepseek-ocr-nv... -
| against this image:
| https://static.simonwillison.net/static/2025/ft.jpeg
| jjcm wrote:
| Looks like this did really solid, with the exception of the
| paragraph directly below the quote. It hallucinated some filler
| there and bridge it with the next column.
|
| Thanks for running the test quickly!
| djmips wrote:
| By my eye it just bridge. I didn't see any filler. It went
| from "Code is a language" - above the quote and then to "in a
| garden by name." which was the top of the next column but
| missing the chicken subject.
| throwaway314155 wrote:
| > by running Claude Code as root in a new Docker container
|
| How do you get the "as root" part of that to work?
|
| (sorry if it's explained in your article)
| simonw wrote:
| Run it on a root account and do: IS_SANDBOX=1
| claude --dangerously-skip-permissions
| arkmm wrote:
| Wow, this deserves its own submission.
| CaptainOfCoit wrote:
| It missed the initial "A" in the text which I sort of
| understand, seems not a lot of news articles were put in the
| dataset. But more interestingly, it missed the entire
| "Hallucination is a risk and...", the article "theme" next to
| the author name also the final email.
| prats226 wrote:
| Top 3 models on huggingface are all OCR models. Most automation
| projects involve documents where you need a model finetuned to
| understand all elements inside documents and provide grounding
| and confidence scores etc which is why these subset of models are
| gaining popularity
___________________________________________________________________
(page generated 2025-10-20 23:01 UTC)