ReMyndwhat is ocr?
ocr stands for optical character recognition, and it is the step that turns an image of text into text a computer can actually search.
point it at a screenshot and it returns the words, plus a box saying where each one sat. that is the whole job. everything interesting about ocr in 2026 happens either side of it, because recognition itself is close to solved on screen text.
this matters more than it used to. as of september 2026 most of what you read at work never becomes searchable text: a slide in a call, a pdf in a preview pane, a dashboard, a terminal. ocr is the thing standing between those pixels and a search box.
how ocr actually works
ocr runs in four stages, and the last one causes most of the trouble.
- preprocess. deskew, denoise, normalise contrast. on screen captures this stage is nearly free, because the image was rendered rather than photographed.
- detect. find the regions that contain text at all, and draw a box around each line or word.
- recognise. convert the pixels inside each box into characters. modern engines use a neural sequence model rather than matching glyph templates one at a time.
- order. decide what reads after what. this is the step that produces scrambled tables and interleaved columns.
stages 2 and 3 are where 40 years of research went, and where the accuracy now is. tesseract, still the best known open source engine, started at hp between 1984 and 1994, was open sourced in 2005, and has been developed under google since 2006. stage 4 is where your results break in 2026, because reading order is a layout inference rather than a character problem.
who ships ocr in 2026
every large vendor now runs ocr somewhere, and each stopped at the surface it controls.
| who | where the ocr runs | scope |
|---|---|---|
| apple | vision framework, on device since 2017; live text since 2021 | any image or screen on the mac |
| microsoft | recall, rebuilt with on device encryption and biometric gating | the windows screen, on copilot+ pcs |
| tesseract stewardship since 2006, plus lens | images you hand it | |
| openai | gpt vision reads images you attach to chatgpt | one conversation at a time |
| anthropic | claude reads attached images and screenshots | one conversation at a time |
the split is consistent. apple and microsoft put ocr in the operating system, where it is free and local. openai and anthropic put it in a model, where it is metered and remote. neither group builds you a searchable archive of everything you looked at, which is the gap the screen memory category exists to fill.
screen text is the easy case
there is a large ocr accuracy gap between the two things people both call ocr.
| source | typical difficulty | why |
|---|---|---|
| rendered screen text | lowest | perfect glyphs, known fonts, no lens, no lighting |
| digital pdf | low | text often already embedded, ocr only needed for scans |
| photo of a document | medium | perspective, focus, shadow, paper texture |
| dense table or form | medium to high | recognition is fine, reading order is not |
| handwriting | highest | unbounded variation, no font to anchor to |
a screenshot is the best possible input. the characters were drawn by the operating system a moment earlier, at a known size, in a known typeface, with no camera in between. that is why screen memory tools can lean on ocr in a way a receipt scanning app cannot.
the gap is wide enough that benchmark work now separates the two. ocrbench, published in may 2023, exists specifically because large multimodal models are strong at describing an image and much weaker at reading every character in it correctly.
on device ocr is now a default, not a feature
running ocr on your own mac stopped being an achievement around 2021, and a lot of marketing has not caught up.
apple shipped the apple vision framework in 2017 and added live text to macos in 2021. any mac app can call recognizing text in images and get words plus bounding boxes back on device, on the neural engine, with no network request, no api key and no per page cost.
so "our ocr runs locally" describes the platform, not the product. it is still worth knowing, because it decides three things that do matter:
- cost. local ocr is free per image. cloud ocr is priced per page, which quietly caps how much you can afford to read.
- latency. no round trip, so recognition keeps up with a screen that changes many times a minute.
- exposure. the image never leaves. that is a real property, and it is narrower than "nothing ever leaves your machine", which is a claim almost no shipping app can honestly make.
when a tool says its ocr on mac is on device, the follow up question is what happens to the text afterwards. that is usually where the network appears, and it is the fork we take apart in private ai on your mac.
what ocr costs in disk, measured
the text is almost free and the pictures are not.
we measured a working screen memory install rather than estimating, because nobody publishes this number. across 17 days of capture it held 26gb. video was roughly 24.6gb of that, about 95%. the searchable index, which is the ocr output plus its metadata, was 1.4gb, about 5%. that is roughly 1.5gb per recorded day.
the ratio is the useful part. a year of ocr text runs around 20gb. a year of the video it came from runs around 360gb, which is 18 times more for the part you almost never replay. keep the index long and the recordings short. the full working is in screen recording storage on a mac.
ocr and ai vision are different tools
ocr and a vision model fail in opposite directions, and they get conflated constantly.
| ocr | vision model | |
|---|---|---|
| returns | characters and their positions | a description or an answer |
| runs | locally, in milliseconds | usually remote, in seconds |
| cost per image | effectively zero | metered per call |
| deterministic | yes | no |
| good at | finding the words that were there | understanding what the picture means |
| bad at | reading order, layout, meaning | exact strings, and being cheap enough to run on everything |
the practical pattern in 2026 is both, in that order. ocr everything, because it is cheap enough to run on every frame you keep. then send the small number of images that genuinely need interpretation to a vision model. running a vision model across an entire screen history would cost more than the history is worth.
that split explains a common disappointment. finding the exact error message you saw in july is ocr and a search index doing the work. summarising what your week was about is a language model reading retrieved text, which is retrieval augmented generation with your own screen as the corpus.
where ocr quietly fails
five failure modes are worth knowing before you trust any ocr backed search.
- reading order in tables. the characters are right and the rows are wrong, so a number ends up against the wrong label.
- low contrast interfaces. grey on grey placeholder text and disabled controls often are not detected at all.
- text inside images inside documents. a screenshot pasted into a slide is two layers of rendering, and the inner one is smaller than it looks.
- partial capture. if only the focused window is recorded, anything in a background window was never an ocr problem, because it was never an image.
- the thing nobody typed. ocr reads what was on screen. a decision made out loud on a call leaves no pixels.
the last two are scope limits rather than accuracy limits, and they are the ones people mistake for bugs. this is also why retrieval over a personal archive has a hard ceiling that no recognition quality fixes, which is the argument in what is context engineering.
why any of this is worth the disk
people spend a lot of their day looking things up, and almost none of that search reaches their own work.
per the anthropic economic index for may 2026, the single most common work task in sampled claude conversations is searching electronic sources for information, at 4.95%. reference searching is second at 3.74%. together that is 8.69% of conversations spent looking something up.
now compare that with what people can actually reach.
| request topic | share of conversations |
|---|---|
| content creation and copywriting | 22.72% |
| education and learning | 13.23% |
| software development | 11.51% |
| research and intelligence | 10.94% |
| document processing and extraction | 4.32% |
| knowledge retrieval and enterprise search | 3.61% |
| personal ai assistant | 2.86% |
| conversation and meeting intelligence | 0.26% |
document processing and extraction, which is the ocr shaped task, is 4.32%. searching your own knowledge is 3.61%. anything about a meeting is 0.26%. those are supply numbers rather than demand numbers: people are not uninterested in their own work, they simply have no index of it. work accounts for 43.36% of classified conversations and 51.38% of all usage is augmentation rather than automation, meaning long sessions where a person and a model think together. ocr over your own screen is one of the few ways to give those sessions anything of yours to think with.
where remynd sits
remynd records the focused window on a mac, runs ocr locally through apple's vision framework, and keeps the resulting index on your machine. that index is what makes a search for something you saw in june actually return it.
the scope, stated exactly: capture is the focused window rather than every pixel of every display, ocr and storage stay on your mac, recordings default to 30 days of retention, you can exclude specific apps or sites from capture entirely, and your history is read only everywhere in the app. call transcription also runs on device, using an mlx speech model on apple silicon.
what it does not do: asking a question sends the retrieved slices to a cloud model, so this is not a claim that nothing ever leaves the machine. the honest version is that the archive lives with you.
for the category rather than the component, start with screen memory. for the practical version, read how to find something you saw on your screen.