All work06 / 07

Own product · On-device screenshot search

Screenshot Memory

On-device screenshot search that puts the right screenshot in the top three for 97.9% of test queries.

of queries find it in the top 3
97.9%
median search, on an emulator
528 ms
less peak memory at 1,250 screenshots
70%
to sort 1,000 screenshots with a vision model
$0.135
From the repo

Android in closed testing

Role
Solo — product, design and engineering
For
Independent product
When
Oct 2026 — now
Stack
Kotlin · Jetpack Compose · ML Kit · llama.cpp · EmbeddingGemma · Swift · SwiftUI · Cloudflare Workers · D1 · Python
Start a project

Building something like this? Tell me about it.

Screenshot Memory on iPhone: 1,200 test screenshots, private to the phone, sorted into stacks

The brief

Find any screenshot by what you remember.

An Android and iOS app that reads and searches your screenshots on the phone — by a word, a booking code, a date or a price. Nothing is uploaded unless you opt in to one of the two features that need a server.

For a lot of people screenshots are the to-do list and the memory: the booking code, the price, the address. Gallery search can’t find them. Screenshot Memory has to be exact where it matters — a code is either right or useless — and forgiving where memory is vague, without sending anyone’s screenshots anywhere.

What I built

What I built

  • 01

    Exact where it must be

    On-device text recognition and a small embedding model are fused with exact matchers for codes, dates and prices — so “VN7K2P” finds the booking and “hotel in Vietnam” finds it too.

  • 02

    On the phone by default

    Screenshots, their text and your searches stay on the device. The two opt-in features send one reduced image at a time, and anything that looks like a password, a one-time code, a card or an ID is never sent.

  • 03

    A cost model before a price

    Sorting with a hosted vision model costs about $0.135 per thousand screenshots, so the free tier, the rewarded ads and the daily budget cap were sized from that number rather than guessed.

  • 04

    A benchmark with two honest views

    Search quality is reported strictly and generously side by side, on a held-out split fixed by hash. One change — embedding the query as typed — lifted strict top-1 from 0.49 to 0.58.

What it looks like

  • Searching “hotel booking in Vietnam” returns eleven matching screenshots (store screenshot, synthetic test data)

  • Searching the booking code VN7K2P returns the two screenshots that contain it

  • A travel booking opened with one-tap actions: add to calendar, copy the code, remind me

  • A product screenshot on iPhone with its price detected

A leak that wasn’t where it looked

A benchmark run died out of memory, and the obvious suspect was the screenshot index. Heap dumps at 250 to 1,250 screenshots said otherwise: the stored data was small and the image cache was flat, while the number of live screenshot objects grew in a straight line — 87,250 of them at 1,250 screenshots.

The path led into the UI toolkit’s grid, which was holding on to earlier layouts: a known bug, already fixed upstream. The fix was a dependency bump with no source changes — peak memory fell from 166 to 50 MiB and live objects from 87,250 to 3,785. The original crash was never reproduced as an out-of-memory error, so the write-up says its exact cause is still unproven.

Not done yet

It’s in closed testing, not yet public on Google Play. Every Android number here comes from an emulator, not a phone. The iPhone app runs on my own device but isn’t on the App Store, the subscription isn’t launched, and text recognition is Latin-script only.

Have something that needs to be right?

Open to select projects and roles. I read every message myself and reply within one working day.

Next case study

AutoDS

AutoML that beat AutoGluon on 9 of 16 datasets and reports its own accuracy to within 0.015.

Read the case study