Camera text recognition on Android — building a Google Lens-style OCR app
Reading text off a live camera feed is a pipeline problem rather than a single API call: acquire frames, hand them to a recognition engine, and reconcile a stream of noisy per-frame results into something stable enough to show a user.
- 2020-06-05
- Programmatic DIB
- Android · OCR · Computer vision
The pipeline
Three stages. Camera frames arrive continuously; a detector runs recognition on a frame; the result is drawn back over the preview. The interesting engineering is in the gaps between those stages rather than in any one of them.
Recognition is slower than the frame rate, so every frame cannot be processed. The usual approach is to process the most recent frame whenever the detector is free and drop the rest, which keeps latency bounded instead of building a backlog.
Why on-device matters here
On-device recognition means no network round trip per frame, which is what makes live overlay feel immediate rather than laggy. It also means the camera feed never leaves the phone — a meaningful property for an application pointed at documents.
Where accuracy comes from
Most real-world OCR failures are acquisition failures rather than model failures: motion blur, poor lighting, extreme angle, or text too small in frame. Guiding the user toward a better frame does more for perceived accuracy than any post-processing.
Temporal aggregation helps too. A single frame is noisy; agreement across several consecutive frames is a much stronger signal, and it stops the overlay flickering between readings.
- Process the latest frame and drop the rest — never queue frames behind a slow detector.
- On-device recognition buys both latency and privacy.
- Most OCR errors are acquisition problems; guide the user to a better frame.
- Aggregating across frames stabilises a noisy per-frame result.