AI tool for receipt recognition
Receipt recognition becomes much more difficult when there is no consistent visual layout to rely on. In Israel, this is a particularly relevant challenge: receipts from different businesses and systems can vary significantly in structure, field placement, fonts, and data formats. The date, total, payment method, or line items may appear in different places from one document to another.
In one of our projects for an Israeli consulting company, we needed to solve this without creating a separate recognition template for every format. We built a prototype around a multimodal LLM that could read receipts and other incoming documents and return the extracted information as structured data for further processing.
About client
The client is an Israeli consulting company that helps businesses implement and improve information systems and data solutions. One use case the company wanted to test was automating the processing of financial documents received from suppliers and other systems.
Tools
To prepare images for recognition, we upscaled them by a factor of three, converted them to grayscale and slightly adjusted the contrast. We used a multimodal LLM to extract structured data from documents without relying on fixed field positions. The results can be reviewed, edited and exported, including in JSON format.
Team
The project team consisted of an AI engineer who also served as tech lead, and a project manager. Together, they designed the solution’s logic, configured image preprocessing and the multimodal LLM, then built and tested the prototype on documents with varying layouts and image quality.
What we did
The system needed to read documents in different formats and of varying quality, extracting specific data: seller details, date and time, receipt number, currency, payment method, line items and totals.
We could not rely on consistent input. Even receipts of the same type could differ considerably in layout, field placement, fonts, text density and data formatting. Some documents also contained handwritten notes that needed to be captured alongside the printed text. Image quality and camera angles could vary widely.
Why a template‑based approach would not work
When documents have a consistent structure, data extraction can rely partly on predefined rules—for example, looking for the date or total in a specific area of the image. In this case, the required information could appear in different places even on visually similar receipts.
Tying amounts, dates, line items or payment methods to fixed coordinates would have been unreliable. Each new format would require its own rules to be created and maintained. We needed an approach that did not depend on fixed field positions.
How we approached it
Before recognition, each image went through preprocessing: high‑quality 3× upscaling, grayscale conversion and a slight contrast adjustment. This helped prepare images containing small or faint text for analysis.
The processed image was then passed to a multimodal LLM. The model analyzed the document as a whole and used context to extract the required information, without needing predefined locations for each field.
In the prototype, the model extracted the seller, date and time, receipt number, currency, payment method, line items, quantities, prices and total amount. Handwritten notes were also included in the recognition results.
Unlike an approach that combines OCR with rigid templates and field extraction rules, this method does not require the location of each value to be specified in advance. The same process can therefore handle documents with different layouts.
Turning extracted text into usable data
Reading text from an image was only part of the task. The output also needed to be ready for further processing.
After recognition, the extracted data is organized into individual fields and line items. Users can review the results, edit values where needed and export them. The data is also available in JSON format, allowing it to be passed to other systems without manually interpreting the recognized text.
This makes document recognition part of a broader workflow—for example, processing incoming invoices and receipts from suppliers, then passing the extracted data to enterprise systems.
Result
In the prototype, the same approach worked across documents with different layouts and image quality, including receipts with handwritten notes, without requiring a separate template for each format.
The output is structured data that users can review, correct and pass to other systems, rather than a block of recognized text. This makes the approach useful both for reading documents and for automating the workflows that begin with them.