New Open-Source AI Model NeoMME Reads Text and Images With One Shared Network

Hcompany has released NeoMME, a family of two open-source encoder models — with 260 million and 800 million parameters — designed to turn both text and images into a shared mathematical representation, called an embedding, that computers can compare and search. Unlike many current AI systems that pair a separate image-recognition module with a text-generating language model, NeoMME uses a single Transformer network, a type of neural network design used in most modern AI systems, to process text tokens and raw image patches together. It was trained from scratch using a technique called masked discrete diffusion, in which the model learns by guessing words that have been deliberately hidden from a passage of text.

The developers fine-tuned NeoMME for visual document retrieval — searching for the right document page based on its actual appearance, such as layout, tables, charts and fonts, rather than text extracted from it beforehand. On this task, called NeoMME-Retriever, both model sizes fall on what the developers call the Pareto frontier of the ViDoRe v3 benchmark, a curve showing the best accuracy achievable for a given model size. At a resolution of 2048-by-2048 pixels on an Nvidia L40S graphics processor, the 260-million-parameter model can encode about 51 document pages per second, roughly twice the speed of a comparable model called ColModernVBERT.

The models can also shrink how much space they need to store this document information. Using methods the developers call hierarchical token pooling and asymmetric quantization, NeoMME cuts the storage needed for one page from roughly 1.5 megabytes to about 6 kilobytes — 255 times smaller — while keeping more than 95% of the original search accuracy.

NeoMME supports a context length of 16,384 tokens, enough to process up to two standard 4K-resolution images at once, and was trained on multilingual text alongside images using a 131,000-word vocabulary. Each model processed about 524 billion tokens of training data in total, of which 290 billion came from text-only examples — a smaller training budget than the 2 trillion tokens used for the earlier ModernBERT model. Hcompany has released the model checkpoints on Hugging Face Transformers under the Apache 2.0 open-source license.