LFM2.5-Encoder-350M — LiteRT

LiquidAI/LFM2.5-Encoder-350M converted to LiteRT (.tflite) for on-device inference. A multilingual (15 languages) bidirectional encoder on the LFM2 hybrid backbone — gated short-convolutions plus grouped-query attention — for embeddings, retrieval, classification heads, and masked-token prediction, fully offline on CPU.

Model description

File	Recipe	Size	Target
LFM2.5-Encoder-350M_wi8fc.tflite	int8 dynamic-range (linears + embedding, convs float)	376 MB	mobile + desktop
LFM2.5-Encoder-350M_fp16.tflite	fp16 weights, float compute	713 MB	desktop — XNNPACK's per-signature fp32 unpacking exceeds iPhone memory limits

All signatures take batch-1, right-padded static shapes: input_ids int32 [1, S] and attention_mask int32 [1, S] (1 = real token, 0 = pad).

Signature	Output
encode_64 / encode_128 / encode_256 / encode_512	last_hidden_state float32 [1, S, 1024], zeroed at padded positions
mlm_128	masked-LM logits float32 [1, 128, 65536]

Padded positions are fully masked inside the graph, in both the convolution path and attention, so the output at valid positions does not depend on how much padding follows: encode_64, encode_128 and encode_256 agree bitwise on the same sentence and match the unpadded PyTorch reference.

For a smaller sibling see LFM2.5-Encoder-230M.

How to use

1. Install dependencies

pip install ai-edge-litert numpy tokenizers huggingface_hub

2. Save the script below as embed.py:

#!/usr/bin/env python3
"""Embed sentences with litert-community/LFM2.5-Encoder-350M and rank them by similarity."""
import argparse

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

REPO = "litert-community/LFM2.5-Encoder-350M"
MODEL_FILE = "LFM2.5-Encoder-350M_wi8fc.tflite"


def embed(runner, tokenizer, text, seq_len):
    """Mean-pools the encoder states over the real tokens into one vector."""
    ids = tokenizer.encode(text).ids
    if len(ids) > seq_len:
        raise SystemExit(f"{len(ids)} tokens exceed --seq-len {seq_len}")
    input_ids = np.zeros((1, seq_len), np.int32)
    attention_mask = np.zeros((1, seq_len), np.int32)
    input_ids[0, : len(ids)] = ids
    attention_mask[0, : len(ids)] = 1
    states = list(runner(input_ids=input_ids, attention_mask=attention_mask).values())[0]
    vector = states[0, : len(ids)].mean(axis=0)
    return vector / np.linalg.norm(vector)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--query", required=True, help="The sentence to match.")
    parser.add_argument("--candidate", action="append", required=True,
                        help="A candidate sentence, repeatable.")
    parser.add_argument("--seq-len", type=int, default=128,
                        choices=[64, 128, 256, 512])
    parser.add_argument("--threads", type=int, default=8)
    args = parser.parse_args()

    model_path = hf_hub_download(REPO, MODEL_FILE)
    tokenizer = Tokenizer.from_file(hf_hub_download(REPO, "tokenizer.json"))
    interpreter = Interpreter(model_path=model_path, num_threads=args.threads)
    runner = interpreter.get_signature_runner(f"encode_{args.seq_len}")

    query = embed(runner, tokenizer, args.query, args.seq_len)
    scored = [(float(query @ embed(runner, tokenizer, c, args.seq_len)), c)
              for c in args.candidate]
    for score, text in sorted(scored, reverse=True):
        print(f"{score:6.3f}  {text}")


if __name__ == "__main__":
    main()

3. Run it

python embed.py --query "What is your refund policy?" \
  --candidate "Our refund policy allows 30 days" \
  --candidate "Steps to recover a forgotten login" \
  --candidate "The weather in Osaka is mild in spring"

 0.604  Our refund policy allows 30 days
 0.579  The weather in Osaka is mild in spring
 0.570  Steps to recover a forgotten login

Mean-pooling the raw encoder states is the simplest sentence representation and is what the numbers above use; for retrieval at quality you would normally train a pooling head or fine-tune on your own pairs. For masked-token prediction use the mlm_128 signature and read the logits at the [MASK] position.

On Android/iOS use the LiteRT runtime's SignatureRunner APIs with the same signature names; the tokenizer is the standard Hugging Face tokenizer.json, which the Rust/Swift/Kotlin tokenizers bindings all read.

Performance

int8 (wi8fc) file, CPU only.

Device	Threads	encode_128	encode_512	mlm_128
Apple M4 Max (macOS)	8	36.5 ms	111.2 ms	44.7 ms
iPhone 17 Pro	6	42 ms	145 ms	73 ms

Mac figures are the median of 20 warm runs (ai-edge-litert 2.1.6, XNNPACK, otherwise idle machine). iPhone figures come from the on-device gate (TFLite C API + SignatureRunner + XNNPACK); each is the last of three consecutive measurements, not a median.

Budget for one slow first call. The first inference after loading pays a one-time graph preparation. On the Mac that first call took 908 ms against a 36.5 ms steady state; on the iPhone the three consecutive encode_128 measurements were 102, 51 and 42 ms, and the three encode_512 measurements were 229, 152 and 145 ms. Model load was 1.48 s on the iPhone, with a peak footprint of about 1.65 GiB.

Those three consecutive values are not a language effect — the signatures are fixed-shape, so every input of a given signature costs the same. Measured warm on the Mac with the run order reversed, one signature takes 36.4 / 37.3 / 36.6 ms on English, Japanese and Arabic sentences of 17, 21 and 27 tokens.

Thread count matters more than anything else here: at the interpreter default this model measures 52.4 ms and 227.1 ms for encode_128 and encode_512, against 36.5 ms and 111.2 ms at 8 threads.

Accuracy note

Parity against the PyTorch fp32 reference over 16 sentences covering all 15 supported languages — mean-pooled sentence-embedding cosine against the original Lfm2BidirectionalModel, plus top-5 fill-mask agreement on English, French, German and Japanese cloze prompts:

Variant	Pooled cosine (min / mean)	Per-token correlation (min)	Fill-mask
fp16	1.000000 / 1.000000	1.000000	top-5 sets identical (4/4 prompts)
int8 (wi8fc)	0.996563 / 0.998592	0.990448	top-1 on 3/4, at least 3/5 top-5 overlap on all

On the iPhone 17 Pro the int8 file reproduces the Mac outputs bit-exactly — cosine 1.000000, max absolute difference 0.0 — across every tested language and signature.

License

LFM Open License v1.0 (see LICENSE, unchanged from the base model). Note the license's commercial-use threshold (Section 5). This repository redistributes converted Derivative Works of LiquidAI/LFM2.5-Encoder-350M with modification notices per Section 4; all credit for the model to Liquid AI.