Instructions to use litert-community/LFM2.5-2.6B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/LFM2.5-2.6B with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/LFM2.5-2.6B \ --prompt="Write me a poem"
- LiteRT
How to use litert-community/LFM2.5-2.6B with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).
Measured on device (edge-compat, lfm2.5-2.6b-int4): Galaxy S26 Β· LiteRT-LM 0.16.0 Β· GPU Β· decode 25.3 tok/s Β· prefill 297 tok/s Β· TTFT 730 ms Β· all 936 ops delegated (2026-08-24); Raspberry Pi 5 Β· LiteRT-LM 0.16.1 Β· CPU, 4 threads Β· decode 4.9 tok/s Β· prefill 28 tok/s Β· TTFT 9.19 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-2.6b-int4/CARD.md
Measured on device (edge-compat, lfm2.5-2.6b-int8): Galaxy S26 Β· LiteRT-LM 0.16.0 Β· GPU Β· decode 19.7 tok/s Β· prefill 489 tok/s Β· TTFT 470 ms Β· all 936 ops delegated (2026-08-24); Galaxy S26 Β· LiteRT-LM 0.16.0 Β· CPU Β· decode 15.5 tok/s Β· prefill 222 tok/s Β· TTFT 990 ms (2026-09-06); Raspberry Pi 5 Β· LiteRT-LM 0.16.1 Β· CPU, 4 threads Β· decode 3.5 tok/s Β· prefill 44 tok/s Β· TTFT 6.13 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/lfm2.5-2.6b-int8/CARD.md
LFM2.5-2.6B β LiteRT-LM
Try it in your browser: the LiteRT-LM Web chat demo loads LFM2.5-2.6B_int4.litertlm (1.67 GB download) and runs it with WebGPU.
LiquidAI/LFM2.5-2.6B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (earliest verified runtime for the current files: litert-lm 0.19.0; for their Android GPU path, see Known runtime issues).
LFM2.5-2.6B is the flagship of Liquid AI's LFM2.5 on-device family: a hybrid architecture (22 gated short-convolution blocks + 8 grouped-query attention layers) and a thinking model β it reasons inside a <think>β¦</think> block before answering. The file declares the thought channel, so LiteRT-LM apps receive the reasoning and the answer as separate streams.
| File | Recipe | Size | GSM8K (n=100) |
|---|---|---|---|
LFM2.5-2.6B_int8.litertlm |
int8 dynamic (linears + convs + embedding) | 2.87 GB | 88% (bf16 reference: 92%) |
LFM2.5-2.6B_int4.litertlm |
int4 blockwise-32 + OCTAV linears, int8 embedding, convs float | 1.67 GB | 83% |
2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.
| Context (KV cache) | 4096 max (see --max-num-tokens note) |
| Backend | CPU, and GPU (for the Android GPU path, see Known runtime issues); iOS Metal still fails at engine creation, tracked upstream in LiteRT-LM#3129 |
| Template | in the file β ChatML-style with tool-list support; generation prompt pre-fills <think> (the model's trained convention); thought channel declared |
| Base model | LiquidAI/LFM2.5-2.6B (LFM Open License v1.0) |
Quality β GSM8K
GSM8K, greedy, 0-shot chain-of-thought, max-tokens 2048 (thinking model), n=100, same evaluation script for all rows:
| Configuration | GSM8K |
|---|---|
| PyTorch bf16 (reference, MPS) | 92% |
| LiteRT int8 (this repo) | 88% |
| LiteRT int4-b32 OCTAV (this repo) | 83% |
Both files pass an 8-question sanity check 8/8 with zero degenerate outputs (CPU), a 42-length prefill sweep with zero corrupt first tokens (fresh engine per length), and a 3-turn conversation check (fact recall + arithmetic).
Usage
pip install litert-lm
litert-lm run ./LFM2.5-2.6B_int4.litertlm --prompt "What is 17 + 25?"
The model thinks before answering β through the Python/C++ API the reasoning arrives on the thought channel and the reply stays clean; the CLI prints both.
--max-num-tokens tip: reasoning turns are long β give the model at least 2048 tokens of budget for math/complex questions, or the think block may not close. Decode speed drops as the token budget grows, so don't set it higher than you need.
Multi-turn note: on the current runtime the conversation context retains previous turns' reasoning, so long chats fill the context faster than the visible text suggests. For unrelated questions, prefer fresh conversations, each on a new Engine (see One Engine per conversation below).
One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's short-convolution state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). In a test that asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac), the _int8 file gave the same three one-word answers on both Engines, in its one run. The _int4 file was run twice (a test is run a second time when its first run shows a difference or when the fresh Engine misses one of the three expected answers: Paris, 391, blue): in its first run the arithmetic answer came back as an empty reply on the fresh Engine; in its second run the same three one-word answers came back on both Engines. Those three short questions say nothing about longer conversations. Create a new Engine for each conversation. The first conversation on an Engine is unaffected. A fix landed on LiteRT-LM main on 2026-11-03 (e1f86b05). No release carries it yet (v0.19.1 does not); this note will be updated when one does.
Tool calls on litert-lm 0.18.0. Since the 2026-10-13 re-pack (metadata only: llm_model_type lfm2, and the chat template now renders tool calls and tool results), the runtime parses the LFM2 tool call and the tool result reaches the model; the earlier files returned the call as message text (<|tool_call_start|>[get_weather(location="Paris")]<|tool_call_end|>). The lfm2 model type needs litert-lm β₯ 0.15.0; the tool-call parser is 0.18.0's β on 0.15β0.17 the call still arrives as text (read from the runtime source, not run here). The 2026-10-13 re-pack changes the model type, and with it the runtime's GPU executor defaults (every non-generic LiteRT-LM model type shares them: allow_src_quantized_fc_conv_ops on, OpenCL hint_waiting_for_completion off, kernel batch size unset); the phone GPU rows on this card were measured with the previous files and have not been re-run β the Mac GPU digit check passed on 0.18.0.
Replies that take over 10 minutes. On litert-lm 0.18.x and 0.19.0, a blocking send_message that is still generating after 10 minutes fails with DEADLINE_EXCEEDED (a fixed timeout with no setting), and the generation it leaves running can crash the process. For long replies on a slow backend (long thinking in particular), lower max_output_tokens or use the streaming send_message_async, which has no such limit.
Speed
litert-lm benchmark, CPU backend, Mac M4 Max, max-num-tokens 1024 (prefill-1024 rows measured at max-num-tokens 2048):
| Variant | Prefill (256) | Prefill (1024) | Decode | TTFT |
|---|---|---|---|---|
| int8 | 169 tok/s | 434 tok/s | 38.0 tok/s | 1.54 s |
| int4 | 156 tok/s | 193 tok/s | 43.7 tok/s | 1.66 s |
GPU backend (litert-lm benchmark β¦ --backend gpu --cache no, litert-lm 0.16.0, same machine; both variants verified to generate correct text on GPU before quoting):
| Variant | Prefill (256) | Decode |
|---|---|---|
| int8 | 1893 tok/s | 125.6 tok/s |
| int4 | 1972 tok/s | 161.6 tok/s |
For the Android GPU path of both files, see Known runtime issues. Pixel 8a (Tensor G3), litert_lm_main v0.16.0 built from the release tag, 261-token prompt with generation capped at 256, 3 runs per backend (an earlier int4 file of this repo):
| Backend | Prefill (261 tok) | Decode | TTFT |
|---|---|---|---|
| GPU (OpenCL) | 87.8β88.4 tok/s | 11.0β11.2 tok/s | 3.04β3.06 s |
| CPU | 16.2β30.2 tok/s | 7.4β10.2 tok/s | 8.7β16.3 s |
At this size the GPU won across the board on that phone β ~3β5Γ prefill, ~3β5Γ faster to the first token, and a steadier decode (the CPU rows spread because they throttle; the GPU rows repeat to within 2%) β so prefer the GPU backend where the runtime offers it. A short prompt shifted the ratio further toward the GPU: with a 19-token prompt the same file prefilled at 37.7β38.8 tok/s on GPU against 4.1 tok/s on CPU.
iPhone 17 Pro (CPU backend, cold first runs): the int4 file answers the 8-question quality check 8/8 and decodes at ~20 tok/s warm (the first, cold run is slower while caches build). The int8 file also loads and runs on iPhone 17 Pro when the host app carries the extended-virtual-addressing / increased-memory entitlements (its 2.87 GB single weight section exceeds what a default-entitlement app will memory-map) β int4 is the recommended phone variant (smaller, faster on phone-class memory bandwidth).
Galaxy S26 β GPU backend
This section describes earlier files of this repo, the ones it held when they were checked (LFM2.5-2.6B_int4.litertlm on 2026-08-24, LFM2.5-2.6B_int8.litertlm on 2026-08-25): both ran on the Android GPU backend, LiteRT took the whole graph in each, and the model generated. For the Android GPU path of the current files, see Known runtime issues.
| file | GPU backend | delegation | peak |
|---|---|---|---|
LFM2.5-2.6B_int4.litertlm |
ran | 11132 / 11132 ops across 12 subgraphs on LiteRT GPU |
1231 MiB |
LFM2.5-2.6B_int8.litertlm |
ran | 11132 / 11132 ops across 12 subgraphs on LiteRT GPU |
1189 MiB |
Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Checked 2026-08-24 and 2026-08-25.
Speed. These checks measured delegation, generation and peak memory, not speed. Speeds measured on this handset with earlier files of this repo are in the Galaxy S26 rows of litertlm_manifest.json (GPU: int4 on 2026-08-24, int8 on 2026-08-25; CPU: int8 on 2026-09-06).
GPU wiring, including the Gallery import toggle: GPU guide.
Run on Android
Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat. For the Android GPU (litert_lm_main --backend=gpu, or the SDKs) with both files, see Known runtime issues; the Gallery app bundles its own runtime β if its GPU toggle fails, use CPU there.
In a Flutter app, flutter_edge_ai_litertlm (1.14.0) installs this repo's file through FlutterEdgeAi.resolveHuggingFace('litert-community/LFM2.5-2.6B'), which reads the repo's litertlm_manifest.json and picks the file and backend for the device; in React Native, react-native-litert-lm (0.8.0) does the same through resolveFromManifest(). Neither route is device-verified by this card; each package's README states its tested platforms.
Conversion notes
Converted with released litert-torch 0.9.2 (upstream lfm2 hybrid support incl. the ShortConv prefill-pad fix) and packaged for litert-lm β₯ 0.14 (β₯ 0.15 since the 2026-10-13 re-pack; the executor-metadata section binds the 22 conv states + 16 KV caches, and files exported without it do not run on 0.15). The current files have an earliest verified runtime of 0.19.0.
- The generation prompt pre-fills
<think>exactly as the vendor chat template does. This matters: with a bareassistantprompt, think-block emission becomes the model's choice, and the int4 quantization degrades that discipline first (unscaffolded rambling in place of answers on multi-turn). With the pre-fill, all variants answer tersely and the thought channel routes cleanly. - The checkpoint is notably sparse: OCTAV int4-b32 produced 746k all-zero weight blocks whose zero scales the XNNPACK delegate rejects at load. The zero scales are patched to the tensor's smallest nonzero scale (dequantization unchanged β the blocks are all zeros).
- int8 quantizes convs at export time (safe); post-hoc conv int8 breaks generation. int4 keeps convs float.
- Multi-length prefill signatures (1β1024) are exported so the runtime picks tight chunks; a 42-length first-token sweep verifies no prefill-padding state corruption.
Raspberry Pi 5 (CPU)
Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (minβmax in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|---|---|---|---|
LFM2.5-2.6B_int4.litertlm |
28.5 (28.5β28.5) | 4.9 (4.9β4.9) | 9.2 s | 2.9 GB |
LFM2.5-2.6B_int8.litertlm |
43.8 (43.6β43.9) | 3.5 (3.5β3.5) | 6.1 s | 3.9 GB |
Known runtime issues
Measurements of LFM2.5-2.6B_int4.litertlm on this page made before 2026-11-09 used earlier files, not the current one. Measurements of LFM2.5-2.6B_int8.litertlm on this page made before 2026-10-30 used earlier files, not the current one.
Android GPU with LiteRT-LM v0.18.0 and v0.18.1. LFM2.5-2.6B_int4.litertlm and LFM2.5-2.6B_int8.litertlm keep some of their weights outside the model graph (LiteRT's external weights), and on a Pixel 8a the GPU backend of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on a Qwen3-0.6B file with external weights (LiteRT-LM#3986).
Long greedy replies on the GPU vary between runs. On the GPU's default path, where the sampler writes the inputs of the next decode step, a long greedy reply can differ from one process to the next. Measured on LFM2.5-2.6B_int4.litertlm with a question asking for a ~400-word essay (the thought and the essay, about 9,200 characters), for the file at revision d471a46 with the weights inside the graph and for the current file: on a Mac (Apple M4 Max, WebGPU), LiteRT-LM built from source (commit dab2e708) gave another reply in 2 of 8 runs of each file, and the PyPI litert-lm 0.19.1 CLI in 3 of 15 runs of the earlier file and 2 of 15 of the current one; on a Pixel 11 Pro GPU (PyPI litert-lm-api 0.19.0), 6 runs gave 3 different replies with the earlier file and 2 with the current one, both of them replies of the earlier file. With --sampler_handles_input=false (litert_lm_advanced_main built from source; the Python, Kotlin and Swift APIs of 0.19.1 have no such setting), both files gave one and the same reply in 8 of 8 runs.
From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.
Structured output: LiteRT-LM refuses constrained decoding for files of llm_model_type lfm2 when the conversation is created (Constrained decoding is not supported for Lfm2DataProcessor.), so such a file does not take a JSON ResponseFormat. That was measured on LFM2.5-1.2B-Instruct_int4.litertlm from LFM2.5-1.2B-Instruct with litert-lm 0.16.0 and 0.19.1 on a Mac CPU. Both files here carry llm_model_type lfm2; they were not run with constrained decoding.
On LiteRT-LM 0.18.1 and 0.19.1, once a message fails because it does not fit in the context window, every later message in that conversation fails as well (the next one with Task failed with state: 7). Start a new conversation after such an error. For these files, start it on a new Engine (see One Engine per conversation above).
- Downloads last month
- 1,439