What it means to run an LLM on a microcontroller
Running an LLM on a microcontroller means fitting a full language model, including its parameters and inference code, into the tight flash, RAM, and compute budget of a tiny embedded chip so it can generate text offline without depending on a cloud server or external accelerator.
In this case, the microcontroller is an ESP32-S3, a Wi‑Fi and Bluetooth chip that usually ends up in smart plugs or hobby boards, not AI workstations. A developer has managed to fit a 28.9‑million‑parameter LLM onto it that can tell coherent stories and track variables. It runs entirely on‑chip at about 9 tokens per second, which is fast enough for simple prompts and short outputs on such limited hardware. This is a narrow, practical demo, not a replacement for cloud models, but it shows that edge AI inference with language models on microcontrollers is now within reach of hobbyists and device builders.
How per-layer embeddings make ESP32 language models possible
The core trick that makes this ESP32 language model work is a Gemma‑style method called per‑layer embeddings. In a typical LLM, most parameters sit in a huge embedding table, which you read from but do not repeatedly compute on. The developer keeps roughly 25 million parameters in a per‑layer embedding table stored in memory‑mapped flash, reading only around six rows per token, or about 450 bytes, while a smaller dense core of about 560,000 parameters lives in fast RAM.
This layout means the large table is almost "free" to run because it stays in flash and is only sampled as needed. The total model has six layers, a hidden size of 96, four attention heads, and a 32,768‑token vocabulary. Quoting the project’s description, “roughly 559,000 parameters form the dense core, about 3.1 million sit in the input embedding and output head, and about 25 million live in the Per‑Layer Embeddings table”. For embedded machine learning, this separation between lookup data and compute‑heavy weights is what makes a 28.9M‑parameter LLM on a microcontroller feasible.
Step-by-step: getting ESP32-AI running without the usual gotchas
Before touching any code, you need to know what you are signing up for. The exported model file is about 14.9 MB in size, which only fits on ESP32‑S3 boards with enough flash and external RAM. The project uses 16 MB of flash and 8 MB of OPI PSRAM, as found on specific DevKitC‑1 variants such as the N16R8V. A common 8 MB flash board will not hold the model and is a frequent source of weekend frustration. The main mistake people make is buying any “ESP32‑S3” board and assuming it is compatible; the second is expecting a general chatbot instead of a tiny story generator tuned for narrow tasks.
- Confirm your board’s hardware: check that your ESP32‑S3 board has 16 MB of flash and 8 MB of OPI PSRAM, matching known compatible variants like DevKitC‑1 N16R8V, because boards with only 8 MB of flash cannot store the 14,912,332‑byte model file.
- Clone the ESP32‑AI repository from its public hosting page and build the firmware using the documented toolchain so that the per‑layer embedding table is mapped into flash while the 560K‑parameter dense core is compiled into fast RAM.
- Flash the built firmware and model onto your ESP32‑S3, then power it from USB and open a serial terminal to send simple prompts so you can verify that text tokens are generated locally at around 9 tokens per second without any cloud connection.
Treat these steps as a proof‑of‑concept workflow, not a polished product pipeline. The open‑source implementation lets you adapt or retrain the model for your own tasks, but the build process can still feel fragile, and hardware details decide whether you get a working offline AI chip or a dead board and some confusion.
What this offline AI chip can and cannot do
Once running, the expected result is modest but useful: the ESP32‑AI model can tell simple English stories and keep track of variables in those stories. It runs entirely on‑chip, with no Wi‑Fi calls or external accelerators, so you get about 9 tokens per second of local text generation and full control over what data stays on the device. Offline inference avoids the constant need for connectivity, API calls, billing, and permission to send sensor data away from the device.
The demo is not a general assistant: it is closer to TinyStories‑style text generation than to answering open‑ended questions, writing code, or reasoning over live sensor streams. For embedded machine learning, this is enough for tasks like classification, command parsing, short structured responses, and narrow local text generation tasks where the model only needs to decide whether a reading deserves attention. You do not get frontier‑model behavior from an ESP32‑S3, but you do get a lightweight edge AI inference engine that can filter events, parse constrained commands, or decide when to wake a larger system.
Why it is worth trying and what to explore next
The main payoff of putting an LLM on a microcontroller is control. You get an offline AI chip that responds without network latency and keeps sensitive data on the device, which matters for factory sensors, field monitors, and consumer devices that must work when Wi‑Fi drops. It also proves that a microcontroller can carry more model‑shaped behavior than many hardware teams assumed, pushing the floor of edge AI lower than boards with NPUs or camera processors.
The ESP32‑AI project is open‑source, so you can download the code, inspect the per‑layer embeddings trick, and adapt it for your own applications, from command parsers to structured responses. This is still early: models must be trained for real device tasks, latency and power need careful measurement, and build pipelines have to become less fragile. For now, treat this as a guide and a starting point. If you keep expectations narrow and hardware checked, running a 28.9M‑parameter LLM on an ESP32‑S3 is both feasible and worth experimenting with for embedded projects.
