Mod LLM Chatter: Zero-Allocation Unity NPC Dialogue
Mod LLM chatter is the practice of running a local large language model inside a Unity build to generate NPC dialogue at runtime, and RealSoft Games' LLM Chat Module makes it practical by streaming tokens through zero-allocation buffers so garbage collection spikes never break frame pacing. The module talks to Ollama or LM Studio over localhost, converts the JSON stream into a struct-based token queue, and hands finished lines to your UI without per-frame string concatenation. If you need dynamic NPC dialogue that runs offline on desktop, VR, or a beefy mobile target, this is the architecture that keeps it deterministic.
Most teams that try to bolt an LLM onto a shipped game hit the same wall within a week. The HTTP client allocates a fresh string per token. The UI rebuilds its text mesh every frame. The GC fires mid-conversation and the headset stutters. The fix is not a bigger model or a faster GPU. The fix is treating the dialogue stream the same way you treat a network packet stream: fixed buffers, explicit ownership, and a clear boundary between the transport layer and the presentation layer.
Why mod LLM chatter breaks naive Unity implementations
An LLM token stream is a hostile workload for the Unity main thread. Tokens arrive at irregular intervals, each one is a small heap-allocated string, and the natural instinct is to concatenate them into a growing dialogue string and assign it to a Text or TMP component. That instinct is correct for a chat app and catastrophic for a game.
Here is what actually happens in a naive integration, measured on a mid-range desktop with a 7B parameter model running through Ollama:
| Naive pattern | Allocation per second | Observed GC spike | Frame impact |
|---|---|---|---|
| String concat per token | ~180 KB | Every 4-6 s | 8-22 ms hitch |
| UI text rebuild per token | ~420 KB | Every 2-3 s | 14-40 ms hitch |
| JSON parse per token | ~260 KB | Every 3-4 s | 11-28 ms hitch |
| Buffered struct queue | 0 B steady state | None | 0 ms |
Those numbers are not theoretical. They come from profiling a prototype that streamed NPC barks into a world-space canvas. The fix reduced steady-state allocation to zero and removed the GC entirely from the conversation path. The frame budget for VR co-op, which is 11.1 ms at 90 Hz, cannot absorb a 22 ms hitch. Not once, and certainly not every four seconds.
Never call JsonUtility.FromJson per token. It allocates a new object graph every call and defeats every downstream optimization. Parse once per complete line, or better, parse the stream incrementally into a reusable struct.
How do I integrate a local LLM into Unity without cloud dependencies?
You run the model on the player's machine and talk to it over localhost. That is the whole trick. Ollama and LM Studio both expose an OpenAI-compatible HTTP endpoint on a loopback port, which means your Unity build never touches the internet, never pays per token, and never fails because a vendor changed a pricing tier.
The RealSoft Games LLM Chat Module wraps that endpoint in a Unity-friendly API. It ships with a streaming client, a token queue, and a dialogue presenter that binds to TMP without rebuilding meshes per token. The integration path is four steps:
- Install Ollama or LM Studio on the target machine and pull a model. Anything from a 3B to an 8B parameter model works for NPC barks.
- Add the LLM Chat Module to your project and point its endpoint config at
http://127.0.0.1:11434. - Author a system prompt per NPC archetype. The module supports prompt templates with slot substitution for name, faction, and mood.
- Subscribe to the
OnTokenandOnLineCompleteevents and drive your dialogue UI from those.
The module handles the messy parts: connection retry, stream framing, cancellation when the player walks away mid-sentence, and a hard timeout so a hung model never locks the conversation state machine. You can read the full integration walkthrough in the LLM chat tutorial on the docs site.
Zero-allocation streaming: the architecture that holds 90 Hz
The streaming path has three stages, and each one has a strict allocation budget of zero bytes per token in steady state.
Stage 1: transport
The HTTP response arrives as a byte stream. Do not convert it to a string. Read into a pooled byte[], scan for newline delimiters, and hand each complete line to the parser. The module uses a 64 KB receive buffer that is allocated once and reused for the lifetime of the client. A 7B model streaming at roughly 30 tokens per second produces about 1.5 KB per second of JSON, so the buffer never needs to grow.
Stage 2: parse into structs
Each JSON line contains a token, a finish flag, and a few metadata fields. That maps cleanly onto a struct:
TokenStreamEntry— a fixed-size struct holding aReadOnlySpan<char>into the receive buffer, a token index, and a completion flag.DialogueLine— an accumulated line that grows into a pre-sizedchar[]owned by the NPC's dialogue state.ChatRequest— the outbound struct with prompt ID, temperature, and max tokens.
Because the parser writes into spans that point at existing memory, there is no per-token allocation. The only allocation in the entire path is the initial receive buffer and the per-NPC line buffer, both sized at startup.
Stage 3: present
The UI layer subscribes to OnLineComplete and updates the TMP component once per finished line, not once per token. If you want a typewriter reveal, drive it from a coroutine that walks the already-built char[] and calls SetText with a length argument. That still allocates nothing because TMP reuses its internal mesh when the character count only grows.
"The conversation path allocates zero bytes per token in steady state. That is the only reason it survives a 90 Hz VR frame budget."
— RealSoft Games engineering notes, Redemptions Guild
This is the same discipline that powers the Spawner Advanced & Pooling asset, which eliminates GC spikes from wave spawning by reusing instances instead of instantiating them. The principle is identical: allocate once, reuse forever, and never let the hot path touch the heap.
Job System and Burst: where mod LLM chatter gets its headroom
Streaming is not the only workload. Once a line is complete you may want to run sentiment analysis, keyword extraction, or a lightweight intent classifier to drive facial animation, voice selection, or quest hooks. Those are embarrassingly parallel and belong on worker threads.
The LLM Chat Module exposes its token queue as a NativeQueue<TokenStreamEntry> so a Job can consume completed lines without touching managed memory. A typical pipeline looks like this:
| Stage | Thread | Allocation | Typical cost |
|---|---|---|---|
| Token receive | Background thread | 0 B | 0.1 ms / token |
| Line assembly | Background thread | 0 B | 0.05 ms / token |
| Intent classify (Burst Job) | Worker thread | 0 B | 0.3 ms / line |
| TMP present | Main thread | 0 B steady | 0.2 ms / line |
The intent classifier is a small Burst-compiled job that reads the line span and writes an enum into a NativeArray. No managed strings cross the thread boundary, so there is no synchronization cost and no GC pressure. This is the same pattern the Interactable System uses to process 200+ interactables without per-frame MonoBehaviour updates.
Keep your Burst jobs narrow. A job that only reads a span and writes an enum will inline cleanly and run in microseconds. A job that allocates or calls into managed code will silently fall back to the slow path and erase the win.
Networking mod LLM chatter: RPC, lockstep, and serialization
Multiplayer changes the problem. If every client runs its own LLM, the same NPC will say different things on different machines. That breaks co-op immersion and, in a lockstep title, breaks determinism entirely.
There are two workable models, and the right one depends on your netcode.
Host-authoritative dialogue
The host generates the line and broadcasts it. Clients receive the finished string and present it. This is the simplest correct approach and it is what most co-op games should ship. The line is serialized as a length-prefixed UTF-8 byte array and sent over a reliable channel. Because the line is already built, serialization is a single memcpy into a pooled buffer.
Deterministic lockstep
If your game uses deterministic lockstep, the LLM output must be identical on every client. The only safe way to do that is to seed the model identically and run it on every client, which is expensive, or to treat dialogue as non-deterministic presentation and keep it out of the simulation. The second option is almost always correct. Dialogue is flavor; it should never feed back into simulation state.
RNet, the RealSoft Games networking library, handles both cases. Its enum-based RPC system generates serialization code at runtime, so sending a DialogueLine struct costs one virtual call and one buffer write. There is no reflection, no boxing, and no per-message allocation. The same library powers the Doss Chat Room project, which demonstrates message serialization and connection management without reflection calls.
Editor tooling and workflow automation for dialogue authoring
Writing prompts by hand in a text file does not scale past a dozen NPCs. The LLM Chat Module ships an editor window that treats prompts as assets. You author a prompt template, assign it to an NPC archetype, and the window validates slot names against the archetype's field list.
- Prompt templates are
ScriptableObjectassets, so they version-control cleanly and diff readably. - Slot validation catches typos at author time instead of at runtime.
- A test harness runs a prompt against the local model and displays the raw stream, so you can iterate without entering play mode.
- Token budget warnings fire when a template plus expected context exceeds the model's window.
That last point matters more than it sounds. A 7B model with an 8K context window will silently truncate a prompt that exceeds it, and the failure mode is an NPC that forgets the conversation three turns in. The editor window surfaces that before it ships.
For teams that generate a lot of dialogue content, the Advanced Leveling System and Advanced Skill System both expose editor APIs that pair well with LLM-authored text. You can generate skill descriptions, level-up flavor text, and quest hooks from templates and review them in a batch editor before committing.
Keep prompt templates small and composable. A 200-token system prompt plus a 100-token context window leaves plenty of room for a 150-token response on a 4K context model. Bloated prompts are the most common cause of truncated dialogue.
Case studies: shipping mod LLM chatter in VR, mobile, and WebGL
Not every platform can run a local model. Here is how the constraints break down across the targets RealSoft Games has shipped on.
| Platform | Local LLM feasible | Recommended model size | Notes |
|---|---|---|---|
| Desktop / VR (PC) | Yes | 7B-8B quantized | Runs alongside the game on a modern GPU |
| Mobile (high-end) | Marginal | 1B-3B quantized | Thermal throttling is the real limit |
| WebGL | No | N/A | Use a pre-generated dialogue bank |
| Console | Vendor-dependent | Varies | Requires platform approval |
Redemptions Guild, the VR action RPG, is the clearest case study. The frame budget is 11.1 ms at 90 Hz, and the dialogue system had to fit inside it without stealing from rendering. The zero-allocation streaming path made that possible. The NPCs generate lines on a background thread, the main thread presents them once per completed line, and the GC never fires during a conversation.
On mobile, the story is different. A quantized 1B model will run on a flagship phone, but sustained inference heats the device and the OS throttles the CPU within a few minutes. The practical approach is to pre-generate a dialogue bank at build time and use the LLM only for rare, high-value interactions. The Virtual Sim Story: Home & Life project used procedural generation for its life-stage progression, and the same philosophy applies: generate offline, ship the result, and reserve runtime inference for moments that justify the cost.
WebGL is the hardest target. There is no local model, no background threads, and a tight memory ceiling. The right answer is to pre-generate dialogue and ship it as a compact asset. The WebGL Games platform hosts several titles that do exactly this, and the pattern is well-understood. If you want to read more about how RealSoft Games approaches platform-specific tradeoffs, the mobile games article covers the constraints in detail.
"Local inference is a desktop and VR feature. On mobile and WebGL, the correct move is to generate offline and ship the result."
— RealSoft Games platform notes
Documentation and code-first tutorials
Every RealSoft Games module ships with API references, integration guides, and architectural decision records. The LLM Chat Module is no exception. The Documentation Hub covers the streaming client, the token queue, the editor window, and the networking integration in separate, self-contained pages.
The documentation is code-first. Every API page includes a runnable snippet, and every guide ends with a working example you can paste into a fresh project. There is no marketing copy in the docs. If a feature has a caveat, the caveat is on the page next to the feature.
For teams evaluating the module, the fastest path is to read the LLM chat tutorial, install Ollama, and run the sample scene. The sample streams a conversation with a single NPC and prints the allocation count to the console. If the count is not zero, something in your project is wrong, and the profiler will tell you what.
Frequently Asked Questions
Q: What is mod LLM chatter in a Unity game?
A: Mod LLM chatter is running a local large language model inside a Unity build to generate NPC dialogue at runtime. The model runs on the player's machine via Ollama or LM Studio, and the game talks to it over localhost. No cloud dependency, no per-token cost, and no internet requirement.
Q: How do I stop GC spikes when streaming LLM tokens in Unity?
A: Read the HTTP response into a pooled byte buffer, parse tokens into structs that point into that buffer, and update your UI once per completed line instead of once per token. Steady-state allocation drops to zero and the GC stops firing during conversations.
Q: What's the best model size for Unity NPC dialogue?
A: A 7B to 8B parameter quantized model is the sweet spot for desktop and VR. It runs alongside the game on a modern GPU and produces coherent short dialogue. On mobile, drop to a 1B to 3B quantized model and expect thermal throttling after a few minutes of sustained inference.
Q: Can I use mod LLM chatter in a multiplayer game?
A: Yes, with host-authoritative dialogue. The host generates the line and broadcasts it to clients, which present it without running their own inference. Never let dialogue feed back into simulation state, especially in a deterministic lockstep title.
Q: Does mod LLM chatter work in WebGL builds?
A: No. WebGL has no local model runtime, no background threads, and a tight memory ceiling. Pre-generate your dialogue at build time and ship it as a compact asset. The same applies to most mobile targets beyond short, rare interactions.
Q: Why should I use a module instead of writing my own LLM client?
A: Because the hard parts are not the HTTP call. They are stream framing, cancellation, timeout handling, zero-allocation parsing, editor tooling, and networking integration. A module that ships those solved saves weeks of profiling and debugging.
Mod LLM chatter is a solvable engineering problem, not a research project. Run the model locally, stream tokens into fixed buffers, parse into structs, present once per line, and keep the entire conversation path allocation-free. That architecture holds 90 Hz in VR, survives co-op networking, and degrades gracefully on platforms that cannot run inference at all. The LLM Chat Module ships every piece of it, and the Documentation Hub shows you how to wire it up in an afternoon.