Understanding NVIDIA ACE and On-Device AI NPCs
NVIDIA ACE Game Agent SDK is an on-device agentic framework that lets developers build intelligent Unreal Engine AI NPCs and on-device AI companions which perceive, reason, and respond in real time by running speech, language, and decision models directly on RTX GPUs instead of remote cloud servers. It focuses on small, efficient models optimized for RTX game development and plugs into existing Unreal Engine 5 workflows. With the NVIDIA RTX Branch of Unreal Engine and the NVIDIA DLSS plugin, teams gain direct access to ray-traced lighting, frame generation, and AI-accelerated inference in a single pipeline. NVIDIA describes ACE as a lightweight, open-source C/C++ framework with Agent, Chat, and RAG APIs that share context and game state. These tools power examples like Ally in PUBG: BATTLEGROUNDS and the Total War: PHARAOH AI advisor, showing how AI character behavior can move beyond fixed scripts.
Setting Up Unreal Engine 5 with the RTX Branch and DLSS
To start, migrate or create your project using the NVIDIA RTX Branch of Unreal Engine so you can access RTX rendering and AI features. This branch integrates ray-traced lighting, frame generation, and the NVIDIA DLSS Unreal Engine plugin, giving you a consistent path for RTX game development and AI character behavior in the same build. Install the DLSS 4.5 plugin to enable Super Resolution and Dynamic Multi Frame Generation, which help offset the GPU cost of on-device AI by boosting frame rates. With Streamline-based integration, you can selectively enable DLSS features per platform or quality level. Once the RTX branch and DLSS are enabled in your project settings, confirm that your build runs on an RTX-capable GPU and profile performance. This baseline ensures your future AI NPC pipeline will share the same hardware-accelerated path as your graphics stack.
Installing NVIDIA ACE Unreal Engine Plugins
Next, add the NVIDIA ACE Unreal Engine 5 plugins that bring ready-to-use on-device AI capabilities into your project. Install the Automatic Speech Recognition plugin to get the nemo-conformer-ctc-120m English model and optional downloads for seven additional languages. This gives you live transcription and voice commands for AI companions. Add the Small Language Model plugin to enable local GGUF models such as the Qwen 3.5 4B SLM for low-latency dialogue and decision logic. Finally, install the Text-to-Speech plugin, which includes the Chatterbox Turbo 350M model and example voices to turn NPC responses into spoken audio. Each plugin supports both Blueprint and C++ integration and includes sample levels that demonstrate full speech-in, text reasoning, and speech-out loops. Because everything runs on-device, you avoid the latency and operating variability that come with cloud-based NPC systems.
Integrating the ACE Game Agent SDK into Gameplay
With the Unreal plugins installed, connect the NVIDIA ACE Game Agent SDK to your game logic. Use the Agent API when you want a stateful AI companion that owns conversation history, tracks tasks, and performs multistep reasoning with minimal application code. The agent can call tools that query game state, issue commands to AI controllers, or interact with the world. For granular control, use the Chat API to run stateless prompts from Blueprints or C++, ideal for one-off barks or system messages. To ground your AI NPCs in reliable knowledge, connect the RAG API to your own databases, such as quest logs, item catalogs, or balance tables. The Total War: PHARAOH advisor shows how RAG can query more than 1,200 linked tables to answer player questions; you can apply the same pattern to keep NPC advice accurate and current.
Designing Responsive AI Character Behavior
Finally, design AI character behavior loops that tie speech, reasoning, and animation into believable NPCs. In Unreal Engine, use Blueprints or C++ to route microphone input through ASR, send the transcript to your ACE agent, and then pass its response both to TTS and to gameplay systems. You can, for example, drive ally tactics, looting decisions, or navigation commands from the agent’s tool calls while simultaneously voicing responses through the TTS plugin. According to NVIDIA, Ally in PUBG: BATTLEGROUNDS uses natural voice interaction to interpret gameplay and responds dynamically while running fully on-device on GeForce RTX. To avoid infinite loops, gate AI actions with timers, state machines, and clear objectives, and keep your prompts grounded through RAG. With Unreal’s editor and Epic’s Model Context Protocol, you can extend this setup by experimenting with external models while keeping ACE as your low-latency runtime companion.
