NPC Talk is an open-source toolkit that allows players to converse with non-player characters in video games using artificial intelligence and voice cloning technology.
The GTA V Showcase
The project was built first for Grand Theft Auto V, where players can walk up to anyone in Los Santos and just talk. When a player speaks, the NPC answers in character, in a cloned voice, about a second later, and acts on what was said.
In the game’s story mode, interactions range from Lamar roasting your hoodie to a cop telling you to file a civil claim and flipping you off. Michael defends his robe like it is a business suit, while every line is written live in character and spoken in that character’s cloned voice by Fish Audio Drama 3 or S2 Pro.
The system uses NVIDIA Nemotron ASR for transcription, Qwen 3.8 27B for real-time dialogue, and Fish Audio Drama 3 for cloned voices. Technical documentation reports a delay of 1.2 to 2.2 seconds for the NPC’s response. The GTA V example needs Windows, the .NET SDK (8+) to build it, and includes 19 characters, homes, and more.
How It Works
The modders gathered all the skills learned by Claude AI into one repository built on open models. Most of the code does not know it is in GTA, so the same pieces work in other games: the skills say what a game has to provide, and how to build the rest.
The agent starts with the make-npcs-talk skill and runs a loop that reconnoiters the game, picks engines, stands up the voice server, makes reference voices from recordings you have the rights to, and adds one NPC.
Inside a clone, .agents/skills and .claude/skills are copies of skills/, and AGENTS.md holds the instructions. Each agent gets the same skills in Agent Skills format, the fal MCP server, and the npctalk CLI. The toolkit includes skills for AI coding agents like Claude Code, Codex, Cursor, Gemini CLI, Copilot, and OpenCode.
The voice server includes drama3.py as the default model, s2pro.py with fine-tuned voices, keepwarm.py, and fal which runs S2 Pro on a realtime WebSocket. The npctalk CLI is a Python tool where every command has –help with examples.
Built for Modders and AI Agents
This is not a highly specialized mod but a tool, so other tech enthusiasts can try to adapt it for other games. Since the system relies on open models and a self-hosted voice server, users can set up a Fish Audio Drama 3 voice server on Modal.
The practical upshot is that you can make the villagers in your own game answer when you talk to them, in their own voices. Users need Git, Python 3.10+, uv, and ffmpeg to run the environment variables for accounts and keys.
Any tech enthusiast can install the tool anywhere else using the command uv tool install git+https://github.com/christos-antono/NPCTalk. The project is MIT licensed and has a Discord community for support.
Beyond GTA
The system records five 8-second clips of NPCs arguing that look like normal gameplay to create reference voices. Users can verify the voice input, characters, scene facts, and every step’s timing logged before showing it.
This approach demystifies how AI agents wire up games without needing highly specialized knowledge. The repository allows users to pick their agent and run a loop that measures the voices before claiming anything works.
Agent Skill Architecture and Configuration
The NPCTalk repository organizes every AI‑driven capability as a set of reusable “skills.” Each agent—whether Claude, Gemini, or a custom LLM—receives an identical collection of these skills in the standardized Agent Skills format, along with the fal MCP server and the npctalk CLI. Within a cloned repository the directories .agents/skills and .claude/skills are exact copies of the master skills/ folder, ensuring that the same functional blocks are available regardless of which model powers the conversation.
All high‑level instructions for assembling an agent live in AGENTS.md. This markdown file documents the step‑by‑step process for initializing the voice server, creating reference voice clips, and wiring the four interaction hooks that let an NPC receive typed or spoken input, generate a response, and trigger in‑game actions. The companion files CLAUDE.md and GEMINI.md simply point readers to AGENTS.md, reinforcing that the skill set is model‑agnostic.
When an agent is launched, the npctalk CLI reads the skill definitions, starts the fal MCP server, and executes the loop described in the documentation: recon the game, select engines (speech‑to‑text, LLM, voice), stand up the voice server, and finally attach a single NPC with its persona, reference voice, and scene facts. This modular design lets developers swap out the underlying language model or voice backend without rewriting the surrounding integration code.
Voice Server Architecture and Deployment
The NPCTalk voice server lives in the server/ directory and bundles several Python modules. drama3.py implements the Fish Audio Drama 3 model as the default TTS engine, while s2pro.py provides the S2 Pro model with fine‑tuned voices. Supporting scripts such as keepwarm.py keep the inference process ready, and the fal/ sub‑directory runs S2 Pro over a realtime WebSocket via the “fal” MCP server. A finetune/ folder contains the dataset, LoRA adapters, and the scorer that underlies the reported voice‑quality numbers.
Because the server is self‑hosted, users can run it on a serverless GPU platform such as Modal. The documentation notes that a Fish Audio Drama 3 voice server can be set up on Modal and then used to clone a reference voice from a file like narrator.wav. This deployment model keeps the heavy TTS inference off the game client and enables the sub‑second response times (1.2 – 2.2 seconds) described in the technical documentation.
Sources
- GitHub – christos-antono/NPCTalk: Make any NPC in any game talk back with Fish Audio: skills, a self-hosted voice server and tools for AI coding agents, with a full GTA V example.
- Modders Taught GTA V Characters to Respond to Players in Real-Time with Their Own Voices
- Fake Web3 Accelerator Uses npm Packages to Spread Malware and Hijack AI Agent Traffic
Photo: Wikideas1 (CC0)
