Natural Conversations with game NPC characters using AI

[SquirrelModeller]

Overview

For the past few weeks I've been working on a framework which simulates natural flowing conversations by simulating interruptions. What this means is that both you and the AI/LLM(s, yes more. Group have been coded) can interrupt each other. I've always wanted more realistic NPC's so that is why this project was created. Imagine having a causal conversation with an NPC or group of NPCs in a video game.

The source code is available here: https://github.com/SquirrelModeller/seamless-conversation

I call it a framework because it is coded in a way where it can "easily" be implemented for various purposes. For example tech support, someone to talk to, really whatever you want.

Currently there are a few quirks to work out. And a lot of features are missing. For example saving conversations (a database implementation). GPU usage because currently it all runs on the CPU. Multiprocessing for certain providers. The list goes on and on.

The providers are very bare bones. For speech to text there is either FasterWhisper or Vosk (both local models). ChatGPT as the llm provider and Elevenlabs as text to speech. I've attempted to make it simple to create providers for the project.

The final product would be a tool which can "perfectly" simulate conversations and output relevant information. It should be able to get any kind of context information like to inform the LLM of its surroundings/environment so it can react to it by utilizing given functions.

Modding side of things

I plan to integrate this into Skyrim using SKSE. I have only played around with SKSE. So, if anyone here has coding experience using SKSE or even for that sake papyrus and want to contribute to the project, that would be awesome!

I do not have experience managing any kind of big project, so help with that would be very much appreciated!

 

Q&A:

How does this differ from project like Mantella?

- This project focuses on the concept of interrupting each other.

- It was fun for me to code

Can I use it right now?

- For a normal conversation, yes..? But it is still being developed. Bug and errors are everywhere.

When will it be done?

- I am a computer science student. I work on the project in my free time. This is to say, it'll take a while if I am the only one working on the project.