Semantic Kernel Power AI Game Agent

[Spinnernicholas]

I created a game agent in a simulated world in the Godot game engine. Using Semantic Kernel I hooked the agent to chatgpt with function calling. The agent has a goal hierarchy system with several built-in hard coded goals and the ability to create custom natural language goals.

It needs a lot of work. I am currently refactoring the entire codebase.
Please provide feedback/ideas. Or Link information that might be useful.

Video Demo

YouTube video demo: https://www.youtube.com/watch?v=5ll6SfJw9Ms

Issues

• Current uses single chat history thread which is a waste of tokens and leads to repetition

Todo

• Split chat into smaller chats or individual chats per request
• Implement Memory
• Allow LLM to save goal templates
• Fine tuning the line between game AI and LLM
  • Abstract movement goals and use pathfinding?
• Inventory System
• Sending Subset of available functions to LLM based on context
• Switching between different LLMs depending on task
• Add app panels for interacting with/examining the simulation

Things I Learned

• Everything had to run async to prevent game loop from locking up. I throttled the async tasks that call GetChatMessageContentAsync so only one task can be active at a time and I can adjust the rate at which tasks are created by setting a max calls per minute.
• One of the big issues was handling the errors when reaching the OpenAI TPM(token per minute) limit. The exceptions are all aggregated inside the async task but the task doesn't fail and lock up my whole Semantic Kernel process. I added a loop where it waits a second and then checks for exceptions, cancelling the task and pausing new requests for 30 seconds. Splitting up the massive single chat may completely solve this. I will share it here because I have seen other people talking about this issue:

		var cts = new CancellationTokenSource();

		var task = chatCompletionService.GetChatMessageContentAsync(
			history,
			executionSettings: enableFunctions ? openAIPromptExecutionSettings : null,
			kernel: enableFunctions ? kernel : null,
			cancellationToken: cts.Token);

		var looping = true;

		while(looping)
		{
			//wait for the task to complete or timeout in 10 seconds
			await Task.WhenAny(task, Task.Delay(1000));

			if(task.Exception != null)
			{
				GD.Print("### Error\n" + task.Exception.InnerException.Message);
				cts.Cancel();
				await task;
				GD.Print("+++ Waiting 30 seconds before next query.");
				throttleTimer = 30;
				activeQuery = false;
				return null;
			} else if(task.IsCompleted) {
				looping = false;
			}

			GD.Print("### Waiting for LLM...");
		}

Questions:

• Is the vscode plugin obsolete?
• Easy way to easily change the plugins/functions available per GetChatMessageContentAsync call? Or should I just create a new Kernel each time?

[evchaki]

@Spinnernicholas - new function calling coming this week will help for get chat message async.

For the VS Code Extension - is there something you are looking for?

[Spinnernicholas]

@evchaki For the VS Code Extension, I was just going through all the docs, trying to learn the scope of Semantic Kernel. The extension appears to use handlebars(skprompt.txt...) for plugin development and I saw somewhere in the Semantic Kernel docs that it is not recommended to use handlebars because everything is being switched to OpenAI's format or something. I'm just trying to understand. My functions are all written as C# native plugins so it's not that important. But I do have template prompts stored in individual files that would be cool to test through the extension. The extension seems likes it's become out of sync with SK development and if that's the case, it probably just adds unnecessary confusion to perspective users.

[Spinnernicholas]

@evchaki Also, any insights into my second question?
Here's an (unrealistic) example that illustrates my problem:
Say I have a set of 10 functions. On each request, I want to provide the model with 3 function chosen at random. Is there a way to do that with a single kernel instance? Or should I just create a new kernel for each request?

There is a real world case for this. Say you have hundreds of functions. When a chat end user makes a request, you identify a substantially smaller subset of just the most relevant functions(identified using embeddings for example) and only pass those to the model.

[SergeyMenshykh]

Hi @Spinnernicholas, we are new releasing function calling model that should cover your scenario. We'll add an example demonstrating that.

[evchaki]

@Spinnernicholas - we are looking at updating the VS Code Extension next year to align with new patterns and tech that is on its way.

[SergeyMenshykh]

Hi @Spinnernicholas, we've just released the new function calling model. Here's the example - ContextDependentAdvertising demonstrating the way different functions provided to AI model depending on the context.