Architecture

Unreal Neural Director is a single Unreal Engine module (UnrealNeuralDirector) that implements a fully recursive agentic AI loop with tool execution, knowledge injection, and multi-provider streaming. This page explains how a user message travels through the system and comes back as editor actions.

Core Data Flow

Every user message follows this path through the system:

The loop is fully recursive: after each tool execution, results are appended to the conversation and a new provider request is made. This continues until the AI calls attempt_completion or the turn budget is exhausted.

Agent Loop

The FUNDAgentLoop class is the heart of UND. It manages the recursive tool-use cycle and implements several safety mechanisms.

Turn Budget

Each user message gets a budget of tool-call round-trips (default: 100 turns, configurable via MaxAgentTurns). At 75% of the budget, a warning is injected into the conversation so the AI can wrap up its work. If the budget is fully exhausted, the loop stops.

Cycle Detection

UND tracks recent tool call hashes and detects when the AI is stuck in a repetitive loop. If the same tool call pattern repeats more than 3 times consecutively, the loop breaks to prevent infinite cycles.

Self-Healing

When a tool execution produces a compile error or Python traceback, UND increments an error retry counter and re-submits the conversation with the error context. The AI sees what went wrong and can fix it. The counter resets only when the relevant tool succeeds without errors, preventing the AI from bypassing the retry limit by interleaving unrelated tool calls. Maximum consecutive retries: 5.

Nesting Depth

UND supports multi-agent delegation. The delegate_to_agent tool creates a sub-task with an isolated conversation context. Sub-agents can delegate further, up to a maximum nesting depth of 3 levels. Only the attempt_completion output from a sub-agent returns to its parent.

Built-in agent profiles:

Profile	Purpose
Orchestrator	Breaks complex tasks into sub-tasks and delegates
Coder	General-purpose implementation (default)
Architect	System design and planning
Reviewer	Code review and quality checks
LevelDesigner	Level layout and environment work

Provider System

UND supports three AI provider backends through a common interface.

Provider Interface

IUNDProvider defines the contract:

• StreamRequest() -- Non-blocking. Starts an SSE stream and delivers results through callbacks on the game thread.
• CancelRequest() -- Cancels an in-progress stream.
• IsConfigured() -- Returns whether the provider has valid credentials.
• GetMaxContextTokens() -- Returns the model's context window size.

Supported Providers

Provider	API	Notes
Anthropic	Messages API with SSE	Claude models, supports extended thinking
OpenAI-Compatible	Chat Completions API	Covers OpenAI, OpenRouter, Ollama, LM Studio, Mistral, Groq, xAI, and any compatible endpoint
Gemini	generateContent API with SSE	Google Gemini models

Lifetime Safety

Providers use a TSharedPtr<bool> bValid guard pattern. When a provider is destroyed (e.g., due to settings change during a request), the destructor sets *bValid = false. All async callbacks hold a shared pointer copy and check this flag before accessing the provider, preventing use-after-free crashes.

SSE Streaming

The FUNDSSEStream class handles HTTP streaming on a background thread. It batches incoming chunks and flushes them to the game thread at approximately 60fps (every ~16ms or at 4KB thresholds). This prevents flooding the game thread with thousands of tiny async tasks while maintaining responsive streaming output.

Tool System

UND ships with 31 built-in tools organized into categories.

Tool Interface

Every tool implements IUNDTool with these key methods:

• GetName() -- Snake_case identifier (e.g., execute_python, read_file)
• GetDescription() -- Human-readable description sent to the AI
• GetInputSchemaJson() -- JSON Schema defining expected parameters
• Execute() -- Runs the tool and returns a result (Ok, Error, or Done)
• IsReadOnly() -- Whether the tool has no side effects
• IsDestructive() -- Whether the tool requires extra caution
• IsThreadSafe() -- Whether the tool can run on a background thread

Registry and Filtering

The FUNDToolRegistry singleton holds all registered tools with thread-safe access. When building tool descriptors for a provider request, the registry filters tools based on the active agent profile's AllowedTools and DeniedTools lists. This lets different agent profiles have access to different tool sets.

Execution and Caching

The FUNDToolExecutor handles the execution pipeline:

1. Approval check -- Consults FUNDApprovalManager against auto-approval rules and the current approval mode.
2. Thread routing -- Read-only, thread-safe tools can run on background threads. Destructive or game-thread-only tools run on the game thread.
3. LRU caching -- Results are cached with a capacity of 64 entries and a 60-second TTL. Identical tool calls within the TTL return cached results without re-execution.

Tool Categories

Category	Tools	Description
FileSystem	read_file, write_file, edit_file, apply_patch, list_files, search_files, search_in_files	File I/O with path traversal protection
Editor	compile_and_check, compile_blueprints, query_ue_api, capture_viewport, live_coding_compile, hot_reload	Unreal Editor operations
Shell	execute_python, execute_shell, git_query, git_commit	Script execution and version control
Agent	attempt_completion, ask_followup_question, delegate_to_agent, fork_conversation, update_todo_list, memory tools, knowledge tools	Agent control and state management
SemanticSearch	semantic_search, index_project_for_search	BM25 code search
Web	fetch_url	HTTP content retrieval

The execute_python Tool

The most powerful tool in UND. It runs arbitrary Python code with full import unreal access, giving the AI the entire Unreal Editor API. Key behaviors:

• Checks that the PythonScriptPlugin module is loaded before execution
• Wraps code in try/except with stdout/stderr capture
• Runs garbage collection before execution to prevent UE5 world leak detector asserts
• Output is truncated at 50,000 characters
• Certain crash-prone APIs (like open_editor_for_asset) are blocked

Knowledge System

UND automatically injects relevant documentation into the system prompt based on what the user is asking about.

Knowledge Sources

• UE Engine -- 31 built-in markdown modules covering core Unreal systems (Blueprints, GAS, materials, input, UI, etc.). Shipped read-only with the plugin.
• Project -- Auto-generated modules describing the current project's structure, classes, plugins, and assets. Created at editor startup by the Project Scanner.

Module Scoring

When building a prompt, UND scores each knowledge module against the current context:

Signal	Score
Keyword match from user message	+1.0
Active tool trigger match	+5.0
Agent profile trigger match	+3.0
Priority tiebreaker	priority / 100

Top-scoring modules are selected within a configurable token budget (default: 2,000 tokens).

Auto-RAG

In addition to knowledge modules, UND maintains a BM25 inverted index of project source code. On every third turn, it queries this index with the current conversation context and injects the most relevant code snippets into the prompt. This gives the AI awareness of existing project code without requiring the user to manually reference files.

Context Management

UND uses a sliding-window approach to manage conversation length:

• Token estimation -- 1 token is approximated as 4 characters.
• Window truncation -- When the conversation exceeds the provider's context window, the oldest non-system messages are removed.
• Dropped message summarization -- Truncated messages are summarized so the AI retains awareness of earlier context.
• AI-powered condensing -- The /condense command triggers a full conversation summary, replacing all but the last N messages with a compact system message.

MCP Server Support

UND can connect to external Model Context Protocol (MCP) servers, extending its tool set with third-party capabilities.

• Servers are managed as subprocesses communicating via stdio (JSON-RPC 2.0)
• Health checks run every 5 seconds with automatic restart on crash (max 3 attempts, exponential backoff)
• MCP tools appear in the tool registry as mcp__{ServerName}__{ToolName} and are used by the AI like any built-in tool