Unity Autonomous Agent MCP

🤖 A comprehensive autonomous agent framework for Unity 2022.3.22f1 that combines the power of Model Context Protocol (MCP) with advanced AI decision-making capabilities.

🌟 Key Features

🧠 Autonomous Agent Capabilities

Intelligent Task Planning: AI-driven task decomposition and execution planning
Decision Making: Context-aware decision trees with learning capabilities
Self-Improvement: Continuous learning from user interactions and outcomes
Multi-Agent Coordination: Support for multiple autonomous agents working together

🎮 Unity Integration

Full Editor Control: Complete Unity Editor automation and manipulation
Runtime AI: In-game autonomous NPC behavior and debugging
Asset Management: Intelligent asset creation, optimization, and organization
Scene Management: Automated scene setup, optimization, and testing

🔧 Advanced MCP Features

Extensible Architecture: Plugin-based system for custom tools and handlers
Real-time Communication: Low-latency TCP/IP communication with external AI services
Multi-Provider Support: Compatible with Claude, GPT, Gemini, and custom LLM providers
Resource Management: Intelligent resource allocation and optimization

🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    Unity Autonomous Agent MCP              │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌──────────────┐ │
│  │   Agent Core    │  │   Task Planner  │  │ Decision     │ │
│  │                 │  │                 │  │ Engine       │ │
│  │ - State Mgmt    │  │ - Task Decomp   │  │ - ML Models  │ │
│  │ - Learning      │  │ - Priority      │  │ - Context    │ │
│  │ - Memory        │  │ - Scheduling    │  │ - Reasoning  │ │
│  └─────────────────┘  └─────────────────┘  └──────────────┘ │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌──────────────┐ │
│  │  MCP Server     │  │  Unity Bridge   │  │ Plugin       │ │
│  │                 │  │                 │  │ System       │ │
│  │ - Protocol      │  │ - Editor API    │  │ - Custom     │ │
│  │ - Handlers      │  │ - Runtime API   │  │ Tools        │ │
│  │ - Transport     │  │ - Asset Mgmt    │  │ - Extensions │ │
│  └─────────────────┘  └─────────────────┘  └──────────────┘ │
├─────────────────────────────────────────────────────────────┤
│                    Unity Engine (2022.3.22f1)              │
└─────────────────────────────────────────────────────────────┘

📋 Requirements

Unity 2022.3.22f1 or later (tested up to Unity 6.1)
.NET/C# 9.0
Node.js 18.0.0+ with npm (for TypeScript server)
Python 3.8+ (for ML/AI components)
External LLM Provider (Claude, OpenAI, Gemini, or custom)

🚀 Quick Start

Re-analysis details and capability gaps are documented in docs/capability-matrix.md.

1. Installation

# Clone the repository
git clone https://github.com/KinofSin/UnityAutonomousMCP.git
cd UnityAutonomousMCP

# Install Unity Package
# In Unity Editor: Window > Package Manager > Add package from git URL
# Enter: file:///path/to/UnityAutonomousMCP/com.autonomous-unity.mcp

2. Setup TypeScript Server

npm install
npm run build
npm run smoke

npm run smoke validates planner + executor behavior (including run_tests -> get_test_job polling) for both successful and failed test-job terminal paths, without requiring a live Unity Editor.

3. Run modes

# Autonomous bootstrap (local dry-run using mock Unity bridge)
npm run dev -- -- "inspect scene and update scripts"

# MCP stdio mode (for Claude/Cursor/Windsurf MCP client config)
npm run dev -- --mcp

# Force mock bridge explicitly
npm run dev -- --mock -- "inspect scene and update scripts"

4. Transport bridge configuration (real Unity connection)

Unity package host (inside Unity Editor) serves both transports:

HTTP endpoint: POST /mcp/tool on UNITY_HTTP_PORT (default 8080)
TCP endpoint: newline-delimited JSON on UNITY_TCP_PORT (default 8081)

Node-side bridge environment variables:

UNITY_TRANSPORT=http       # http | tcp | mock
UNITY_HOST=127.0.0.1
UNITY_HTTP_PORT=8080
UNITY_TCP_PORT=8081
UNITY_TIMEOUT_MS=10000
UNITY_TEST_POLL_ATTEMPTS=30
UNITY_TEST_POLL_INTERVAL_MS=1000
UNITY_TEST_POLL_TIMEOUT_MS=120000

5. Configure Claude Desktop

{
  "mcpServers": {
    "unity-autonomous-agent": {
      "command": "node",
      "args": ["path/to/server/dist/index.js"],
      "env": {
        "UNITY_HOST": "localhost",
        "UNITY_TRANSPORT": "http",
        "UNITY_HTTP_PORT": "8080",
        "UNITY_TCP_PORT": "8081",
        "AI_PROVIDER": "anthropic",
        "AI_API_KEY": "your-api-key"
      }
    }
  }
}

6. Start Using

Open Unity Project
Navigate to Edit > Preferences > Autonomous Agent MCP
Configure host + HTTP/TCP ports
Click "Connect" to start Unity transport host
Start MCP server (npm run dev -- --mcp)
Begin interacting with AI through your preferred MCP client

✅ Current implementation status (this repo)

Capability matrix + gap analysis: docs/capability-matrix.md
Autonomous planner core: server/src/planner.ts
Policy guardrails: server/src/policy.ts
Plan executor: server/src/executor.ts
MCP server tools (autonomous_plan, unity_tool_call, list_capabilities): server/src/mcpServer.ts
Real Unity bridge transports (HTTP/TCP + env-driven mode): server/src/unityBridge.ts
Concrete step contracts for autonomous_plan: server/src/contracts.ts
Unity 2022.3.22f1 package scaffold: com.autonomous-unity.mcp/
Editor settings/provider: Editor/AutonomousMcpSettingsProvider.cs
Real transport host: Editor/AutonomousMcpTransportHost.cs
Tool dispatcher: Editor/AutonomousMcpToolDispatcher.cs
Runtime entry component: Runtime/AutonomousMcpRuntime.cs

🔁 End-to-end autonomous_plan contract mapping

autonomous_plan now emits concrete Unity tool payloads:

read_console → { level, limit }
manage_scene → { action: "inspect_active_scene" }
manage_script → { action: "create_or_update", scriptPath, contents } (when goal implies code changes)
validate_script → { strict } (after script edits)
run_tests → { mode } (when goal mentions tests; returns jobId)
get_test_job → { jobId } (executor polls until completed or failed)
batch_execute → { operations: [{ tool: "manage_scene", params: { action: "save_active_scene" } }] }

Test execution flow

run_tests starts an async Unity Test Runner job (editmode or playmode)
Server receives { jobId, status: "queued" }
Executor automatically calls get_test_job until terminal state
Final test summary (passed/failed/skipped + per-test details) is included in execution results

Test polling policy limits

UNITY_TEST_POLL_ATTEMPTS: max polling iterations for get_test_job (clamped 1..300)
UNITY_TEST_POLL_INTERVAL_MS: delay between polls in milliseconds (clamped 100..60000)
UNITY_TEST_POLL_TIMEOUT_MS: overall timeout for polling cycle (clamped 1000..1800000)

🤖 Autonomous Agent Features

Intelligent Task Planning

Goal Decomposition: Break complex tasks into manageable subtasks
Priority Management: Dynamic task prioritization based on context
Resource Allocation: Optimal distribution of system resources
Dependency Resolution: Handle task dependencies and conflicts

Learning & Adaptation

Pattern Recognition: Learn from user behavior and preferences
Performance Optimization: Improve efficiency over time
Error Recovery: Learn from mistakes and avoid repetition
Context Awareness: Adapt to project-specific requirements

Multi-Agent Coordination

Agent Communication: Coordinate multiple specialized agents
Task Distribution: Distribute workload across agent instances
Conflict Resolution: Handle competing priorities and resource conflicts
Collaborative Problem Solving: Combine multiple agent perspectives

🔧 MCP Tools & Capabilities

🎮 Unity Editor Tools

Scene Management: Create, modify, optimize scenes automatically
Asset Operations: Intelligent asset creation and organization
GameObject Control: Advanced object manipulation and optimization
Component Management: Dynamic component addition and configuration
Build Automation: Automated build processes and optimization

🧩 Advanced Development Tools

Code Generation: AI-assisted script writing and optimization
Testing Automation: Automated test creation and execution
Performance Analysis: Real-time performance monitoring and optimization
Debugging Assistant: Intelligent error detection and resolution
Documentation Generation: Auto-generate technical documentation

🎯 Runtime AI Features

NPC Behavior: Dynamic, intelligent NPC behavior systems
Game Balance: Automated game balance testing and adjustment
Player Analytics: Real-time player behavior analysis
Dynamic Difficulty: Adaptive difficulty adjustment systems
Content Generation: Procedural content creation and optimization

🔌 Plugin System

Create custom plugins to extend functionality:

[AutonomousPlugin("custom-tool")]
public class CustomToolPlugin : IAutonomousTool
{
    public async Task<ToolResult> ExecuteAsync(ToolContext context)
    {
        // Your custom tool logic
        return new ToolResult { Success = true, Data = result };
    }
}

📊 Performance & Monitoring

Real-time Metrics

Agent Performance: Monitor agent efficiency and accuracy
Resource Usage: Track CPU, memory, and network utilization
Task Completion: Measure task success rates and timing
Learning Progress: Track agent improvement over time

Optimization Features

Caching System: Intelligent caching for frequently used data
Batch Processing: Optimize multiple operations together
Predictive Loading: Anticipate and preload required resources
Adaptive Performance: Adjust performance based on system capabilities

🛡️ Security & Safety

Safety Measures

Sandboxed Execution: Isolate agent operations from critical systems
Permission System: Granular control over agent capabilities
Audit Logging: Complete audit trail of all agent actions
Rollback Capability: Undo system for agent modifications

Best Practices

Regular Backups: Automatic backup before major operations
Validation Checks: Verify operation safety before execution
User Confirmation: Require confirmation for destructive operations
Error Handling: Comprehensive error recovery mechanisms

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

Clone repository
Install dependencies (npm install in server directory)
Open Unity project
Enable Developer Mode in preferences
Start contributing!

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Built upon the excellent foundations of Unity-MCP and UnityMCP
Inspired by the latest advances in autonomous agents and AI decision-making
Community feedback and contributions have been invaluable

---

🚀 Ready to transform your Unity development with AI-powered autonomy?

Start building intelligent, self-improving Unity projects today!

UnityAutonomousMCP