Shadow Workspace & Git Worktrees

1. Concept Overview & Systemic Problem

When an autonomous agent begins to modify the codebase in the developer's active directory, a critical Workflow Lockout occurs. If the agent edits 25 files, compiles code, installs packages, and runs tests, the developer cannot simultaneously write code, check their local dev server, or switch branches: uncommitted changes from both the developer and the agent inevitably mix, causing conflicts.

Moreover, if the agent fails, hangs, or hallucinates, resetting the worktree (git checkout -f or git clean) may inadvertently erase the engineer's uncommitted work.

Shadow Workspace based on Git Worktrees solves this problem once and for all. The agent receives a fully isolated directory on disk, bound to its own branch. The engineer continues to write code in their favorite editor while the background agent concurrently performs complex refactoring in the shadow environment without any impact on the active workstation.

2. Architectural Taxonomy & Mental Model

The architecture for interacting with shadow workspaces is based on disk partitioning and a shared Git object database:

┌─────────────────────────────────────────────────────────────┐
│                 SHARED REPOSITORY OBJECTS (.git)            │
└──────────────┬───────────────────────────────┬──────────────┘
               │                               │
               ▼                               ▼
┌─────────────────────────────┐ ┌─────────────────────────────┐
│  PRIMARY WORKSPACE (Human)  │ │ SHADOW WORKSPACE (AI Agent) │
│ • Branch: feature/payments  │ │ • Branch: agent/fix-auth    │
│ • Path: ~/projects/app      │ │ • Path: ~/.worktrees/task-42│
│ • Port: 3000 (Active)       │ │ • Port: 3001 (Isolated)     │
│ • Uncommitted local edits   │ │ • Automated ReAct Loop      │
└─────────────────────────────┘ └─────────────────────────────┘

1. Shared Object Database:
   • All workspaces utilize a single local Git object database (.git/objects). There is no need to re-download gigabytes of history or perform a slow git clone.
2. Dynamic Agent Worktree:
   • Created in milliseconds with the command git worktree add -b <agent-branch> <shadow-path> <base-commit>.
   • Provides the agent with a completely clean worktree free of human clutter and unfinished edits.
3. Environment Decoupling:
   • Separate environment configuration files (.env.test), isolated network ports, and unique temporary build directories.
4. Reconciliation & Cleanup Mechanism:
   • After verification and successful test completion, changes are committed as an atomic commit, after which the shadow directory is safely removed (git worktree remove).

3. Technical Pipeline & Internal Mechanics

The lifecycle of a task in the shadow workspace:

1. Initiation and Allocation of Shadow Space: The orchestrator (or developer via a command like agy-run-in-worktree) generates a unique task ID and creates a directory:

   git worktree add -b agent/auth-refactor ../shadow-auth main
   

2. Fast Dependency Linking: To avoid waiting minutes for npm install, the orchestrator creates symbolic links to shared node_modules or uses pnpm's content cache:

   ln -s $(pwd)/node_modules ../shadow-auth/node_modules
   

3. Autonomous Agent Operation: The agent runs with the working directory Cwd = ../shadow-auth. It autonomously:
   • Reads and edits files.
   • Runs a local server on an isolated port (PORT=3005 npm run test:e2e).
   • Iterates in a ReAct loop until complete success.
4. Result Validation: All tests run in the shadow workspace. If the agent breaks the build, the engineer experiences no issues in their main IDE.
5. Commit and Integration: The agent makes a final commit to the agent/auth-refactor branch. The orchestrator removes the shadow directory:

   git worktree remove ../shadow-auth
   

6. Presentation of Results to the Engineer: The engineer receives a notification that the task is ready and can perform a quick git merge or open a PR.

4. Production Engineering Scenarios

01. Parallel Feature Development Without Interrupting the Developer

The engineer is designing a new settings screen in the main branch:

• Simultaneously, they launch a background agent to write database migrations and controllers in the shadow workspace.
• The agent runs tests, fixes types, and prepares the backend in isolation.
• The engineer does not interrupt their screen design for a second.

02. Tournament Testing of Competing Hypotheses (Multi-Agent Bake-off)

Searching for the fastest algorithm for processing large data arrays:

• The orchestrator deploys three parallel worktrees: agent-a, agent-b, agent-c.
• Three different models (Claude 3.7 Sonnet, DeepSeek R1, GPT-4o) are tasked with optimization.
• A benchmark runs; the solution with the best latency metric is selected, while the other two workspaces are automatically deleted.

03. Safe Execution of Unverified Agent Scripts

Testing a radical system kernel update:

• If the agent completely breaks the file structure or generates non-functional code, the developer simply deletes the workspace directory in 1 second without fear of damaging the local Git repository.

5. Pitfalls, Common Mistakes & Security

• Port Collisions: If the shadow workspace runs tests that implicitly attempt to start a web server on a fixed port 3000, they will fail with an EADDRINUSE error if that port is occupied by the developer. Always configure port parameterization.
• Worktree Branch Locks: Git prohibits mounting the same branch in two different worktrees simultaneously. Shadow agents must always operate in new, unique branches.
• Disk Space Accumulation: If the orchestrator crashes without a cleanup step (git worktree remove), dozens of forgotten repository folders may remain on disk. Periodically execute git worktree prune.
• Leakage of Uncontrolled Files (.env.local): Files in .gitignore are not automatically copied to the new worktree. The agent must be explicitly granted access to test environment variables for proper test execution.