UltimateAgents/docs/Backend-Developer.md

Backend-Developer.md — Developer Notes

Project: UltimateAgents — Orchestration Service
Stack: C# 13 / .NET 9 / ASP.NET Core / SQLite / Dapper
Last Updated: 2026-04-07

These notes are for developers working on the backend. Read Architecture.md first for the full system design. This document focuses on practical day-to-day guidance: how to set up, where things live, what rules to follow, and common pitfalls.

---

Table of Contents

1. Getting Started
2. Project Layout Cheat Sheet
3. Domain Rules — Must Know
4. Adding a New Feature — Step-by-Step
5. Database & Migrations
6. Working with the Orchestrator Engine
7. Agent Matching
8. Scheduling & Events
9. Container Runtime
10. Error Handling & Escalation
11. Testing Guidelines
12. Logging Conventions
13. Common Pitfalls
14. Environment Variables & Config
15. Useful Commands

---

1. Getting Started

Prerequisites

• .NET 9 SDK (dotnet --version should show 9.x.x)
• dotnet restore

# Restore packages
dotnet restore

# Run the API
dotnet run --project src/UltimateAgents.Api

# Run the CLI
dotnet run --project src/UltimateAgents.Cli -- task list

The service uses a local SQLite database and in-process C# events. No external message broker or Docker is required for local development.

---

2. Project Layout Cheat Sheet

src/
├── UltimateAgents.Domain/        ← Pure C# models, no framework dependencies
├── UltimateAgents.Application/   ← Use cases, commands, queries, interfaces
├── UltimateAgents.Infrastructure/← Dapper, runtime adapter, RabbitMQ
├── UltimateAgents.Api/           ← ASP.NET Core host, controllers, DTOs
└── UltimateAgents.Cli/           ← System.CommandLine frontend

Dependency direction (strict — never reverse):

Api → Application → Domain
Infrastructure → Application → Domain
Cli → Api (via HTTP client)

• Domain has zero NuGet dependencies.
• Application depends only on Domain and its own interfaces (defined in Application/Interfaces/).
• Infrastructure implements those interfaces. It is the only project that touches Dapper, the runtime adapter, or RabbitMQ.
• Controllers in Api call Application services only — never repositories directly.

---

3. Domain Rules — Must Know

These rules are enforced in the domain layer. Do not bypass them in controllers or services.

Complexity is always auto-evaluated

// Always use the evaluator — never set ComplexityLevel manually in external code
var level = ComplexityEvaluator.Evaluate(task.ResolveTimeEstimate);

ResolveTimeEstimate	Result
null	ExtremHard
> 60	Hard
> 30	Middle
≤ 30	Low

Tasks are always flat — no parent/subtask nesting

Use next_task and TaskGroup to compose tasks. Never introduce a “parent task” concept.

• next_task / previous_task express ordering within or across groups.
• ForeachTask expansion creates independent task instances linked by next_task / previous_task, placed into a sequential group.
• GotoTask jumps to an arbitrary target (task or group) and merges an additional prompt into it.
• ConditionTask evaluates a boolean expression and routes to a true or false branch.

Chain cycles are rejected at write time

DependencyGraph.HasCycle() is called inside CreateTaskCommand and CreateGroupCommand before the task or group is persisted. It validates the entire chain — task NextTaskId links, group Next pointers, and parent‚child relationships — as one unified directed graph. If a cycle is detected, throw DependencyCycleException — the API maps this to HTTP 422.

Agent isolation is absolute

Agents run inside their assigned container. The domain model enforces that an agent cannot reference tools or volumes outside its declared container spec. Never pass cross-container paths in task prompts.

---

4. Adding a New Feature — Step-by-Step

Follow this checklist to keep the architecture clean.

1. Domain first. Does the feature need a new entity, value object, or enum? Add it to UltimateAgents.Domain. Write unit tests for any domain logic.
2. Define the interface. If the feature needs infrastructure (DB query, external call), add an interface to Application/Interfaces/.
3. Write the use case. Add a command or query class in Application/. Use a plain service class to implement the handler. No Dapper here.
4. Implement infrastructure. Implement the new interface in Infrastructure/. Use Dapper and schema scripts for persistence.
5. Expose via API. Add or extend a controller in Api/Controllers/. Add a DTO and a FluentValidation validator.
6. Wire up DI. Register the new service/repository in Program.cs or a dedicated ServiceCollectionExtensions class.
7. Add CLI command (if user-facing). Add a subcommand to the appropriate file in Cli/Commands/.
8. Write tests. Unit-test domain + application layers; integration-test the API endpoint with a local SQLite database.

---

5. Database & Migrations

Database schema

• The SQLite schema is defined in Infrastructure/Persistence/DatabaseSchema.sql.
• Use Dapper for lightweight mapping and parameterized SQL queries.
• Many-to-many relationships are represented using explicit join tables.
• Schema updates are managed via versioned SQL scripts, not EF Core migrations.

---

6. Working with the Orchestrator Engine

OrchestratorEngine runs as an IHostedService. It watches for eligible tasks and drives them through their lifecycle.

Task lifecycle states

Created → Queued → Running → Succeeded
                           ↘ Failed → Retrying → Succeeded
                                               ↘ Escalated

Rules for state transitions

• Only OrchestratorEngine is allowed to write task execution state. Controllers must use the StartTaskCommand application service, which queues the intent — never set state directly from a controller.
• State is persisted in SQLite after every transition for status queries. The local database is the source of truth.
• Goto task: when a task completes and GotoTaskId is set, the engine merges GotoPrompt into the target task's Prompt (append by default) before transitioning. Never overwrite the original prompt.

Group execution

Tasks are organised into TaskGroup entities. A group has a GroupType and a Children list of GroupChildRef. Children can be tasks (any type) or nested TaskGroup instances.

• Sequential — the engine activates the first child; when it completes, activates the next sibling child in order. After the last child completes, the group itself is marked complete; the engine then activates group.Next (if set).
• Parallel — all children are dispatched simultaneously. When every child is complete, the group is marked complete and group.Next is activated.
• Nested groups follow the same rules recursively — a nested group must itself be fully complete before its parent considers it done.
• group.Next is a GroupChildRef and can point to a task or another group.
• The engine uses local coordination and database transactions to prevent the same group from being activated twice.

---

7. Agent Matching

Agent selection uses the Strategy pattern. The matching strategy is resolved from the agent's ChooseRule field.

// Registered in DI as keyed services
services.AddKeyedScoped<IAgentMatchStrategy, ExactSingleTagStrategy>(AgentMatchRule.ExactSingle);
services.AddKeyedScoped<IAgentMatchStrategy, SupersetTagStrategy>(AgentMatchRule.Superset);
services.AddKeyedScoped<IAgentMatchStrategy, AllRequiredTagsStrategy>(AgentMatchRule.AllRequired);

Order of precedence when multiple agents match: prefer agents with the fewest extra skills (most specific match). If equal, pick the agent with the lowest current load.

No match found: the dispatcher retries matching after the configured backoff period. After max_retries, the task is escalated.

---

8. Scheduling & Events

Schedule types

Type	Behavior
Immediate	Queued for dispatch as soon as the task is created.
TimeBased	Cron or ISO 8601 repeat expression. Evaluated by SchedulerService.
DatetimeEvent	Fires once at the specified UTC datetime. Use ISO 8601 format.

This version does not use signal-based schedule triggers. Tasks are scheduled using immediate, time-based, or datetime-event rules. Internal coordination is handled through in-process C# events dispatched via IEventDispatcher.

---

9. Container Runtime

The IContainerRuntime interface abstracts the local isolated runtime. Switch the implementation via config:

"RuntimeAdapter": "LocalSandbox"

Local development

Use the local runtime adapter. No Docker or Kubernetes runtime is required.

Runtime spec validation

Before StartAsync is called, the ContainerSpec is validated:

• All declared tools must have a name and version.
• resources.cpu and resources.memory must be positive values.
• network_policy must be explicitly declared (empty is allowed, but null is rejected).

Never pass user-controlled strings directly into container exec calls. Always use the tool definition's pre-declared usage_guidelines to build the command. This prevents command injection.

---

10. Error Handling & Escalation

Controller layer

All controllers are wrapped by the global exception middleware in Api/Middleware/ExceptionMiddleware.cs. Do not add try/catch blocks in controllers for domain exceptions — let the middleware handle them.

// Good
var result = await _createTaskService.ExecuteAsync(command, ct);
return Ok(result);

// Bad — don't do this
try { ... } catch (TaskNotFoundException) { return NotFound(); }

Escalation flow

1. Agent reports failure with Escalation.Reason and Escalation.Prompt.
2. EscalationHandler checks the task's RetryPolicy.
3. If retries remain: re-queue the task with incremented attempt counter and apply backoff.
4. If retries exhausted: publish an EscalationEvent and set task state to Escalated.
5. The human operator receives the escalation via the configured notification channel (webhook or log alert — configured externally).

Always populate Escalation on failure. A missing escalation object on a failed task is treated as a bug and logged at Error level.

---

11. Testing Guidelines

Unit tests (UltimateAgents.Domain.Tests, UltimateAgents.Application.Tests)

• Test domain rules in isolation — no database, no Docker.
• Mock ITaskRepository, IContainerRuntime, and IEventDispatcher with Moq.
• Cover every ComplexityEvaluator.Evaluate() branch.
• Cover cycle detection: test graphs with cycles, linear chains, and parallel groups.

dotnet test tests/UltimateAgents.Domain.Tests
dotnet test tests/UltimateAgents.Application.Tests

Integration tests (UltimateAgents.Api.Tests)

• Use local SQLite or file-backed database instances for integration testing.
• Test the full HTTP request → DB → response cycle.
• Seed data via the configured persistence layer in the test fixture, not via the API.

dotnet test tests/UltimateAgents.Api.Tests

Coverage target

• Domain layer: ≥ 90% line coverage.
• Application layer: ≥ 80% line coverage.
• API controllers: covered by integration tests, not unit tests.

---

12. Logging Conventions

Use Serilog with structured properties. Always include correlation context.

_logger.LogInformation(
    "Task {TaskId} transitioned to {NewState} by agent {AgentId} in container {ContainerId}",
    task.Id, newState, agent.Id, container.Id);

Mandatory fields for task/agent log entries:

Field	Description
TaskId	Guid of the task
AgentId	Guid of the acting agent (if applicable)
ContainerId	Runtime container ID
Event	Short event name (e.g. TaskStarted)
ErrorContext	Exception message / stack trace on error

Log levels:

Level	When to use
Debug	Detailed internal flow (disabled in production)
Information	State transitions, task/agent lifecycle events
Warning	Retries, unexpected but recoverable situations
Error	Escalations, unhandled exceptions, data integrity issues

Never log task Prompt content at Information or above — it may contain sensitive instructions. Use Debug only.

---

13. Common Pitfalls

Pitfall	Why it happens	Fix
DependencyCycleException at runtime	A next_task or next_group chain forms a cycle that wasn’t caught at write-time	Always call DependencyGraph.HasCycle() in CreateTaskCommand and CreateGroupCommand before saving
Agent never matched	Agent’s TaskTags don’t intersect with task’s TaskTags	Check ChooseRule on the agent; verify tags are using the same enum string values
Cache state out of sync with DB	The local state cache or temporary store is stale compared to the database	Write to SQLite first in a transaction, then update any local cache. Never update cache before the DB commit.
Container start fails silently	IContainerRuntime.StartAsync threw but the exception was swallowed	Always await container calls; propagate exceptions to EscalationHandler
Foreach tasks run in wrong order	ForeachTask.ForeachTemplateTaskId set but next_task/previous_task links not generated	ForeachExpander in the application layer must set ordering links on generated task instances
Double-scheduling in multi-instance deploy	Two API instances pick up the same eligible task	Use local database locking and coordination; ensure task state transitions are serialized through SQLite.
User input injected into container commands	Prompt strings passed directly to ContainerRuntime.ExecAsync	Only use pre-declared tool usage_guidelines to construct exec commands; validate/sanitize any user-supplied values

---

14. Environment Variables & Config

All settings follow the ASP.NET Core configuration hierarchy: environment variables override appsettings.json.

Environment Variable	Default	Purpose
Database__ConnectionString	(required)	SQLite connection string or file path
ContainerRuntime	LocalSandbox	Local runtime adapter name
Jwt__Authority	(required in production)	OIDC authority URL
Jwt__Audience	orchestration-api	JWT audience claim
Orchestrator__RetryPolicy__MaxRetries	3	Global default max retries
Orchestrator__RetryPolicy__Backoff	exponential	fixed or exponential

For local development, copy appsettings.Development.json.example to appsettings.Development.json and fill in values. This file is .gitignored.

---

15. Useful Commands

# Run all tests
dotnet test

# Run only unit tests
dotnet test --filter "Category=Unit"

# Add a new schema migration
# (Use SQL scripts or schema versioning tools as appropriate.)

# Roll back last schema change (dev only)
# (Use SQL script rollback or schema versioning practices.)

# Watch-mode for API development
dotnet watch run --project src/UltimateAgents.Api

# Build the CLI as a self-contained binary
dotnet publish src/UltimateAgents.Cli -c Release -r linux-x64 --self-contained

# Check for vulnerable NuGet packages
dotnet list package --vulnerable --include-transitive

# Format code
dotnet format

# View structured logs (if using Seq locally)
open http://localhost:5341

---

Questions? Check Architecture.md for design decisions, or features.md for the full feature specification.