# Backend-Developer.md — Developer Notes

**Project:** UltimateAgents — Orchestration Service  
**Stack:** C# 13 / .NET 9 / ASP.NET Core / SQLite / Dapper  
**Last Updated:** 2026-04-07

> These notes are for developers working on the backend. Read [Architecture.md](Architecture.md) first for the full system design. This document focuses on practical day-to-day guidance: how to set up, where things live, what rules to follow, and common pitfalls.

---

## Table of Contents

1. [Getting Started](#1-getting-started)
2. [Project Layout Cheat Sheet](#2-project-layout-cheat-sheet)
3. [Domain Rules — Must Know](#3-domain-rules--must-know)
4. [Adding a New Feature — Step-by-Step](#4-adding-a-new-feature--step-by-step)
5. [Database & Migrations](#5-database--migrations)
6. [Working with the Orchestrator Engine](#6-working-with-the-orchestrator-engine)
7. [Agent Matching](#7-agent-matching)
8. [Scheduling & Events](#8-scheduling--events)
9. [Container Runtime](#9-container-runtime)
10. [Error Handling & Escalation](#10-error-handling--escalation)
11. [Testing Guidelines](#11-testing-guidelines)
12. [Logging Conventions](#12-logging-conventions)
13. [Common Pitfalls](#13-common-pitfalls)
14. [Environment Variables & Config](#14-environment-variables--config)
15. [Useful Commands](#15-useful-commands)

---

## 1. Getting Started

### Prerequisites

- .NET 9 SDK (`dotnet --version` should show `9.x.x`)
- `dotnet restore`

```bash
# Restore packages
dotnet restore

# Run the API
dotnet run --project src/UltimateAgents.Api

# Run the CLI
dotnet run --project src/UltimateAgents.Cli -- task list
```

The service uses a local SQLite database and in-process C# events. No external message broker or Docker is required for local development.

---

## 2. Project Layout Cheat Sheet

```
src/
├── UltimateAgents.Domain/        ← Pure C# models, no framework dependencies
├── UltimateAgents.Application/   ← Use cases, commands, queries, interfaces
├── UltimateAgents.Infrastructure/← Dapper, runtime adapter, RabbitMQ
├── UltimateAgents.Api/           ← ASP.NET Core host, controllers, DTOs
└── UltimateAgents.Cli/           ← System.CommandLine frontend
```

**Dependency direction (strict — never reverse):**

```
Api → Application → Domain
Infrastructure → Application → Domain
Cli → Api (via HTTP client)
```

- `Domain` has **zero** NuGet dependencies.
- `Application` depends only on `Domain` and its own interfaces (defined in `Application/Interfaces/`).
- `Infrastructure` implements those interfaces. It is the only project that touches Dapper, the runtime adapter, or RabbitMQ.
- Controllers in `Api` call Application services only — never repositories directly.

---

## 3. Domain Rules — Must Know

These rules are enforced in the domain layer. Do not bypass them in controllers or services.

### Complexity is always auto-evaluated

```csharp
// Always use the evaluator — never set ComplexityLevel manually in external code
var level = ComplexityEvaluator.Evaluate(task.ResolveTimeEstimate);
```

| `ResolveTimeEstimate` | Result        |
|-----------------------|---------------|
| `null`                | `ExtremHard`  |
| `> 60`                | `Hard`        |
| `> 30`                | `Middle`      |
| `≤ 30`                | `Low`         |

### Tasks are always flat — no parent/subtask nesting

Use `next_task` and `TaskGroup` to compose tasks. Never introduce a “parent task” concept.
- `next_task` / `previous_task` express ordering within or across groups.
- `ForeachTask` expansion creates independent task instances linked by `next_task` / `previous_task`, placed into a sequential group.
- `GotoTask` jumps to an arbitrary target (task or group) and merges an additional prompt into it.
- `ConditionTask` evaluates a boolean expression and routes to a true or false branch.

### Chain cycles are rejected at write time

`DependencyGraph.HasCycle()` is called inside `CreateTaskCommand` and `CreateGroupCommand` **before** the task or group is persisted. It validates the entire chain — task `NextTaskId` links, group `Next` pointers, and parent‚child relationships — as one unified directed graph. If a cycle is detected, throw `DependencyCycleException` — the API maps this to HTTP 422.

### Agent isolation is absolute

Agents run inside their assigned container. The domain model enforces that an agent cannot reference tools or volumes outside its declared container spec. Never pass cross-container paths in task prompts.

---

## 4. Adding a New Feature — Step-by-Step

Follow this checklist to keep the architecture clean.

1. **Domain first.** Does the feature need a new entity, value object, or enum? Add it to `UltimateAgents.Domain`. Write unit tests for any domain logic.
2. **Define the interface.** If the feature needs infrastructure (DB query, external call), add an interface to `Application/Interfaces/`.
3. **Write the use case.** Add a command or query class in `Application/`. Use a plain service class to implement the handler. No Dapper here.
4. **Implement infrastructure.** Implement the new interface in `Infrastructure/`. Use Dapper and schema scripts for persistence.
5. **Expose via API.** Add or extend a controller in `Api/Controllers/`. Add a DTO and a FluentValidation validator.
6. **Wire up DI.** Register the new service/repository in `Program.cs` or a dedicated `ServiceCollectionExtensions` class.
7. **Add CLI command** (if user-facing). Add a subcommand to the appropriate file in `Cli/Commands/`.
8. **Write tests.** Unit-test domain + application layers; integration-test the API endpoint with a local SQLite database.

---

## 5. Database & Migrations

### Database schema

- The SQLite schema is defined in `Infrastructure/Persistence/DatabaseSchema.sql`.
- Use Dapper for lightweight mapping and parameterized SQL queries.
- Many-to-many relationships are represented using explicit join tables.
- Schema updates are managed via versioned SQL scripts, not EF Core migrations.

---

## 6. Working with the Orchestrator Engine

`OrchestratorEngine` runs as an `IHostedService`. It watches for eligible tasks and drives them through their lifecycle.

### Task lifecycle states

```
Created → Queued → Running → Succeeded
                           ↘ Failed → Retrying → Succeeded
                                               ↘ Escalated
```

### Rules for state transitions

- Only `OrchestratorEngine` is allowed to write task execution state. Controllers must use the `StartTaskCommand` application service, which queues the intent — never set state directly from a controller.
- State is persisted in SQLite after every transition for status queries. The local database is the source of truth.
- Goto task: when a task completes and `GotoTaskId` is set, the engine merges `GotoPrompt` into the target task's `Prompt` (append by default) before transitioning. Never overwrite the original prompt.

### Group execution

Tasks are organised into `TaskGroup` entities. A group has a `GroupType` and a `Children` list of `GroupChildRef`. Children can be tasks (any type) or nested `TaskGroup` instances.

- `Sequential` — the engine activates the first child; when it completes, activates the next sibling child in order. After the last child completes, the group itself is marked complete; the engine then activates `group.Next` (if set).
- `Parallel` — all children are dispatched simultaneously. When every child is complete, the group is marked complete and `group.Next` is activated.
- Nested groups follow the same rules recursively — a nested group must itself be fully complete before its parent considers it done.
- `group.Next` is a `GroupChildRef` and can point to a task or another group.
- The engine uses local coordination and database transactions to prevent the same group from being activated twice.

---

## 7. Agent Matching

Agent selection uses the Strategy pattern. The matching strategy is resolved from the agent's `ChooseRule` field.

```csharp
// Registered in DI as keyed services
services.AddKeyedScoped<IAgentMatchStrategy, ExactSingleTagStrategy>(AgentMatchRule.ExactSingle);
services.AddKeyedScoped<IAgentMatchStrategy, SupersetTagStrategy>(AgentMatchRule.Superset);
services.AddKeyedScoped<IAgentMatchStrategy, AllRequiredTagsStrategy>(AgentMatchRule.AllRequired);
```

**Order of precedence when multiple agents match:** prefer agents with the fewest extra skills (most specific match). If equal, pick the agent with the lowest current load.

**No match found:** the dispatcher retries matching after the configured `backoff` period. After `max_retries`, the task is escalated.

---

## 8. Scheduling & Events

### Schedule types

| Type             | Behavior                                                              |
|------------------|-----------------------------------------------------------------------|
| `Immediate`      | Queued for dispatch as soon as the task is created.                   |
| `TimeBased`      | Cron or ISO 8601 repeat expression. Evaluated by `SchedulerService`.  |
| `DatetimeEvent`  | Fires once at the specified UTC datetime. Use ISO 8601 format.        |

This version does not use signal-based schedule triggers. Tasks are scheduled using immediate, time-based, or datetime-event rules. Internal coordination is handled through in-process C# events dispatched via `IEventDispatcher`.

---

## 9. Container Runtime

The `IContainerRuntime` interface abstracts the local isolated runtime. Switch the implementation via config:

```json
"RuntimeAdapter": "LocalSandbox"
```

### Local development

Use the local runtime adapter. No Docker or Kubernetes runtime is required.

### Runtime spec validation

Before `StartAsync` is called, the `ContainerSpec` is validated:
- All declared tools must have a `name` and `version`.
- `resources.cpu` and `resources.memory` must be positive values.
- `network_policy` must be explicitly declared (empty is allowed, but `null` is rejected).

> **Never pass user-controlled strings directly into container exec calls.** Always use the tool definition's pre-declared `usage_guidelines` to build the command. This prevents command injection.

---

## 10. Error Handling & Escalation

### Controller layer

All controllers are wrapped by the global exception middleware in `Api/Middleware/ExceptionMiddleware.cs`. Do **not** add try/catch blocks in controllers for domain exceptions — let the middleware handle them.

```csharp
// Good
var result = await _createTaskService.ExecuteAsync(command, ct);
return Ok(result);

// Bad — don't do this
try { ... } catch (TaskNotFoundException) { return NotFound(); }
```

### Escalation flow

1. Agent reports failure with `Escalation.Reason` and `Escalation.Prompt`.
2. `EscalationHandler` checks the task's `RetryPolicy`.
3. If retries remain: re-queue the task with incremented attempt counter and apply backoff.
4. If retries exhausted: publish an `EscalationEvent` and set task state to `Escalated`.
5. The human operator receives the escalation via the configured notification channel (webhook or log alert — configured externally).

**Always populate `Escalation` on failure.** A missing escalation object on a failed task is treated as a bug and logged at `Error` level.

---

## 11. Testing Guidelines

### Unit tests (`UltimateAgents.Domain.Tests`, `UltimateAgents.Application.Tests`)

- Test domain rules in isolation — no database, no Docker.
- Mock `ITaskRepository`, `IContainerRuntime`, and `IEventDispatcher` with Moq.
- Cover every `ComplexityEvaluator.Evaluate()` branch.
- Cover cycle detection: test graphs with cycles, linear chains, and parallel groups.

```bash
dotnet test tests/UltimateAgents.Domain.Tests
dotnet test tests/UltimateAgents.Application.Tests
```

### Integration tests (`UltimateAgents.Api.Tests`)

- Use local SQLite or file-backed database instances for integration testing.
- Test the full HTTP request → DB → response cycle.
- Seed data via the configured persistence layer in the test fixture, not via the API.

```bash
dotnet test tests/UltimateAgents.Api.Tests
```

### Coverage target

- Domain layer: **≥ 90%** line coverage.
- Application layer: **≥ 80%** line coverage.
- API controllers: covered by integration tests, not unit tests.

---

## 12. Logging Conventions

Use **Serilog** with structured properties. Always include correlation context.

```csharp
_logger.LogInformation(
    "Task {TaskId} transitioned to {NewState} by agent {AgentId} in container {ContainerId}",
    task.Id, newState, agent.Id, container.Id);
```

**Mandatory fields for task/agent log entries:**

| Field         | Description                              |
|---------------|------------------------------------------|
| `TaskId`      | Guid of the task                         |
| `AgentId`     | Guid of the acting agent (if applicable) |
| `ContainerId` | Runtime container ID                     |
| `Event`       | Short event name (e.g. `TaskStarted`)    |
| `ErrorContext`| Exception message / stack trace on error |

**Log levels:**

| Level       | When to use                                              |
|-------------|----------------------------------------------------------|
| `Debug`     | Detailed internal flow (disabled in production)          |
| `Information` | State transitions, task/agent lifecycle events         |
| `Warning`   | Retries, unexpected but recoverable situations           |
| `Error`     | Escalations, unhandled exceptions, data integrity issues |

Never log task `Prompt` content at `Information` or above — it may contain sensitive instructions. Use `Debug` only.

---

## 13. Common Pitfalls

| Pitfall | Why it happens | Fix |
|---|---|---|
| `DependencyCycleException` at runtime | A `next_task` or `next_group` chain forms a cycle that wasn’t caught at write-time | Always call `DependencyGraph.HasCycle()` in `CreateTaskCommand` and `CreateGroupCommand` before saving |
| Agent never matched | Agent’s `TaskTags` don’t intersect with task’s `TaskTags` | Check `ChooseRule` on the agent; verify tags are using the same enum string values |
| Cache state out of sync with DB | The local state cache or temporary store is stale compared to the database | Write to SQLite first in a transaction, then update any local cache. Never update cache before the DB commit. |
| Container start fails silently | `IContainerRuntime.StartAsync` threw but the exception was swallowed | Always `await` container calls; propagate exceptions to `EscalationHandler` |
| Foreach tasks run in wrong order | `ForeachTask.ForeachTemplateTaskId` set but `next_task`/`previous_task` links not generated | `ForeachExpander` in the application layer must set ordering links on generated task instances |
| Double-scheduling in multi-instance deploy | Two API instances pick up the same eligible task | Use local database locking and coordination; ensure task state transitions are serialized through SQLite. |
| User input injected into container commands | Prompt strings passed directly to `ContainerRuntime.ExecAsync` | Only use pre-declared tool `usage_guidelines` to construct exec commands; validate/sanitize any user-supplied values |

---

## 14. Environment Variables & Config

All settings follow the ASP.NET Core configuration hierarchy: environment variables override `appsettings.json`.

| Environment Variable                     | Default                    | Purpose                              |
|------------------------------------------|----------------------------|--------------------------------------|
| `Database__ConnectionString`             | (required)                 | SQLite connection string or file path |
| `ContainerRuntime`                       | `LocalSandbox`             | Local runtime adapter name           |
| `Jwt__Authority`                         | (required in production)   | OIDC authority URL                   |
| `Jwt__Audience`                          | `orchestration-api`        | JWT audience claim                   |
| `Orchestrator__RetryPolicy__MaxRetries`  | `3`                        | Global default max retries           |
| `Orchestrator__RetryPolicy__Backoff`     | `exponential`              | `fixed` or `exponential`             |

For local development, copy `appsettings.Development.json.example` to `appsettings.Development.json` and fill in values. This file is `.gitignore`d.

---

## 15. Useful Commands

```bash
# Run all tests
dotnet test

# Run only unit tests
dotnet test --filter "Category=Unit"

# Add a new schema migration
# (Use SQL scripts or schema versioning tools as appropriate.)

# Roll back last schema change (dev only)
# (Use SQL script rollback or schema versioning practices.)

# Watch-mode for API development
dotnet watch run --project src/UltimateAgents.Api

# Build the CLI as a self-contained binary
dotnet publish src/UltimateAgents.Cli -c Release -r linux-x64 --self-contained

# Check for vulnerable NuGet packages
dotnet list package --vulnerable --include-transitive

# Format code
dotnet format

# View structured logs (if using Seq locally)
open http://localhost:5341
```

---

> **Questions?** Check [Architecture.md](Architecture.md) for design decisions, or [features.md](features.md) for the full feature specification.