382 lines
17 KiB
Markdown
382 lines
17 KiB
Markdown
# Backend-Developer.md — Developer Notes
|
||
|
||
**Project:** UltimateAgents — Orchestration Service
|
||
**Stack:** C# 13 / .NET 9 / ASP.NET Core / SQLite / Dapper
|
||
**Last Updated:** 2026-04-07
|
||
|
||
> These notes are for developers working on the backend. Read [Architecture.md](Architecture.md) first for the full system design. This document focuses on practical day-to-day guidance: how to set up, where things live, what rules to follow, and common pitfalls.
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
1. [Getting Started](#1-getting-started)
|
||
2. [Project Layout Cheat Sheet](#2-project-layout-cheat-sheet)
|
||
3. [Domain Rules — Must Know](#3-domain-rules--must-know)
|
||
4. [Adding a New Feature — Step-by-Step](#4-adding-a-new-feature--step-by-step)
|
||
5. [Database & Migrations](#5-database--migrations)
|
||
6. [Working with the Orchestrator Engine](#6-working-with-the-orchestrator-engine)
|
||
7. [Agent Matching](#7-agent-matching)
|
||
8. [Scheduling & Events](#8-scheduling--events)
|
||
9. [Container Runtime](#9-container-runtime)
|
||
10. [Error Handling & Escalation](#10-error-handling--escalation)
|
||
11. [Testing Guidelines](#11-testing-guidelines)
|
||
12. [Logging Conventions](#12-logging-conventions)
|
||
13. [Common Pitfalls](#13-common-pitfalls)
|
||
14. [Environment Variables & Config](#14-environment-variables--config)
|
||
15. [Useful Commands](#15-useful-commands)
|
||
|
||
---
|
||
|
||
## 1. Getting Started
|
||
|
||
### Prerequisites
|
||
|
||
- .NET 9 SDK (`dotnet --version` should show `9.x.x`)
|
||
- `dotnet restore`
|
||
|
||
```bash
|
||
# Restore packages
|
||
dotnet restore
|
||
|
||
# Run the API
|
||
dotnet run --project src/UltimateAgents.Api
|
||
|
||
# Run the CLI
|
||
dotnet run --project src/UltimateAgents.Cli -- task list
|
||
```
|
||
|
||
The service uses a local SQLite database and in-process C# events. No external message broker or Docker is required for local development.
|
||
|
||
---
|
||
|
||
## 2. Project Layout Cheat Sheet
|
||
|
||
```
|
||
src/
|
||
├── UltimateAgents.Domain/ ← Pure C# models, no framework dependencies
|
||
├── UltimateAgents.Application/ ← Use cases, commands, queries, interfaces
|
||
├── UltimateAgents.Infrastructure/← Dapper, runtime adapter, RabbitMQ
|
||
├── UltimateAgents.Api/ ← ASP.NET Core host, controllers, DTOs
|
||
└── UltimateAgents.Cli/ ← System.CommandLine frontend
|
||
```
|
||
|
||
**Dependency direction (strict — never reverse):**
|
||
|
||
```
|
||
Api → Application → Domain
|
||
Infrastructure → Application → Domain
|
||
Cli → Api (via HTTP client)
|
||
```
|
||
|
||
- `Domain` has **zero** NuGet dependencies.
|
||
- `Application` depends only on `Domain` and its own interfaces (defined in `Application/Interfaces/`).
|
||
- `Infrastructure` implements those interfaces. It is the only project that touches Dapper, the runtime adapter, or RabbitMQ.
|
||
- Controllers in `Api` call Application services only — never repositories directly.
|
||
|
||
---
|
||
|
||
## 3. Domain Rules — Must Know
|
||
|
||
These rules are enforced in the domain layer. Do not bypass them in controllers or services.
|
||
|
||
### Complexity is always auto-evaluated
|
||
|
||
```csharp
|
||
// Always use the evaluator — never set ComplexityLevel manually in external code
|
||
var level = ComplexityEvaluator.Evaluate(task.ResolveTimeEstimate);
|
||
```
|
||
|
||
| `ResolveTimeEstimate` | Result |
|
||
|-----------------------|---------------|
|
||
| `null` | `ExtremHard` |
|
||
| `> 60` | `Hard` |
|
||
| `> 30` | `Middle` |
|
||
| `≤ 30` | `Low` |
|
||
|
||
### Tasks are always flat — no parent/subtask nesting
|
||
|
||
Use `next_task` and `TaskGroup` to compose tasks. Never introduce a “parent task” concept.
|
||
- `next_task` / `previous_task` express ordering within or across groups.
|
||
- `ForeachTask` expansion creates independent task instances linked by `next_task` / `previous_task`, placed into a sequential group.
|
||
- `GotoTask` jumps to an arbitrary target (task or group) and merges an additional prompt into it.
|
||
- `ConditionTask` evaluates a boolean expression and routes to a true or false branch.
|
||
|
||
### Chain cycles are rejected at write time
|
||
|
||
`DependencyGraph.HasCycle()` is called inside `CreateTaskCommand` and `CreateGroupCommand` **before** the task or group is persisted. It validates the entire chain — task `NextTaskId` links, group `Next` pointers, and parent‚child relationships — as one unified directed graph. If a cycle is detected, throw `DependencyCycleException` — the API maps this to HTTP 422.
|
||
|
||
### Agent isolation is absolute
|
||
|
||
Agents run inside their assigned container. The domain model enforces that an agent cannot reference tools or volumes outside its declared container spec. Never pass cross-container paths in task prompts.
|
||
|
||
---
|
||
|
||
## 4. Adding a New Feature — Step-by-Step
|
||
|
||
Follow this checklist to keep the architecture clean.
|
||
|
||
1. **Domain first.** Does the feature need a new entity, value object, or enum? Add it to `UltimateAgents.Domain`. Write unit tests for any domain logic.
|
||
2. **Define the interface.** If the feature needs infrastructure (DB query, external call), add an interface to `Application/Interfaces/`.
|
||
3. **Write the use case.** Add a command or query class in `Application/`. Use a plain service class to implement the handler. No Dapper here.
|
||
4. **Implement infrastructure.** Implement the new interface in `Infrastructure/`. Use Dapper and schema scripts for persistence.
|
||
5. **Expose via API.** Add or extend a controller in `Api/Controllers/`. Add a DTO and a FluentValidation validator.
|
||
6. **Wire up DI.** Register the new service/repository in `Program.cs` or a dedicated `ServiceCollectionExtensions` class.
|
||
7. **Add CLI command** (if user-facing). Add a subcommand to the appropriate file in `Cli/Commands/`.
|
||
8. **Write tests.** Unit-test domain + application layers; integration-test the API endpoint with a local SQLite database.
|
||
|
||
---
|
||
|
||
## 5. Database & Migrations
|
||
|
||
### Database schema
|
||
|
||
- The SQLite schema is defined in `Infrastructure/Persistence/DatabaseSchema.sql`.
|
||
- Use Dapper for lightweight mapping and parameterized SQL queries.
|
||
- Many-to-many relationships are represented using explicit join tables.
|
||
- Schema updates are managed via versioned SQL scripts, not EF Core migrations.
|
||
|
||
---
|
||
|
||
## 6. Working with the Orchestrator Engine
|
||
|
||
`OrchestratorEngine` runs as an `IHostedService`. It watches for eligible tasks and drives them through their lifecycle.
|
||
|
||
### Task lifecycle states
|
||
|
||
```
|
||
Created → Queued → Running → Succeeded
|
||
↘ Failed → Retrying → Succeeded
|
||
↘ Escalated
|
||
```
|
||
|
||
### Rules for state transitions
|
||
|
||
- Only `OrchestratorEngine` is allowed to write task execution state. Controllers must use the `StartTaskCommand` application service, which queues the intent — never set state directly from a controller.
|
||
- State is persisted in SQLite after every transition for status queries. The local database is the source of truth.
|
||
- Goto task: when a task completes and `GotoTaskId` is set, the engine merges `GotoPrompt` into the target task's `Prompt` (append by default) before transitioning. Never overwrite the original prompt.
|
||
|
||
### Group execution
|
||
|
||
Tasks are organised into `TaskGroup` entities. A group has a `GroupType` and a `Children` list of `GroupChildRef`. Children can be tasks (any type) or nested `TaskGroup` instances.
|
||
|
||
- `Sequential` — the engine activates the first child; when it completes, activates the next sibling child in order. After the last child completes, the group itself is marked complete; the engine then activates `group.Next` (if set).
|
||
- `Parallel` — all children are dispatched simultaneously. When every child is complete, the group is marked complete and `group.Next` is activated.
|
||
- Nested groups follow the same rules recursively — a nested group must itself be fully complete before its parent considers it done.
|
||
- `group.Next` is a `GroupChildRef` and can point to a task or another group.
|
||
- The engine uses local coordination and database transactions to prevent the same group from being activated twice.
|
||
|
||
---
|
||
|
||
## 7. Agent Matching
|
||
|
||
Agent selection uses the Strategy pattern. The matching strategy is resolved from the agent's `ChooseRule` field.
|
||
|
||
```csharp
|
||
// Registered in DI as keyed services
|
||
services.AddKeyedScoped<IAgentMatchStrategy, ExactSingleTagStrategy>(AgentMatchRule.ExactSingle);
|
||
services.AddKeyedScoped<IAgentMatchStrategy, SupersetTagStrategy>(AgentMatchRule.Superset);
|
||
services.AddKeyedScoped<IAgentMatchStrategy, AllRequiredTagsStrategy>(AgentMatchRule.AllRequired);
|
||
```
|
||
|
||
**Order of precedence when multiple agents match:** prefer agents with the fewest extra skills (most specific match). If equal, pick the agent with the lowest current load.
|
||
|
||
**No match found:** the dispatcher retries matching after the configured `backoff` period. After `max_retries`, the task is escalated.
|
||
|
||
---
|
||
|
||
## 8. Scheduling & Events
|
||
|
||
### Schedule types
|
||
|
||
| Type | Behavior |
|
||
|------------------|-----------------------------------------------------------------------|
|
||
| `Immediate` | Queued for dispatch as soon as the task is created. |
|
||
| `TimeBased` | Cron or ISO 8601 repeat expression. Evaluated by `SchedulerService`. |
|
||
| `DatetimeEvent` | Fires once at the specified UTC datetime. Use ISO 8601 format. |
|
||
|
||
This version does not use signal-based schedule triggers. Tasks are scheduled using immediate, time-based, or datetime-event rules. Internal coordination is handled through in-process C# events dispatched via `IEventDispatcher`.
|
||
|
||
---
|
||
|
||
## 9. Container Runtime
|
||
|
||
The `IContainerRuntime` interface abstracts the local isolated runtime. Switch the implementation via config:
|
||
|
||
```json
|
||
"RuntimeAdapter": "LocalSandbox"
|
||
```
|
||
|
||
### Local development
|
||
|
||
Use the local runtime adapter. No Docker or Kubernetes runtime is required.
|
||
|
||
### Runtime spec validation
|
||
|
||
Before `StartAsync` is called, the `ContainerSpec` is validated:
|
||
- All declared tools must have a `name` and `version`.
|
||
- `resources.cpu` and `resources.memory` must be positive values.
|
||
- `network_policy` must be explicitly declared (empty is allowed, but `null` is rejected).
|
||
|
||
> **Never pass user-controlled strings directly into container exec calls.** Always use the tool definition's pre-declared `usage_guidelines` to build the command. This prevents command injection.
|
||
|
||
---
|
||
|
||
## 10. Error Handling & Escalation
|
||
|
||
### Controller layer
|
||
|
||
All controllers are wrapped by the global exception middleware in `Api/Middleware/ExceptionMiddleware.cs`. Do **not** add try/catch blocks in controllers for domain exceptions — let the middleware handle them.
|
||
|
||
```csharp
|
||
// Good
|
||
var result = await _createTaskService.ExecuteAsync(command, ct);
|
||
return Ok(result);
|
||
|
||
// Bad — don't do this
|
||
try { ... } catch (TaskNotFoundException) { return NotFound(); }
|
||
```
|
||
|
||
### Escalation flow
|
||
|
||
1. Agent reports failure with `Escalation.Reason` and `Escalation.Prompt`.
|
||
2. `EscalationHandler` checks the task's `RetryPolicy`.
|
||
3. If retries remain: re-queue the task with incremented attempt counter and apply backoff.
|
||
4. If retries exhausted: publish an `EscalationEvent` and set task state to `Escalated`.
|
||
5. The human operator receives the escalation via the configured notification channel (webhook or log alert — configured externally).
|
||
|
||
**Always populate `Escalation` on failure.** A missing escalation object on a failed task is treated as a bug and logged at `Error` level.
|
||
|
||
---
|
||
|
||
## 11. Testing Guidelines
|
||
|
||
### Unit tests (`UltimateAgents.Domain.Tests`, `UltimateAgents.Application.Tests`)
|
||
|
||
- Test domain rules in isolation — no database, no Docker.
|
||
- Mock `ITaskRepository`, `IContainerRuntime`, and `IEventDispatcher` with Moq.
|
||
- Cover every `ComplexityEvaluator.Evaluate()` branch.
|
||
- Cover cycle detection: test graphs with cycles, linear chains, and parallel groups.
|
||
|
||
```bash
|
||
dotnet test tests/UltimateAgents.Domain.Tests
|
||
dotnet test tests/UltimateAgents.Application.Tests
|
||
```
|
||
|
||
### Integration tests (`UltimateAgents.Api.Tests`)
|
||
|
||
- Use local SQLite or file-backed database instances for integration testing.
|
||
- Test the full HTTP request → DB → response cycle.
|
||
- Seed data via the configured persistence layer in the test fixture, not via the API.
|
||
|
||
```bash
|
||
dotnet test tests/UltimateAgents.Api.Tests
|
||
```
|
||
|
||
### Coverage target
|
||
|
||
- Domain layer: **≥ 90%** line coverage.
|
||
- Application layer: **≥ 80%** line coverage.
|
||
- API controllers: covered by integration tests, not unit tests.
|
||
|
||
---
|
||
|
||
## 12. Logging Conventions
|
||
|
||
Use **Serilog** with structured properties. Always include correlation context.
|
||
|
||
```csharp
|
||
_logger.LogInformation(
|
||
"Task {TaskId} transitioned to {NewState} by agent {AgentId} in container {ContainerId}",
|
||
task.Id, newState, agent.Id, container.Id);
|
||
```
|
||
|
||
**Mandatory fields for task/agent log entries:**
|
||
|
||
| Field | Description |
|
||
|---------------|------------------------------------------|
|
||
| `TaskId` | Guid of the task |
|
||
| `AgentId` | Guid of the acting agent (if applicable) |
|
||
| `ContainerId` | Runtime container ID |
|
||
| `Event` | Short event name (e.g. `TaskStarted`) |
|
||
| `ErrorContext`| Exception message / stack trace on error |
|
||
|
||
**Log levels:**
|
||
|
||
| Level | When to use |
|
||
|-------------|----------------------------------------------------------|
|
||
| `Debug` | Detailed internal flow (disabled in production) |
|
||
| `Information` | State transitions, task/agent lifecycle events |
|
||
| `Warning` | Retries, unexpected but recoverable situations |
|
||
| `Error` | Escalations, unhandled exceptions, data integrity issues |
|
||
|
||
Never log task `Prompt` content at `Information` or above — it may contain sensitive instructions. Use `Debug` only.
|
||
|
||
---
|
||
|
||
## 13. Common Pitfalls
|
||
|
||
| Pitfall | Why it happens | Fix |
|
||
|---|---|---|
|
||
| `DependencyCycleException` at runtime | A `next_task` or `next_group` chain forms a cycle that wasn’t caught at write-time | Always call `DependencyGraph.HasCycle()` in `CreateTaskCommand` and `CreateGroupCommand` before saving |
|
||
| Agent never matched | Agent’s `TaskTags` don’t intersect with task’s `TaskTags` | Check `ChooseRule` on the agent; verify tags are using the same enum string values |
|
||
| Cache state out of sync with DB | The local state cache or temporary store is stale compared to the database | Write to SQLite first in a transaction, then update any local cache. Never update cache before the DB commit. |
|
||
| Container start fails silently | `IContainerRuntime.StartAsync` threw but the exception was swallowed | Always `await` container calls; propagate exceptions to `EscalationHandler` |
|
||
| Foreach tasks run in wrong order | `ForeachTask.ForeachTemplateTaskId` set but `next_task`/`previous_task` links not generated | `ForeachExpander` in the application layer must set ordering links on generated task instances |
|
||
| Double-scheduling in multi-instance deploy | Two API instances pick up the same eligible task | Use local database locking and coordination; ensure task state transitions are serialized through SQLite. |
|
||
| User input injected into container commands | Prompt strings passed directly to `ContainerRuntime.ExecAsync` | Only use pre-declared tool `usage_guidelines` to construct exec commands; validate/sanitize any user-supplied values |
|
||
|
||
---
|
||
|
||
## 14. Environment Variables & Config
|
||
|
||
All settings follow the ASP.NET Core configuration hierarchy: environment variables override `appsettings.json`.
|
||
|
||
| Environment Variable | Default | Purpose |
|
||
|------------------------------------------|----------------------------|--------------------------------------|
|
||
| `Database__ConnectionString` | (required) | SQLite connection string or file path |
|
||
| `ContainerRuntime` | `LocalSandbox` | Local runtime adapter name |
|
||
| `Jwt__Authority` | (required in production) | OIDC authority URL |
|
||
| `Jwt__Audience` | `orchestration-api` | JWT audience claim |
|
||
| `Orchestrator__RetryPolicy__MaxRetries` | `3` | Global default max retries |
|
||
| `Orchestrator__RetryPolicy__Backoff` | `exponential` | `fixed` or `exponential` |
|
||
|
||
For local development, copy `appsettings.Development.json.example` to `appsettings.Development.json` and fill in values. This file is `.gitignore`d.
|
||
|
||
---
|
||
|
||
## 15. Useful Commands
|
||
|
||
```bash
|
||
# Run all tests
|
||
dotnet test
|
||
|
||
# Run only unit tests
|
||
dotnet test --filter "Category=Unit"
|
||
|
||
# Add a new schema migration
|
||
# (Use SQL scripts or schema versioning tools as appropriate.)
|
||
|
||
# Roll back last schema change (dev only)
|
||
# (Use SQL script rollback or schema versioning practices.)
|
||
|
||
# Watch-mode for API development
|
||
dotnet watch run --project src/UltimateAgents.Api
|
||
|
||
# Build the CLI as a self-contained binary
|
||
dotnet publish src/UltimateAgents.Cli -c Release -r linux-x64 --self-contained
|
||
|
||
# Check for vulnerable NuGet packages
|
||
dotnet list package --vulnerable --include-transitive
|
||
|
||
# Format code
|
||
dotnet format
|
||
|
||
# View structured logs (if using Seq locally)
|
||
open http://localhost:5341
|
||
```
|
||
|
||
---
|
||
|
||
> **Questions?** Check [Architecture.md](Architecture.md) for design decisions, or [features.md](features.md) for the full feature specification.
|