Testing Strategy

How Avalanche AI is tested, what each tier proves, and what it deliberately does not prove. The strategy follows the upstream testing policy: prefer the real implementation, mock only the expensive or non-deterministic boundary, and verify the world rather than the agent's self-report.

Layers

Layer	What it mounts	What it proves	Where
Unit (service)	One service on a bare Context	API contracts, edge cases, error paths, disposal	packages/avalanche/*/tests/*.spec.ts
Lifecycle (orchestrator)	Real agent loop + real registry + scripted mock model	Intent → policy → plan → tools → verification → receipt, including failure and approval paths	avalanche-orchestrator/tests/orchestrator.spec.ts
Real composition	Real Loader + real shipped cordis.patch.yml + test-only base tree	Every row mounts, the CLI runs, a task routes and executes through the composed runtime, evidence and task records survive a process boundary	avalanche-app/tests/loader-composition.spec.ts

There is no separate browser tier in this phase; the dashboard is Phase 4.

Unit coverage

Package	Spec	Focus
avalanche-context	context.spec.ts	Layer precedence, origin, removal, listing
avalanche-memory	memory.spec.ts	Tier isolation, lexical recall, capacity, durable store, flush on dispose
avalanche-policy	policy.spec.ts	Rule union and precedence, default-deny, risk understatement, mainnet denial, approval lifecycle, fingerprint binding, rule registration/disposal
avalanche-evidence	evidence.spec.ts	Hash chain, tamper detection, receipts, digest verification, JSONL mirror and reload, bounded tool content
avalanche-verification	verification.spec.ts	Registry, unknown kind, thrown verifier, disposal
avalanche-tooling	gate.spec.ts	Catalog registration, duplicate rejection, failure when policy absent, network-argument resolution, deny paths, disposal
avalanche-network	network.spec.ts	Defaults, unknown network, RPC allowlist, JSON-RPC envelope, error surfacing, chain-id observation, tool registration/disposal
avalanche-agents	routing.spec.ts	Descriptor validation, keyword scoring, ties, prompt/mask installation, disposal

Lifecycle coverage

avalanche-orchestrator/tests/orchestrator.spec.ts mounts mountAgentLoopTestDependencies plus the real AgentLoop, registers a scripted MockAdapter, and exercises:

• keyword routing and tie-breaking;
• forced domain selection;
• maxSteps planning;
• awaiting-approval for a require-approval capability;
• mainnet denial via deny-mainnet-state-change;
• fail-closed behavior when the policy service is missing;
• tasksDir write-through and reload;
• inspect/list/cancel;
• blank-objective rejection.

Real composition coverage

avalanche-app/tests/loader-composition.spec.ts is the product-visible test required by the package policy. It:

1. Creates a temporary profile, links the real avalanche-app package into it, and writes a test-only cordis.yml containing only upstream base services (LLM runtime, session store, projection registry, agent registry, default-model config, tool runtime, native executor, system prompt, agent loop).
2. Applies the real shipped cordis.patch.yml through readProfilePatches and boots through the real boot/Loader path.
3. Substitutes exactly two things: the LLM adapter (a scripted MockAdapter) and the RPC transport (a stub returning 0xa869 for eth_chainId).
4. Asserts:
   • every shipped row activates, with no entry left unloaded;
   • seven domains, five catalogued tools, the baseline policy rules, and the chain-id verifier exist;
   • agent list --json prints the seven domains;
   • a submitted task routes to the developer domain, executes a real avalanche_chain_status tool call through the stub transport, and returns the scripted model text;
   • the evidence ledger contains exactly the executor-recorded tool call, the receipt cites it, the policy rule id names the granting rule, verification is verified with kind chain-id, and the disclaimer is present;
   • the task JSON and evidence JSONL are on disk; a second process boot can task inspect the record and a third can evidence verify the reloaded chain.

What the tests do not prove

• Model quality. The mock adapter returns scripted text; the tests prove the runtime drives a model and records what happened, not that a real model reasons correctly. Real-API smoke tests are an upstream tier and are not run in this fork's local gate.
• Real network behavior. The RPC transport is stubbed in the composition test; network unit tests assert call shapes against a recording transport. No test sends a transaction anywhere.
• Process-level boot from built artifacts. A bare tsx apps/cli/src/bin.ts boot requires the repository's built lib/ tree; this checkout has none, so the composition test boots the same Loader path in-process instead. See "Boot verification" below.
• Coverage gate. The upstream 100% per-file gate is a repository-wide gate; this fork's avalanche suite is run targeted. The packages are written to upstream conventions but have not been through the full partition runner.

Commands

# Avalanche suite (source resolution through tsconfig paths; no build needed)
pnpm exec vitest run packages/avalanche

# One package
pnpm exec vitest run packages/avalanche/avalanche-policy

Type-checking uses a temporary aggregate project over packages/avalanche; the exact invocation and result are recorded in PHASE_0_1_REPORT.md.

Boot verification and its current limit

The intended smoke command is:

pnpm exec tsx apps/cli/src/bin.ts --profile avalanche doctor

In a source checkout without built artifacts this fails before the avalanche rows load: upstream base packages export ./typert subpaths whose lib/typert.host.js files are build outputs, and the loader reports "10 typert contributor(s) failed to register". That is an environment prerequisite (pnpm run build), not an avalanche defect. The real composition test exercises the same Loader boot with the same shipped patch and is the authoritative gate in this checkout. Once pnpm run build has produced lib/ artifacts, the smoke command becomes meaningful; a successful doctor should list seven domains, three networks, five tools, six policy rules, one verifier, and an intact evidence chain.

Test data rules

• Chain access in tests is always a stub or a Fuji read; no mainnet calls.
• Temporary directories are created per test and removed in afterEach, including on failure.
• The composition test owns its profile home and disposes the booted context before the next boot.
• Scripted adapters and transports live in the test file or the upstream mock-adapter.ts; no shared mutable global state.

Adding a test for a new tool

1. Unit-test the tool through avalancheTooling.register and a recording transport.
2. Add a lifecycle assertion only if the tool changes orchestration behavior.
3. If the tool is visible in the shipped profile, add it to the composition test's catalog assertion and prove the policy verdict it receives.
4. Never assert success from model text; assert the evidence record and, where applicable, the verification result.