14 Sep 2026Methods
Evaluating agents on tasks that take longer than the context window
Most agent benchmarks finish inside one context window. The work our clients care about does not: a claims investigation, a codebase migration or a month-end reconciliation runs for hours, and the agent has to summarise, forget and recover along the way. An agent that scores well on short tasks can still fail quietly at step two hundred.
We now evaluate long tasks as a sequence of checkpoints rather than a single pass/fail at the end. Each checkpoint asks three questions: is the agent's working state still true, can it explain what it has already done, and would a fresh agent handed only that state reach the same next action? The third question isolates memory loss from reasoning error, which a final score blurs together.
The practical change for teams is in the test set. Instead of more tasks, write fewer, longer ones with planted facts early on that only matter late. If the agent's summary drops them, you find out at the checkpoint, not in production.
Back to all notes
3 Aug 2026Infrastructure
Routing between small and frontier models without losing accuracy
Sending every request to the largest model is the simplest way to be accurate and the fastest way to overspend. Sending everything to a small model is cheap until it is confidently wrong on the requests that matter most. Routing is only worth doing if the router knows which requests are which.
Our router scores each request before it is answered, using features the client can inspect: task type, input length, retrieval coverage and how often similar requests needed escalation in the past. Requests above a threshold go straight to a frontier model; the rest go small, with the small model's answer checked by a lightweight verifier that can escalate after the fact.
The threshold is set per route against the client's own evaluation suite, never against a public benchmark. When a route's quality dips below its bar, PSYVAR Core moves the threshold automatically and records why, so the cost of quality is always visible.
Back to all notes
22 Jun 2026Assurance
Audit trails that regulators can actually read
Most AI audit logs are written for the engineers who built the system: raw prompts, token counts, stack traces. A regulator, risk officer or ombudsman asking “why was this customer declined?” needs something else: the decision, the evidence it rested on, the policy that applied and who could have intervened.
We structure every trace in two layers. The machine layer keeps everything, replayable and diffable. The reading layer is generated from it and fixed in format: decision, inputs that mattered, policy version, model version, human checkpoints passed or skipped. It is short enough to print and stable enough to compare across months.
The test we use is simple: give the reading layer to someone outside the project and ask them to explain the decision back to us. If they cannot, the trail is not finished, however complete the logs are.
Back to all notes
18 May 2026Methods
Calibrated abstention: teaching models to say “I don’t know” under a budget
In regulated work a wrong answer is often more expensive than no answer. Yet most deployments treat abstention as failure and tune it away. We treat it as a product decision with a price: every abstention costs a human review, and every confident error costs whatever it breaks.
We set an abstention budget per route with the client: the share of requests their team can absorb by hand. The model is then calibrated so its confidence is meaningful, and the threshold is chosen to minimise expected cost inside that budget rather than to maximise accuracy on paper.
The result is a system that knows its limits and spends human attention where it is most valuable. It is also far easier to defend in review than one that always answers.
Back to all notes
7 Apr 2026Infrastructure
Air-gapped updates: shipping model weights through a one-way door
Classified and sovereign environments cannot pull updates. Everything that enters, from weights and evaluation suites to guardrail policies, crosses a one-way transfer, often by physical media, and must be verifiable on the far side without calling home.
We package each release as a signed bundle with a manifest that lists every file, its hash and the evaluation results it shipped with. On arrival, the on-premise installation re-runs the client's own evaluation suite before the new version can take traffic, and keeps the previous version warm for immediate rollback.
The constraint turns out to be a discipline worth having everywhere: no release is trusted because of where it came from, only because of what it does on your tests.
Back to all notes
2 Mar 2026Assurance
Red-teaming tool-using agents: attacks that live in the tools, not the prompt
Prompt-injection testing usually targets the user's message. Agents read far more than that: web pages, documents, API responses, tickets written by strangers. Any of them can carry instructions, and the agent cannot always tell data from direction.
Our red-team suites now plant adversarial content inside tool outputs: a retrieved PDF that asks for credentials, a ticket that tries to widen the agent's permissions, a search result that impersonates a policy update. We score whether the agent acts on it, flags it, or ignores it silently.
Defences sit at the route level in PSYVAR Core: tool outputs are tagged as untrusted, high-impact actions require a human checkpoint, and every attempted escalation lands in the audit trail where someone will read it.
Back to all notes