Skip to content

Blog

Measured against your best setup

How the first evaluation would decide whether Assent is worth it: one outcome, three conditions and results that come with their counts.

Design noteThis note describes how the first evaluation is designed. Intent, not results.

Working notederived from the governing documents1 min read
Against your best setup, not a weak oneConcept diagram
  1. 01Each change class is compared three ways
    • your normal workflow
    • your improved existing workflow, including your agent vendor’s own orchestration where you have it
    • Assent
  2. 02One outcometotal human effort per successfully delivered, policy-compliant change

One outcome

The question is total human effort per successfully delivered, policy-compliant change, compared within the same kind of change and observation window.

Unsuccessful, abandoned and blocked attempts count in the effort. If nothing qualifying is delivered, the ratio is undefined, not zero. Completing only the easy work doesn’t count as improvement.

Source: Metrics and economic evidence

Against your best setup, not a weak one

Each change class is compared three ways: your normal workflow, your improved existing workflow, including your agent vendor’s own orchestration where you have it, and Assent. Measuring against an unprepared prompt would flatter anything.

Source: Decision and assumption register

What counts as effort

Minutes are recorded for context preparation, exploration, deciding, approvals, operating agents, review, rework, administering the product and support. Each interval is counted once and recorded by person or role and activity, without unnecessary sensitive content. It is reported only at team level, never ranking individuals.

Our own help counts too. Concierge time is recorded separately; it is not free automation, and moving work from your team to ours isn’t eliminating it.

Source: Metrics and economic evidence

Results with their counts

Every result states its sample counts and how changes were assigned, the agent and model versions, the execution model, the assistance level, the observation window, and the failures, exclusions and uncertainty. Absolute minutes and counts appear beside any percentage, along with a note when the sample is too small to support a conclusion.

Source: Metrics and economic evidence

What would make us stop

The stop conditions are written down before results exist, each with the decision it would put on the table. If your best existing setup, vendor orchestration included, performs as well, we would consider falling back to governing agents rather than running them, improving context and tooling, or stopping. If buyers would rather govern their own agents than have them operated, the positioning would be revisited. If execution cost exceeds the value for a kind of change, managed execution for it would stop or narrow.

Source: Validation program

Design partners

Bring one recent cross-team change.

We reconstruct it with you before showing a solution: the decisions, their owners and the effort, agent operation included.