Skip to main content

Records and metrics

You may take the records for a report card. They are not. A record is the material for following a single run. Whether that run went well as a piece of work is not something the record alone can tell you.

This page explains how to read a run record, and what the numbers do not represent. How to go about reading them is in the Record Guide.

A run record is followed through five questions​

A run record exists to follow one run from where it began to how it ended. Request is whose setting it was and what set it off, Evidence is what it used as the basis for its judgement, What was done is which operations the agent performed and how far their effects could be confirmed. Confirmation is, on a review request's row, the decision on that review request, and Result is how the run ended.

String the five together and you can follow what it took in, what it looked at, what it did, what people decided, and where it ended. Conversely, you cannot judge the whole run from the result alone, nor judge the business outcome from "What was done" alone.

"What was done" shows the agent's operations​

What is listed here are the operations the agent called. What the platform itself does — reading the configuration at start-up, taking a copy of the plan, saving intermediate state — is not. What you want to know is which operations the agent called, so the platform's internal activity is kept but not listed in this field.

There is one more thing to watch for when reading it. Having performed an operation and having confirmed its effect are written separately.

"Performed" means the operation was allowed and ran. "Effect confirmed: succeeded", "Effect confirmed: failed" and "Effect confirmed: error" mean the result could be confirmed. "Attempted only (effect not recorded)" means the call is known but the result is not recorded. "Denied (reason)" means the execution rules, or limits fixed on the product side, stopped it.

"Attempted only" is not a failure. It means nothing could be confirmed. Do not read it as failure.

The result shows how the run ended​

What the result field shows is how that run ended — not whether the business goal was met.

"Completed" means the agent neither reported that it could not finish nor said so in its final result. Reports only work in the direction of lowering the result. A self-declared "done" is not used in the judgement, and a run that ends with no report at all is still "Completed", unless its final result said it could not finish. So a run that changes nothing still comes out as "Completed". Whether that is a problem is yours to decide.

A tool failing part-way does not change the result. Tool failures stay in "What was done" but are not counted in the result, because an agent trying another way is ordinary behaviour.

When the ending is not "Completed", the word tells you which. If the agent reported that it could not finish, or said so in its final result, "Failed (reported by agent)"; on an abnormal exit, "Failed"; on running out of time, "Timed out"; when a person or the platform stopped it, "Failed (stopped)"; when the platform forbade the run itself, "Not allowed by rules". If the run never started, it is "Nothing to do".

Blank, zero and "Nothing to do" are different​

Three things are easily confused.

Blank means no value was obtained. A reason comes with it — nothing was recorded, it is not something measured, it has been deleted and cannot be viewed. Zero means the count came to zero. Do not read blank and zero as the same thing.

"Nothing to do" is neither. It is a result, and it means the run never started — because there was nothing to process, for instance. It is not a gap in the record.

Metrics are counts over the records​

Metrics are counts taken over the records. So what you have to watch when reading the records shows up in the numbers too.

Agent completion rate is the runs that completed normally, divided by the runs that ended. The numerator takes only those whose result came out as “Completed” — not the timed out, the stopped, those not allowed by rules, or the failed. And that “Completed”, as the previous section says, only means it was not shown to be unfinished, so a high number does not mean the work went well. Some runs are left out of the count: nothing to do, still running, and those with records missing are excluded. A retry counts as one run of its own, so trying the same job three times makes three.

Some of the review numbers include automatic processing at the deadline. The counts of send-backs and rejections, and the wait for review, include what was processed automatically once the deadline passed. Other review numbers treat it differently, so when you read them as a record of what people did, check each metric's definition in the Results Reference.

Judge the business outcome from the record and the output​

Everything above comes to one point. A run record is not a report card.

If you want to know whether it went well, looking at the numbers is not enough. Follow the run through the five questions, and judge from "What was done" together with the actual output. What the record answers is what happened; whether that was good enough is yours to decide.

  • Record Guide — reading the run history, what it takes to open the body, exporting the trail
  • Results Reference — what each result word means and when it applies, the agent completion rate formula, the review metric definitions
  • Human review — the mechanism behind the review numbers