Run Results Reference
The words used for the result of a single run, and how the KPI figures are counted. Item names follow what the screen shows. How to read the results is covered in the Record Guide, and the thinking behind the figures in Records and metrics.
The words for the result
How a single run ended is shown as one of these seven.
| Shown as | Meaning | Decided by | Agent completion rate |
|---|---|---|---|
| Completed | The agent ended without reporting that it could not finish, and its final result did not say so either. This includes runs that ended with no report at all, as long as the final result did not say so | Agent | Counted as a success |
| Failed (reported by agent) | The agent itself reported that it could not finish, or ended with a final result saying so. Without a report, the category is “Other” | Agent | Counted as a failure |
| Failed | It ended because of an exception or an abnormality in the runtime | Platform | Counted as a failure |
| Timed out | It reached the time limit and was cut off | Platform | Counted as a failure |
| Failed (stopped) | A person stopped it, or the platform cancelled it | A person or the platform | Counted as a failure |
| Not allowed by rules | The request was stopped by the platform's check on its input (the guardrail) and was not carried out | Platform | Counted as a failure |
| Nothing to do | The run never started | Platform | Not counted |
- “Completed” is no guarantee that the work went well. What was done is checked under “What was done” in the run record.
- An operation stopped by the execution rules does not change the result. Stopped operations are listed under “What was done” as denials.
- The check behind “Not allowed by rules” is not the guardrail you write for an agent (an instruction). The run record gives the reason as “Stopped by the guardrail”.
Reasons for “Failed (reported by agent)”
When an agent reports that it could not finish, it gives what to review and a comment for the person reading it (what it could not do, why, and what to do next). The category is one of the following; where the agent does not choose one, it is derived from what to review. In the run record, the category appears beside the result and as the “Cause” in the “Failure summary” at the top. The comment appears under “Agent's closing report” in the Failure summary. Where the Failure summary points first follows What to review where it narrows things down (the second table below), and otherwise the category (the table below).
| Category | Where the Failure summary points first |
|---|---|
| Could not access the connected service (authentication or permission) | Check connector settings |
| The connected service did not respond | Check connector settings |
| Restricted by the execution rules | Check execution rules |
| Required input was missing | Check step settings and inputs |
| Other | Review what was done |
Where What to review narrows things down, the Failure summary points first as follows.
| What to review | Where the Failure summary points first |
|---|---|
| Instructions (prompt) | Check step settings and inputs |
| Settings and connections (harness) | Check connector settings, Check execution rules, or Check step settings and inputs (which one appears depends on the cause; the cause is written in the explanation under What to review, or in the agent's closing report) |
| Restricted by an administrator or the system | Review what was done |
| External service | Review what was done |
| Unexpected error | Review what was done |
What happens after each result in a workflow
| Result | What happens next |
|---|---|
| Completed | It moves on to the next agent on the same step, or, if there is none, to the next step (on a step set to “Run at the same time”, to the next step once all of the results are in). If the next step is a review step, it waits for the review there |
| Failed (reported by agent) | By default, a review request is created and the run stops there. Set the step's “When the agent reports it could not finish” to “Proceed anyway” and it moves on as if completed. Set it to “Go to a specific step” and it goes to the chosen later step |
| Not allowed by rules | By default, a review request is created and the run stops there. Set the step's “When not allowed by rules” to “Close as failed” and the workflow run ends as failed. Set it to “Go to a specific step” and it goes to the chosen later step |
| Failed / Timed out | The same step is tried again. Once the retries set in “Retries per step (failed / timed out)” are used up, another failure or timeout ends the workflow run as failed |
| Failed (stopped) | It is not tried again, and the workflow run ends as failed |
| Nothing to do | It moves on (on a step set to “Run at the same time”, to the next step once the other agents' results are in) |
When you cancel the workflow run itself, it ends as “Cancelled” (not “Failed”).
In a cycle, no result stops the sweep. Sweeps continue on the schedule you set.
Runs still going, and runs with no recorded result
| State | Meaning | How it is counted |
|---|---|---|
| Running | It has not ended yet | Not counted (“Running” under Excluded) |
| Insufficient records | It has ended, but no result was recorded. The run record says “no result was recorded” | Counted as neither success nor failure (“Insufficient records” under Excluded). Even one of these makes the Agent completion rate provisional |
| Nothing to do | The result “Nothing to do” | Not counted (shown under Excluded) |
Agent completion rate
Agent completion rate = Completed ÷ finished
| Item | Content |
|---|---|
| Completed | The number of runs whose result is “Completed” |
| Finished | Completed plus “Failed / needs action”. “Failed / needs action” is the total of Failed (reported by agent), Failed, Timed out, Failed (stopped) and Not allowed by rules |
| The unit counted | One agent execution. Both a workflow step's execution and a cycle's sweep are counted. A retry counts as one too |
| Period | Runs that started in the period you chose |
| Not included | Nothing to do, running, and insufficient records |
| When nothing has finished and no record is insufficient | The rate shows “No target” (meaning there is nothing to count — not the result “Nothing to do”) |
| When a record is insufficient | It says “Records incomplete: provisional because N runs have no recorded result” (even when nothing has finished) |
The KPI items
These are the items under “KPI” in Overview. You can choose the period (values are in the Settings reference).
Run results
| Item | How it is counted |
|---|---|
| Agent completion rate | The formula above |
| Completed | The number of runs whose result is “Completed” |
| Failed / needs action | The total of Failed (reported by agent), Failed, Timed out, Failed (stopped) and Not allowed by rules |
| Typical duration (p95) | The time from start to end of runs that started and ended in the period. Sorted from shortest, it is the value at the 95% position: about 95% of the runs that ended did so within this time |
| Breakdown of results | The finished runs, by result, with counts and shares (Nothing to do is not included) |
| Excluded | The counts of runs with nothing to do, runs still running, and runs with insufficient records |
Work steps
Counted from the agent executions that ended in the period you chose. Both workflow step runs and cycle sweeps are included.
| Item | How it is counted |
|---|---|
| Counted executions | The number of executions whose result is “Completed”, “Failed (reported by agent)”, “Failed (stopped)” or “Not allowed by rules”. “Failed”, “Timed out”, “Nothing to do” and executions with no recorded result are not counted (their number is shown as “executions not counted”) |
| Tool calls | The total of the recorded tool calls of the counted executions |
| Steps | The total of the work steps of the counted executions. The steps of one execution are settled from that execution and its recorded tool calls; failed external integration calls and failed external changes are not counted, and there is a cap per execution. Executions with no recorded count are left out of the total, and their number is shown alongside |
The tenant-wide total for the current month (counted by UTC) can be seen by a system administrator under “Workflow steps this month” on the “License” screen of the admin console.
Review figures (workflow only)
These count the review requests created in the period you chose.
Review methods and track record by step (for each review step)
For each review step it shows the Step, the Review (the review step's name), the Review method, and the following two rates.
| Item | How it is counted | Automatic action at a deadline |
|---|---|---|
| Approved by people | The number a person approved ÷ the number decided. The number decided is the number a person approved plus the number sent back or rejected | What advanced automatically is in neither the numerator nor the denominator. What was sent back or rejected automatically is in the denominator |
| Send-back rate | The number sent back ÷ the number decided (as above) | What advanced automatically is not in the denominator. What was sent back or rejected automatically is included |
Review activity
| Item | How it is counted | Automatic action at a deadline |
|---|---|---|
| Reviews (per workflow run) | The number of review requests a person decided ÷ the number of workflow runs that started in the period | Not included |
| Typical wait | The median time from a review request being created to its being decided. Sorted from shortest, it is the middle value; with an even number, the shorter of the two in the middle. Requests not yet decided are not included | Included |
| Send-backs, rejections, cancellations | The number sent back or rejected, plus the number of workflow runs that started in the period and were cancelled | Included |
Insights
“Insights” takes the runs of the projects you are a member of together.
| Item | How it is counted |
|---|---|
| Agent completion rate, Completed, Failed / needs action, Insufficient records, Typical duration (p95) | The same as a project's KPI |
| Review figures | Not shown |
The KPI CSV
“CSV” under “KPI” in Overview exports the figures for the period you have chosen. Each line is one item, with the columns metric and value.
| metric | Content |
|---|---|
days | The period (days) |
success_rate | The Agent completion rate (0 to 1). Empty when nothing has finished |
success_rate_status | ok (final) / incomplete (provisional because a record is insufficient, even when nothing has finished) / none (no record is insufficient and nothing has finished) |
succeeded | Completed |
denominator | Finished |
outcome_completed and 6 more lines | The count for each result. completed = Completed, incomplete = Failed (reported by agent), failed = Failed, timeout = Timed out, stopped = Failed (stopped), denied = Not allowed by rules, skipped = Nothing to do |
excluded_skipped / excluded_running / excluded_missing / excluded_unmeasured | The counts under Excluded (nothing to do / running / insufficient records). excluded_unmeasured is always 0 |
duration_p95_minutes | Typical duration (p95), in minutes |
runs_total | The number of workflow runs that started in the period |
runs_closed_done / runs_closed_failed / runs_closed_rejected / runs_cancelled | Of those, the number that ended as done / failed / rejected / cancelled |
p95_minutes | The p95 of the time from start to end of a workflow run, in minutes |
reviews_per_run | Reviews (per workflow run) |
gate_wait_median_minutes | Typical wait (the median time from a review request being created to its being decided), in minutes |
interventions | The number of send-backs, rejections and cancellations |
Lines starting with gate_ | One line for each review step. The metric is gate_ followed by the step number and the step name. value is “review method;number of review requests;approved by people;send-back rate” separated by ;. The number of review requests counts those created on that step in the period, including ones not yet decided. The two rates range from 0 to 1, and are empty when the number decided is zero. The review method is blocking (Stop and wait for a decision) or notify_only (Notify and proceed after the deadline) |
terminated_executions / billable_executions / excluded_executions / tool_calls / steps / steps_unknown_executions / steps_formula_version | Work steps. Executions that ended / counted executions / executions not counted / tool calls / steps / executions with no recorded count / the version of the counting method |