Skip to content
Semaloop Docs

Reading test results

Every Semaloop test run ends in success, failure or error, and comes with an interaction timeline, a video replay, system logs and network activity that show what happened. The three states are:

  • Success: The agent was able to verify the functionality was working.
  • Failure: The agent was unable to verify that your functionality was working as expected.
  • Error: The agent hit a problem, like a model provider outage, and the Semaloop team have been informed.

On completion, there are a number of key pieces of context we present back to you to help you understand what happened. For all tests we show:

  • Interaction timeline: A step-by-step trace of all of the steps our agent took, as well as the reasoning applied to each step.
  • Video replay: A high FPS recording of what happened during the test run.
  • Audio capture: If there was any microphone input or media playback we include that in the video replay.
  • System logs: Anything you’ve chosen to output during the execution of your test.
  • Network activity: We capture all responses, headers, and timings so you can understand what happened under the hood.
  • Issues: Any UI/UX inconsistencies that our agent found along the way, e.g. a slow loading page, or text that gets cut off the screen.

In the case of a failure, we also have:

  • Root cause analysis: We combine all of the context available to our agent into a short summary, highlighting what we think has happened to cause the failure.
  • Suggested fixes: In addition to just providing a failure analysis, we can also suggest how to fix the problem that we’ve encountered.

See the debugging failures documentation.

There are also some handy keyboard shortcuts you can use for interacting with the trace:

  • j and k to move through the trace steps
  • Space to play or pause the video

Automatic re-runs

Section titled “Automatic re-runs”

When a test that ran automatically on a new build fails, Semaloop runs it again once, on the same device, to check the failure wasn’t a one-off. Slack notifications and webhooks report the result of the re-run, so you only hear about a failure once both attempts have failed. Manual runs and errors aren’t re-run this way, though a run cut short by a model provider outage is retried. Re-runs don’t count towards your executions, and they can’t be turned off.