Reading test results

When a test run finishes in Semaloop, there are three states that can manifest:

Success: The agent was able to verify the functionality was working.
Failure: The agent was unable to verify that your functionality was working as expected.
Error: The agent hit a problem, like a model provider outage, and the Semaloop team have been informed.

On completion, there are a number of key pieces of context we present back to you to help you understand what happened. For all tests we show:

Interaction timeline: A step-by-step trace of all of the steps our agent took, as well as the reasoning applied to each step.
Video replay: A high FPS recording of what happened during the test run.
Audio capture: If there was any microphone input or media playback we include that in the video replay.
System logs: Anything you’ve chosen to output during the execution of your test.
Network activity: We capture all responses, headers, and timings so you can understand what happened under the hood.
Issues: Any UI/UX inconsistencies that our agent found along the way, e.g. a slow loading page, or text that gets cut off the screen.

In the case of a failure, we also have:

Root cause analysis: We combine all of the context available to our agent into a short summary, highlighting what we think has happened to cause the failure.
Suggested fixes: In addition to just providing a failure analysis, we can also suggest how to fix the problem that we’ve encountered.

There are also some handy keyboard shortcuts you can use for interacting with the trace: