Best practices

There are three key things to keep in mind when authoring a test in Semaloop:

Give the agent context, not assumptions. The agent can only act on what it can see or what you tell it. “Log in with a premium account” means nothing to the agent unless you also give it a way to get there, a secret, a seeded test account, or a setup step that puts the account in that state before the test runs.
Describe the goal, not the taps. Say what the user’s trying to do and what should be true after. Keep the data (amounts, names, quantities), since that’s what defines a successful outcome, but drop button names, tap order, and screen names.
Check something durable. Anchor to what stays on screen (an item in a list, an updated value), not spinners, toasts, or banners that vanish.

Following these principles you can avoid writing a poorly written test like this:

Click the ‘Log in’ button then sign in with email ‘newuser@test.com’ and password ‘Password123’. If sign in shows “Success,” go to Settings > Delete account. Check you can see “Welcome to my app,” then click ‘Sign up,’ and enter ‘newuser@test.com’ and password ‘Password123.’ Tap the first product thumbnail on the home screen, tap “Add to Basket,” tap the basket icon in the top right, tap “Checkout,” tap “Confirm Payment,” and check the toast says “Order placed!”

Instead, you can write an excellent test that withstands all the UI shifts your application can throw at it:

Reset the test account, then complete sign up with the newly reset account. Add the first item in the catalogue to the basket, and complete checkout. Verify the order confirmation screen shows the correct item and total.

Semaloop also allows you to test non-deterministic flows, like the output from an agent, where you can assert on an outcome rather than exact wording. For example:

Open the assistant and ask “what’s my account balance?” Verify the response is relevant, includes a specific number and doesn’t say the account couldn’t be found.

Set up state in the test, not as a precondition

The agent can’t tell what state an account is in before a test starts, so a line like “Precondition: signed in with an account that has no family” doesn’t work. Either give the agent an account known to be in that state, through a secret or a custom tool, or have the test put it in that state itself:

Sign in. If there’s a profile photo, remove it. Check the profile shows your initials instead.

Pop-ups and system prompts

When a permission prompt, tracking prompt, cookie banner or similar pop-up appears, the agent decides how to handle it from what your app is doing at the time. If your test depends on a particular choice, such as denying location access, say so in the test.

A/B tests and feature flags

When an experiment or a flag changes what’s on screen, the agent works around anything unexpected, such as a new tooltip or a moved button, and carries on towards the goal. Describe what the user is trying to do rather than the exact screens, so the test passes whichever variant the device gets:

Search for a train from London to Manchester tomorrow and add the cheapest ticket to the basket.

If a flag hides the feature a test checks, the test fails, so make sure the flag is on in the builds you send to Semaloop, or turn it on for the test’s account with a custom tool.

Waiting and test length

Most tests take a few minutes, and long or complex ones can take up to an hour. Tests run faster when they cover one goal, and when slow setup, such as creating an account or loading data, happens through a custom tool rather than through the app’s screens.

How long the agent waits for something slow, such as a background scan or an AI reply, depends on what’s on screen. If your app needs longer, say so:

Import the photo library, then wait up to 5 minutes for the timeline to appear.

Over-the-air updates

An over-the-air update, such as an Expo update, isn’t a new build, so it doesn’t start a run on its own. If your app only applies an update on its next launch, say so in the test:

Open the app and wait 10 seconds, then force quit and reopen it so the latest update is applied.