Sign-in and user state

Most tests need a signed-in user in a known state. You get there by combining three things: the device’s built-in accounts (an email inbox, an Apple ID or Google account, and on real iPhones a phone number), secrets holding accounts you’ve created, and custom tools that call your API to create or reset users.

Approaches

There are three ways to give a test the user it needs. For sign-up or onboarding tests in an app anyone can join by email, use a fresh sign-up every run. For everything else, use users created by a tool if you can add a test-only endpoint to your backend, and an account pool if you can’t, with a reset tool if your tests change the account.

If your users sign in with an emailed link or code, Apple or Google, or a phone number, also read sign-in methods.

They differ in how much setup they need and what they can test:

	Fresh sign-up	Tool-created users	Account pool
Setup	None	An endpoint	Accounts to maintain
Tests sign-up	Yes	No	No
Concurrent runs	No change	No change	Capped by pool size
Clean state	Yes	Yes	With a reset tool

Fresh sign-up every run

Choose this approach when

You’re testing sign-up or onboarding itself. It needs no setup, and every run starts with a brand-new user.

The agent signs up with an address unique to the run, such as tobias.nordstrom+9c46b44b-…@semaloop.ai, and reads the verification link or code from its inbox.

Sign up with a unique email and verify it. Complete onboarding as a runner who trains three times a week, then check the home screen shows a weekly plan.

Watch out for:

• Using it for everything. Every test that starts this way signs up and onboards before reaching what it’s testing, so it’s slower, and fails whenever sign-up breaks. Keep it for testing sign-up and onboarding themselves.
• The + in the address. If your sign-up form rejects it, use users created by a tool instead.
• Semaloop’s mail domains. If you block unfamiliar or disposable domains, allow semaloop.ai and semaloop.cc.
• A welcome email straight after the verification email. The agent reads only the newest message, so it can miss the verification. Send the welcome once the address is verified.
• Leftover users. Each run creates one, and the run ID in its address makes them easy to find and purge.

Users created by a tool

Choose this approach when

You can add an endpoint to your backend. Once it exists, any test can get a user in any state, with no accounts to maintain, though every test that uses it fails if the endpoint breaks.

A custom tool calls your backend to create a user in whatever state the test asks for. Your endpoint makes up an email address and password, creates the user and returns them, so the test doesn’t need to supply either. For example, a POST with this payload:

{ "plan": "{{plan}}" }

which responds with:

{ "email": "test-9c46b44b@example.com", "password": "a-throwaway-password" }

The agent reads the response and signs in with those details. If your app signs in with a magic link or emailed code instead, see magic links and email codes.

Create a test user on the premium plan, then sign in as them and check that exporting to PDF is available from the report screen.

Watch out for:

• What the agent sees. The response is shown to the agent and kept with the run, so return throwaway passwords for test users only.
• Slow endpoints. Calls time out after 30 seconds and aren’t retried.
• Leftover users. Every call carries an X-Semaloop-Test-Run-Id header, so tag the users you create with it and purge them later.

Account pool

Choose this approach when

You can’t add an endpoint that creates users. It needs no code, but someone has to keep the accounts in the right state, and the number of accounts in a secret caps how many runs using it can go at once.

1. Create accounts in your backend for each state your tests need, such as free, premium and lapsed.
2. Add one account secret per state, named for it, such as “Premium, annual plan”.
3. Give each secret at least two accounts, ideally one for each run that might use it at the same time. A run claims a free account until it finishes, and waits as pending when none is free.

Sign in with the premium account and check that exporting to PDF is available from the report screen.

Watch out for:

• Secrets with a single account. These aren’t claimed, so runs at the same time all sign in to the same account.
• State drifting. A test that fills a basket, changes a setting or finishes onboarding leaves the account that way for the next run. Add a reset tool.

Adding a reset tool

Add a custom tool that puts an account back into a known state, such as a POST with this payload:

{ "email": "{{email}}" }

The agent fills in {{email}} with the account it was given. Reset at the start of the test, not the end: a failed step ends the run, so a clean-up step only runs when the test passes.

Reset the test account, then sign in, add the first item in the catalogue to the basket and complete checkout. Check the order confirmation shows the correct item and total.

Sign-in methods

The approaches above assume your users sign in with an email address and password, which works the same way with all three. Other ways of signing in need a few adjustments, covered below. If sign-in also asks for a code from an authenticator app, turn that step off for your test accounts.

Magic links and email codes

The agent can only read mail sent to Semaloop’s own inboxes, never to your users’ addresses or those in an account secret. If anyone can sign up, use fresh sign-up every run, which signs up and signs in with the run’s own address.

Otherwise, skip the email: add a custom tool that returns the latest sign-in code or link for an email address, and have the test call it once the app has sent one. The agent then types the code in, or opens the link.

Create a test user on the premium plan and sign in as them, using the sign-in code tool to get the emailed code.

If you can’t add a tool, give your test accounts a way to sign in without email, such as a password, and use an account pool.

Sign in with Apple or Google

The agent signs in with the device’s built-in accounts: a Google account, and on iOS an Apple ID. These accounts are reused from run to run, so your app will usually see a returning user. That’s fine for testing sign-in:

Sign in with Apple and check the home screen greets the user by name.

To test first-time sign-up, the account has to be deleted between runs. The simplest way is your app’s own account deletion, as long as it revokes the Sign in with Apple token, as Apple requires:

Sign in with Apple. If you land on the home screen, delete the account from settings and sign in with Apple again. Share the real email address, then complete onboarding.

Alternatively, add a tool that deletes the user for {{semaloop_device_identity_email}}, which is also the address of the device’s Apple ID and Google account. For Apple, the tool must revoke the user’s token too. Otherwise Apple treats the next sign-in as a returning one and doesn’t share the email address again. When Apple offers Hide My Email, have the test share the real address, so your backend can match the user to the device.

Phone numbers

Signing in with a phone number and SMS code works on real iPhones only, as simulators and Android emulators don’t have a phone number. See supported platforms. As with Apple and Google, the number stays with the device, so your app will usually see a returning user.

Paywalls and subscriptions

To test features behind a paywall, give the user the entitlement, with a “Premium” account secret or a tool that creates a premium user. To test the purchase itself, add a sandbox account secret, which works on real iPhones only, alongside a user who hasn’t subscribed yet.

Every run claims its own sandbox account, so add one for each purchase test you want running at once. Unlink the sandbox secret from tests that don’t make purchases, or it limits how many of them can run at the same time.

Things to keep in mind

• Protect tools that create or reset users with a token in the tool’s headers, so only Semaloop can call them. If the builds you test talk to your production backend, make sure these endpoints can only touch test users.
• Link only the secrets a test uses. New tests get every secret linked by default, and any sandbox secret, or other secret with several accounts, holds one for the whole run even if the test never uses it.
• Never put credentials in the test itself.