Agent capabilities

Semaloop agents are capable of navigating your UI like a human. Under the hood, we use computer vision models to understand the interfaces that we’re looking at, and reasoning models to figure out what to do next.

Given this approach, it means that you can do anything that a human is capable of doing on the device (with some minor limitations), just instruct the agent in the test spec.

Each section below starts with the devices it works on: iOS real devices, iOS simulators, Android emulators, or all three. Supported platforms has the same information side by side.

Built-in accounts

Every device comes with accounts the agent can use without you setting anything up:

	iOS real devices	iOS simulators	Android emulators
Email address	Yes	Yes	Yes
Google account	Yes	Yes	Yes
Apple ID	Yes	Yes	No
Phone number	Yes	No	No

None of these need a secret. Ask for the account in the test spec, such as “Sign in with Google”, “Sign in with Apple” or “Sign up with the device’s phone number”, and the agent uses the one on the device.

In addition to these, you can also provide your own credentials for a test run. See the secrets and credentials documentation.

On top of that, we’ve built capabilities into our agents that either streamline an existing set of actions (e.g. having to open Settings and navigate many menus to update the device region), or unlock something a human tester couldn’t easily do at all (e.g. throttling the network, or spoofing GPS). Each one is described below, along with the kind of phrasing that triggers it. You don’t need to name the capability itself, though a few of them do turn on your wording: crashing the app and force-quitting it are different actions, and the agent picks between them from the words you use.

Voice input

Works on: iOS real devices, Android emulators

The agent can speak into your app’s microphone, either as an exact phrase in a language of your choosing, or as a generated multi-person conversation on a topic you describe.

In your test, say something like:

Play a discussion about tennis

If the phrase has to land in a listening window your app opens (tapping a microphone button, say), put the phrase and the tap in the same step. The agent then has what it needs to render the speech before the tap, and start playback the instant the microphone opens — some apps stop listening before audio that’s rendered afterwards ever arrives. So a spoken step reads like:

Tap the microphone and say “add milk to my shopping list”

GPS location

Works on: iOS real devices, iOS simulators, Android emulators

The agent can put the device anywhere, either by naming a place (which we geocode for you) or by giving explicit coordinates. The location persists for the rest of the test, so it’s worth setting before the steps that depend on it.

In your test, say something like:

Set device location to the Eiffel Tower

Network conditions

Works on: iOS real devices, Android emulators

The agent can slow the network to a standard profile — Edge, 2G, 3G, LTE or WiFi — to check loading states, timeouts and error handling. The condition persists for the rest of the test until it’s cleared. To test with no connection at all, use airplane mode.

In your test, say something like:

Open app with a slow network

Signing in with a saved password

Works on: iOS real devices, iOS simulators, Android emulators

The agent can fill a password field from a secret without the password ever appearing in the test spec. It picks the credential matching the account the form is authenticating, so several accounts can coexist: an app login, a Google account, an Apple ID and a sandbox purchase account.

In your test, say something like:

Login with a test account

Signing up with a fresh email

Works on: iOS real devices, iOS simulators, Android emulators

The agent can generate a unique-per-run email address on the device’s mailbox, so a signup flow is repeatable against every build instead of failing the second time with “account already exists”.

In your test, say something like:

Sign up with a unique email

Magic links and email verification codes

Works on: iOS real devices, iOS simulators, Android emulators

The agent reads the real inbox behind the device’s email address and either clicks the link it receives or types the code back into your app. It waits up to two minutes for the message to arrive.

The inbox it reads is the device’s own mailbox, managed by Semaloop, including any unique address generated for the run as described above. It can’t read mail sent to your own accounts, such as those in a user account secret. To sign in accounts like these with an emailed code or link, see sign-in and user state.

In your test, say something like:

Login

Phone numbers and SMS codes

Works on: iOS real devices

Every real iPhone has its own phone number, and the agent can type it into your app and enter the SMS code your app sends back. There’s nothing to set up: no secret to add and no number to put in the spec. Just ask for it in the test spec, and the agent uses the device’s number.

The number stays with the device from run to run, so your app will usually see a returning user. See sign-in and user state.

In your test, say something like:

Sign in with the device’s phone number

Incoming phone calls

Works on: iOS real devices

The agent can ring the device in the middle of a test, so you can check your app copes when a call interrupts it, for example that a recording keeps going or picks up where it left off. The call rings for up to 20 seconds, and the agent can answer it, decline it or let it ring out, whichever your test asks for. An answered call ends on its own after a short time. While a call is active the microphone belongs to the call, so put any speaking before or after it.

In your test, say something like:

Start a recording and say “the budget review is on Thursday”. Receive a phone call and decline it, then check the recording is still running.

Deeplinks

Works on: iOS real devices, iOS simulators, Android emulators

The agent can jump straight to a deep-linked screen or state, which is how you test the destinations behind push notifications, referral links, and universal links or App Links without needing to send the notification.

In your test, say something like:

Open example.com

Force-quitting and resuming

Works on: iOS real devices, iOS simulators, Android emulators

The agent can background your app and return to it, or force-quit it from the recent apps screen and relaunch, just as a real user would drop out of a flow and come back.

In your test, say something like:

Force quit app

Airplane mode

Works on: iOS real devices, Android emulators

The agent can toggle airplane mode, for testing genuinely offline behaviour including cached content and queued writes.

In your test, say something like:

Enable airplane mode

Language and region

Works on: iOS real devices, iOS simulators, Android emulators

The agent can change the device’s system language and region, so localisation can be checked without maintaining a separately configured device. Either can be set on its own: setting just the region is how you can test currency and date formatting in isolation. Android only offers certain language and region pairs, so if the pair you ask for doesn’t exist (French with a US region, say), the language changes and the region is dropped.

In your test, say something like:

Change the device language and region to French

The agent can also change the device’s time zone, to check how dates and times look to a user somewhere else:

Set the device’s time zone to Auckland

External files

Works on: iOS real devices, iOS simulators, Android emulators

The agent can pull a file in from a link, either a Google Drive share URL or a direct download link (a .pdf, .jpg or .mp4 URL that returns the file itself rather than a preview page), and place it on the device, ready for an upload or import flow to pick up. It lands where your app’s document picker will find it: in the Files app under Semaloop Helper on iOS, and in the Download folder on Android. You can name it explicitly if a later step needs to refer to it by name.

The agent can also save images to the Photos library, keeping their metadata, such as the date and location they were taken, and unpack a ZIP archive, which is the quickest way to load several files at once. Large sets take a while to load, so give each test only the files it needs.

In your test, say something like:

Save example.com/image.jpg to files

Health data

Works on: iOS real devices, iOS simulators

The agent can put data into Apple Health for your app to read, so features built on HealthKit have something to show on a device that otherwise has none. It can set:

• Daily totals: steps, walking and running distance, active energy, flights climbed and water, for any day in the past two years.
• Measurements: heart rate, resting heart rate, heart rate variability, weight, body fat, blood glucose, blood oxygen, respiratory rate, body temperature and VO2 max, at a time of day you choose.
• Workouts: running, walking, cycling, swimming, hiking, strength training, HIIT, yoga or other, with a start time and duration, and optionally distance, active energy and average heart rate.
• Sleep: bedtime and wake time, and optionally how much of it was deep and REM sleep.

Every test starts with no Health data from Semaloop, so set it up at the start of any test that needs it. Your app still asks the device for permission to read Health data, and the agent accepts that prompt like any other.

In your test, say something like:

Add 8,000 steps a day for the past week, and a 5 km run yesterday morning

Biometrics

Works on: iOS simulators

The agent can enrol a biometric (Face ID or Touch ID) before your app’s biometric check, and then send either a matching or a non-matching biometric, so the negative path is testable too. It can also remove enrolment, to see what your app does on a device with no biometric set up.

In your test, say something like:

Log in with Face ID, then lock the app and fail the Face ID check

The clipboard

Works on: iOS real devices, iOS simulators, Android emulators

The agent can both read the clipboard and write to it. Reading it verifies what a share or copy button actually produced. Writing to it sets up a paste without having to type the value somewhere else first.

In your test, say something like:

Copy the invite link and check the clipboard contains a semaloop.com URL

Locking and unlocking the device

Works on: iOS real devices, iOS simulators, Android emulators

The agent can lock the device, and wake and unlock it again. This covers what your app does when it’s interrupted mid-flow and resumed a moment later, and any protection you put behind the lock screen. On Android, the agent can’t yet unlock a device protected by a passcode.

In your test, say something like:

Start a transfer, lock the device, then unlock it and check the transfer screen is still there

Crashing the app

Works on: iOS real devices, iOS simulators, Android emulators

Distinct from a force-quit: the agent can terminate your app abruptly, the way the operating system kills a misbehaving process, without the app getting a chance to save state. Use it to check that an abnormal exit doesn’t lose or corrupt in-progress work.

In your test, say something like:

Add three items to the basket, then crash the app and reopen it

Reading device state

Works on: iOS real devices, iOS simulators, Android emulators

The agent can read the device’s current orientation, whether the screen is off, which app is in the foreground and the device’s date and time. This is mostly useful for checks that depend on state your UI might not display. Note that while the agent can read the orientation, it doesn’t set it.

What the agent can’t do

A few things aren’t possible today. Leave them out of your tests, or contact us if one matters to you:

• Use two devices in one test. Each test runs on one device. To test something that involves another user, such as accepting an invite, act as the other user through a custom tool that calls your API.
• Change the device’s battery or storage, for example to test low battery or a full disk.
• Install other apps, such as WhatsApp or Google Fit, or sign in to Facebook. Only your app and the apps that come with the device are available.
• Act within a precise time. Each step takes a few seconds while the agent looks at the screen and decides what to do, so a test can’t rely on doing something within a second or two of something else.
• Set the orientation. The agent can read it, but not change it.
• Compare against a reference image or design, such as a Figma file. The agent judges what’s on screen against what your test describes.
• Watch animations, transitions or video play out. The agent looks at the screen between its steps, so it sees where an animation ends up but not how it got there.
• Pay with Apple Pay. Sandbox accounts cover in-app purchases and subscriptions, but the agent can’t add a card to Wallet.