Agent capabilities

Semaloop agents are capable of navigating your UI like a human. Under the hood, we use computer vision models to understand the interfaces that we’re looking at, and reasoning models to figure out what to do next.

Given this approach, it means that you can do anything that a human is capable of doing on the device (with some minor limitations), just instruct the agent in the test spec.

Each of our agents has access to a device with:

• On device email address
• Apple ID
• Google account
• Phone number (real devices only)

In addition to these capabilities, you can also provide your own credentials for a test run. See the secrets and credentials documentation.

On top of that, we’ve built capabilities into our agents that either streamline an existing set of actions (e.g. having to open Settings and navigate many menus to update the device region), or unlock something a human tester couldn’t easily do at all (e.g. throttling the network, or spoofing GPS). Each one is described below, along with the kind of phrasing that triggers it. You don’t need to name the capability itself, though a few of them do turn on your wording: crashing the app and force-quitting it are different actions, and the agent picks between them from the words you use.

Voice input

The agent can speak into your app’s microphone, either as an exact phrase in a language of your choosing, or as a generated multi-person conversation on a topic you describe. This only works on real devices.

In your test, say something like:

Play a discussion about tennis

If the phrase has to land in a listening window your app opens (tapping a microphone button, say), put the phrase and the tap in the same step. The agent then has what it needs to render the speech before the tap, and start playback the instant the microphone opens — some apps stop listening before audio that’s rendered afterwards ever arrives. So a spoken step reads like:

Tap the microphone and say “add milk to my shopping list”

GPS location

The agent can put the device anywhere, either by naming a place (which we geocode for you) or by giving explicit coordinates. The location persists for the rest of the test, so it’s worth setting before the steps that depend on it.

In your test, say something like:

Set device location to the Eiffel Tower

Network conditions

The agent can apply one of the network profiles you’d find in Xcode and the iOS Developer settings — Offline, Edge, 2G, 3G, LTE or WiFi — to check loading states, timeouts and error handling. The condition persists for the rest of the test until it’s cleared. This only works on real devices.

In your test, say something like:

Open app with a slow network

Signing in with a saved password

The agent can fill a password field from a secret without the password ever appearing in the test spec. It picks the credential matching the account the form is authenticating, so several accounts can coexist: an app login, an Apple ID, a Google account and a sandbox purchase account.

In your test, say something like:

Login with a test account

Signing up with a fresh email

The agent can generate a unique-per-run email address on the device’s mailbox, so a signup flow is repeatable against every build instead of failing the second time with “account already exists”.

In your test, say something like:

Sign up with a unique email

Magic links and email verification codes

The agent reads the real inbox behind the device’s email address and either clicks the link it receives or types the code back into your app. It waits up to two minutes for the message to arrive.

In your test, say something like:

Login

Deeplinks and universal links

The agent can jump straight to a deep-linked screen or state, which is how you test the destinations behind push notifications, referral links and universal links without needing to send the notification.

In your test, say something like:

Open example.com

Force-quitting and resuming

The agent can background your app and return to it, or force-quit it through the app switcher and relaunch, just as a real user would drop out of a flow and come back.

In your test, say something like:

Force quit app

Airplane mode

The agent can toggle airplane mode, for testing genuinely offline behaviour including cached content and queued writes.

Real devices only.

In your test, say something like:

Enable airplane mode

Language and region

The agent can change the device’s system language and region, so localisation can be checked without maintaining a separately configured device. Either can be set on its own: setting just the region is how you can test currency and date formatting in isolation.

In your test, say something like:

Change the device language and region to French

External files

The agent can pull a file in from a link, either a Google Drive share URL or a direct download link (a .pdf, .jpg or .mp4 URL that returns the file itself rather than a preview page), and place it on the device, ready for an upload or import flow to pick up. The file lands in the Files app under On My iPhone → Semaloop Helper, which is where an in-app document picker will find it. You can name it explicitly if a later step needs to refer to it by name.

Real devices only.

In your test, say something like:

Save example.com/image.jpg to files

Face ID and Touch ID

The agent can enrol a biometric before your app’s biometric check, and then send either a matching or a non-matching biometric, so the negative path is testable too. It can also remove enrolment, to see what your app does on a device with no biometric set up. This only works on simulators.

In your test, say something like:

Log in with Face ID, then lock the app and fail the Face ID check

The clipboard

The agent can both read the clipboard and write to it. Reading it verifies what a share or copy button actually produced. Writing to it sets up a paste without having to type the value somewhere else first.

In your test, say something like:

Copy the invite link and check the clipboard contains a semaloop.com URL

Locking and unlocking the device

The agent can lock the device, and wake and unlock it again. This covers what your app does when it’s interrupted mid-flow and resumed a moment later, and any protection you put behind the lock screen.

In your test, say something like:

Start a transfer, lock the device, then unlock it and check the transfer screen is still there

Crashing the app

Distinct from a force-quit: the agent can terminate your app abruptly, the way stopping it from Xcode does, without the app getting a chance to save state. Use it to check that an abnormal exit doesn’t lose or corrupt in-progress work.

In your test, say something like:

Add three items to the basket, then crash the app and reopen it

Reading device state

The agent can read the device’s current orientation, whether the screen is off, which app is in the foreground and the device’s date and time. This is mostly useful for checks that depend on state your UI might not display. Note that while the agent can read the orientation, it doesn’t set it.