Agenteract

🚀 Getting Started

Agenteract is an experimental bridge that lets coding agents view and interact with running applications - Including React Native / Expo, React, Kotlin Multi Platform, and Swift UI. It's possible to add support for your favorite stack by replicating the existing framework implementations.

Agenteract exposes the app's internal state, view hierarchy, and actionable controls over a secure WebSocket, enabling agents (or test harnesses) to observe and trigger UI events just like a developer or user would.

1. Installation

Agent Based

The easiest way to get started is to let your coding agent do the work. First, create an AGENTS.md file:

npx @agenteract/agents md

You can also specify the name:

npx @agenteract/agents md CLAUDE.md

This appends the Agenteract instructions to your agents markdown.

Next, restart your coding CLI or open a new chat tab (Cursor), then ask it to complete the setup:

"Please add Agenteract support and make sure it works."

Manual

First, you'll need to install the Agenteract CLI. This tool manages the communication between the AI agent and your local development servers.

npm install -g @agenteract/cli

Next, install the appropriate package for your project.

For React Native (Expo):

npm install @react-native-async-storage/async-storage expo-linking

For React (Vite):

npm install @agenteract/react

React Native, Expo, React all use the same import @agenteract/react. Previously Expo and Vite specific packages were used to provide dev server wrappers, but this is now generalized.

For Flutter:

Add to your pubspec.yaml:

dependencies:
  agenteract:
    git:
      url: https://github.com/agenteract/agenteract.git
      path: packages/flutter

2. AGENTS.md

Next, install AGENTS.md - This will allow your coding assistant to understand how Agenteract works.

If you already have an AGENTS.md, our contents will be appended.

You can also specify the output filename:

npx @agenteract/agents md [dest] # You can specify the name, eg AGENTERACT.md

It is advisable to filter the documentation specific to your app. Agents are great at this!

Please filter AGENTERACT.md - We want to remove anything that isn't relevant to this <project type> app.

You can then turn AGENTERACT.md into skills:

Claude Code:

Please convert AGENTERACT.md into skills as per https://code.claude.com/docs/en/skills
Place the skills folders within this app repo.

OpenCode:

Please convert AGENTERACT.md into skills as per https://code.claude.com/docs/en/skills
Place the skills folders within this app repo.

Now you can reference the file for your agent in a message, or restart the CLI for it to take effect.

At this point you can ask your agent to start setting up Agenteract.

3. Configuration

The command below will create an initial agenteract.config.js, or add entries to an existing configuration.

New Format (Generic Dev Server - Recommended):

npx @agenteract/cli add-config <path> <projectName> <command> [port] --scheme myapp

port is auto assigned if not provided. This is the port that Agenteract uses to communicate internally, it's not where the dev server hosts files.

--scheme myapp: Scheme used for QR code / deep link pairing.

Parameters:

path: Path to the project directory
projectName: Project name as supplied to AgentDebugBridge
command: Dev server command (e.g., "npm run dev", "remix dev")
port: PTY bridge port (auto-assigned if omitted)

Examples:

# Expo Go - Supports exp:// scheme by default, the CLI will run from CWD
npx @agenteract/cli add-config . expo-app "npx expo" --scheme exp

# The following examples specify relative paths as would be the case in a mono repo
# Next.js app with explicit port
npx @agenteract/cli add-config ./apps/web next-app "npm run dev" --port 8791

# Remix app with auto-assigned port
npx @agenteract/cli add-config ./apps/remix remix-app "remix dev"

# Custom dev server
npx @agenteract/cli add-config ./apps/custom my-app "pnpm start:dev"

Note: The new generic format supports any dev server command, making Agenteract framework-agnostic.

Here is an example configuration for a monorepo containing multiple projects:

// agenteract.config.js
export default {
  /**
   * The port for the central Agenteract server.
   * The agent connects to this port.
   */
  port: 8766,

  /**
   * An array of projects to manage.
   */
  projects: [
    {
      // A unique identifier for this app. Used for targeting commands.
      name: 'expo-app',
      // The path to the app's root directory, relative to this config file.
      path: './examples/expo-example',
      // Generic dev server configuration
      devServer: {
        command: 'npx expo start',
        port: 8790,
      },
      "scheme": "myapp"
    },
    {
      name: 'react-app',
      path: './examples/react-example',
      devServer: {
        command: 'npx vite',
        port: 8791,
      }
    },
    {
      name: 'flutter-app',
      path: './examples/flutter_example',
      devServer: {
        command: 'flutter run',
        port: 8792,
        scheme: 'my-flutter-app',
        validation: {
          fileExists: ['pubspec.yaml'],
          commandInPath: 'flutter',
        }
      }
    },
    {
      name: 'next-app',
      path: './apps/web',
      devServer: {
        command: 'npm run dev',
        port: 8793,
      }
    },
    {
      name: 'swift-app',
      path: './examples/swift-app',
      type: 'native'  // Native apps don't have dev servers
    }
  ],
};

Configuration Options

port: The main port for the Agenteract server.
projects: An array of project objects.
- name: A unique name for your app (used in agent commands).
- path: The relative path to your app's root directory.
- devServer: Dev server configuration (optional for native apps).
  - command: The shell command to start the dev server (e.g., 'npm run dev', 'flutter run').
  - port: A unique port for the PTY (pseudo-terminal) bridge.
  - cwd: (Optional) Override working directory.
  - env: (Optional) Additional environment variables.
  - validation: (Optional) Pre-flight checks.
    fileExists: Files that must exist (e.g., ['package.json']).
    commandInPath: Commands that must be in PATH (e.g., 'node', 'flutter').
    errorHints: Custom error messages for common issues.
- scheme: (Optional) Scheme used for QR code / deep link pairing.
- type: (Deprecated) Legacy type field. Use devServer instead.

4. Instrumenting Your Application

To allow Agenteract to 'see' and interact with your application, you need to add the AgentDebugBridge component to your app's entry point.

Platform-Specific Setup

For physical device testing (React Native, Expo, Swift, Kotlin), you'll also need to configure deep linking to enable secure pairing. See platform-specific instructions below.

For React Native / Expo - App.tsx:

React Native, Expo, React all use the same import @agenteract/react.

Previously Expo and Vite specific packages were used to provide dev server wrappers, but this is now generalized.

import { View, Text } from 'react-native';
import { AgentDebugBridge } from '@agenteract/react';

export default function App() {
  return (
    <View style={{ flex: 1 }}>
      {/* Your existing application */}
      <Text>Welcome to my app!</Text>

      {/* Add the AgentDebugBridge */}
      { __DEV__ && <AgentDebugBridge projectName="expo-app" /> }
    </View>
  );
}

For React (Vite) - src/App.tsx:

import { AgentDebugBridge } from '@agenteract/react';

function App() {
  return (
    <>
      {/* Your existing application */}
      <h1>Welcome to my app!</h1>

      {/* Add the AgentBridge */}
      { __DEV__ && <AgentDebugBridge projectName="vite-app" /> }
    </>
  );
}

export default App;

For Flutter lib/main.dart:

Packages:

agenteract (Git or local path - not yet on pub.dev)

Installation:

dependencies:
  agenteract:
    git:
      url: https://github.com/agenteract/agenteract.git
      path: packages/flutter

The following can be installed either in the app, or at the monorepo root if applicable:

@agenteract/cli, @agenteract/server, @agenteract/agents

Usage:

import 'package:agenteract/agenteract.dart';
import 'package:flutter/foundation.dart';
// ...
if (kDebugMode) {
  return AgentDebugBridge(
    projectName: 'myFlutterApp',
    child: MyApp(),
  );
}

Making widgets interactive:

// Use the .withAgent() extension on any widget
ElevatedButton(
  onPressed: () => print('clicked'),
  child: Text('Click me'),
).withAgent('submit-button', onTap: () => print('clicked'))

// Text input
TextField(
  onChanged: (text) => print(text),
).withAgent('username-input', onChangeText: (text) => print(text))

For Swift UI

See agenteract-swift

For Kotlin Multiplatform (Compose Multiplatform)

Consult packages/kotlin/README.md for installation and usage instructions.

Usage:

import io.agenteract.AgentDebugBridge
// ...
AgentDebugBridge(projectName = "kmp-app")

Making composables interactive:

import io.agenteract.agent

// Button with tap handler
Button(
    onClick = { handleClick() },
    modifier = Modifier.agent(
        testID = "submit-button",
        onTap = { handleClick() }
    )
) {
    Text("Submit")
}

// Text input
var text by remember { mutableStateOf("") }
TextField(
    value = text,
    onValueChange = { text = it },
    modifier = Modifier.agent(
        testID = "username-input",
        onChangeText = { text = it }
    )
)

5. Running Agenteract

With your configuration in place and your app instrumented, you can now start Agenteract.

Open a terminal and run the following command from the root of your project (where your agenteract.config.js is located):

npx @agenteract/cli dev

This command will:

Start the central Agenteract server on the configured port
Start a PTY bridge for each project on its configured port
Automatically start the development server for each of your configured projects (e.g., npm run dev or npx expo start).

Connecting Devices

For Simulators/Emulators and Web Apps:

Applications automatically connect to localhost:8765 - no additional setup needed!

For Physical Devices (React Native, Expo, Swift, Kotlin):

Physical devices require deep link pairing for secure connections:

1. Configure your app's URL scheme (if not already done):

npx @agenteract/cli add-config . my-app native --scheme myapp

# For Expo Go, use scheme exp:
npx @agenteract/cli add-config . my-app expo --scheme exp

2. Start the dev server (if not already running):

npx @agenteract/cli dev

3. Connect your physical device:

npx @agenteract/cli connect

4. Scan the QR code displayed in the terminal with your device camera

The deep link will configure your app with the server's IP address, port, and authentication token. This configuration is saved permanently and used for all future connections.

Device information is stored in .agenteract-runtime.json. This file should not be checked in to SCM.

Platform-Specific Deep Linking Setup:

React/Expo: See packages/react/README.md
Flutter: See packages/flutter/README.md
Swift/iOS: See agenteract-swift README
Kotlin/Android: See packages/kotlin/README.md

Agent Interaction

AI agents can now connect to the Agenteract server using the tools described in AGENTS.md. The agent can view your app's component hierarchy and perform actions like tapping buttons or typing into text fields.

agenteract.config.js should not be tracked as it may be specific to your local environment.

Your agent is now ready to Agenteract!

Programmatic Testing with AgentClient

For Node.js integration tests and automation scripts, Agenteract provides the AgentClient API - a programmatic alternative to CLI commands:

import { AgentClient } from '@agenteract/core/node';

const client = new AgentClient('ws://localhost:8765');
await client.connect();

// Interaction primitives
await client.tap('expo-app', 'login-button');
await client.input('expo-app', 'username', '[email protected]');
const hierarchy = await client.getViewHierarchy('expo-app');

// Utilities with real-time streaming
await client.waitForLog('expo-app', 'Login successful', 5000);
await client.waitForElement('expo-app', 'dashboard', 5000);

// Cleanup
client.disconnect();

Benefits of AgentClient:

Faster execution: WebSocket connection vs subprocess overhead
Real-time log streaming: Stream logs as they happen
Type-safe: Full TypeScript support with IDE autocomplete
Better error handling: Promise-based with structured errors
Native async/await: Clean, readable test code

When to use each approach:

CLI commands (npx @agenteract/agents tap ...): For AI agents, manual testing, CI scripts
AgentClient (TypeScript/Node.js): For integration tests, test frameworks (Jest/Mocha), automation

Complete example: See tests/e2e/node-client/test-agent-client.ts for a full testing workflow.

All methods available:

tap(project, testID) - Tap a component
input(project, testID, value) - Input text into a field
scroll(project, testID, direction, amount?) - Scroll a view
swipe(project, testID, direction, velocity?) - Swipe gesture
longPress(project, testID) - Long press a component
getViewHierarchy(project) - Get UI hierarchy
agentLink(project, url) - Send deep link action
getLogs(project, count?) - Get console logs
waitForLog(project, pattern, timeout?) - Wait for log message
waitForElement(project, testID, timeout?) - Wait for element
waitForCondition(project, predicate, timeout?) - Wait for custom condition

Agent Links (agentLink)

Agent Links provide a way to trigger app-specific actions through deep link-style URLs sent over the WebSocket connection. Unlike pairing deep links (which configure the server connection), agentLinks are sent to already-connected apps to trigger custom behaviors.

Use Cases:

Reset app state during automated testing
Navigate to specific screens
Reload or restart the app
Trigger custom app actions

Command:

npx @agenteract/cli agent-link <project> <url>

Examples:

# Reset application state
npx @agenteract/cli agent-link expo-app agenteract://reset_state

# Navigate to a specific screen
npx @agenteract/cli agent-link expo-app agenteract://navigate?screen=settings

# Reload the application
npx @agenteract/cli agent-link expo-app agenteract://reload

To handle agentLinks in your application, provide an onAgentLink handler to AgentDebugBridge:

React/Expo:

import { AgentDebugBridge } from '@agenteract/react';

const handleAgentLink = async (url: string): Promise<boolean> => {
  const { hostname, queryParams } = parseURL(url);
  
  switch (hostname) {
    case 'reset_state':
      // Reset your app state
      resetAppState();
      return true; // Handled by app
    case 'navigate':
      // Navigate to screen from query params
      navigation.navigate(queryParams.screen);
      return true;
    default:
      return false; // Let AgentDebugBridge handle config links
  }
};

<AgentDebugBridge 
  projectName="expo-app"
  onAgentLink={handleAgentLink}
/>

Flutter:

import 'package:agenteract/agenteract.dart';

Future<bool> handleAgentLink(String url) async {
  final uri = Uri.parse(url);
  
  switch (uri.host) {
    case 'reset_state':
      // Reset your app state
      resetAppState();
      return true;
    case 'navigate':
      // Navigate to screen
      Navigator.pushNamed(context, uri.queryParameters['screen'] ?? '/');
      return true;
    default:
      return false;
  }
}

AgentDebugBridge(
  projectName: 'flutter-app',
  onAgentLink: handleAgentLink,
  child: MyApp(),
)

Kotlin (Compose Multiplatform):

import io.agenteract.AgentDebugBridge

val handleAgentLink: suspend (String) -> Boolean = { url ->
    val uri = URI(url)
    when (uri.host) {
        "reset_state" -> {
            resetAppState()
            true
        }
        "navigate" -> {
            val screen = uri.query?.split("=")?.get(1)
            navController.navigate(screen ?: "home")
            true
        }
        else -> false
    }
}

AgentDebugBridge(
    projectName = "kmp-app",
    onAgentLink = handleAgentLink
)

Return Values:

Return true if your handler processes the agentLink
Return false to let AgentDebugBridge handle it (for pairing/config links)

URL Format:

Scheme: agenteract://
Hostname: Action identifier (e.g., reset_state, navigate, reload)
Query parameters: Optional parameters (e.g., ?screen=settings&tab=profile)

Helper Utilities:

The example apps include URL parsing utilities: /examples/expo-example/app/utils/deepLinkUtils.ts and /examples/react-example/src/utils/deepLinkUtils.ts. These provide a parseURL() function to extract the hostname and query parameters from agentLink URLs.

🔒 Security & Scope

Designed for local development, testing, and accessibility research.
Future versions will include authentication, session control.

⛏️ Development

This guide covers how to set up the local development environment to run the Expo example app and the agent server.

1. Prerequisites

First, clone the repository and ensure you have pnpm installed.

git clone https://github.com/agenteract/agenteract.git
cd agenteract
git submodule update --init --recursive
npm install -g pnpm

2. Install Dependencies & Build

Install all dependencies for the monorepo and build the packages from the root directory.

npm install -g pnpm
pnpm install
pnpm build

We need to use linking to work locally:

Usage:

cd packages/server
pnpm link --global
cd packages/agents
pnpm link --global
cd packages/cli
pnpm link --global
cd packages/pty
pnpm link --global

3. Run Development Environment

Start the Agenteract development environment using the unified CLI:

pnpm agenteract dev

# once published, you can use:
npx @agenteract/cli dev

This will:

Start the central Agenteract server
Start PTY bridges for each configured project
Automatically launch all development servers defined in your agenteract.config.js

The multiplexed output will show logs from all your running applications in a single terminal.

4. Observe and Interact

Once the app is running, it will automatically connect to the agent server. You should see connection logs in the multiplexed output.

You can manually simulate an agent command to test the connection:

curl -s -X POST http://localhost:8766/expo-app -d '{"action":"getViewHierarchy"}'

The server will forward this to the app, which responds with a JSON payload of its view hierarchy.

5. Agent Interaction

This step creates or appends to your AGENTS.md file. This informs coding agents how to interact with the app.

npx @agenteract/agents md [dest] # You can specify the name, eg AGENTERACT.md

If you are using a separate agent to your IDE, start it now, otherwise you can use the built in agent (Tested with Cursor, Gemini CLI)

Issue some instructions. You might need to prime the agent the first time

You can use the Get View Hieararchy tool to inspect the current app state.
Add a button that disappears when it is clicked.
Confirm that it works using a simulated action.

Agents should view the current hierarchy, modify the code, view again, simulate a tap, then confirm that the button disappeared by viewing the hierarchy one final time.

Because packages/agents/AGENTS.md contains instructions about how to interact with the app, you don't need to explicitly tell it to use the AgentDebugBridge.

✅ Verification Checklist

All packages build successfully with pnpm build.
The Agenteract server starts and listens on the configured port (default 8766).
All configured apps start and connect to the server.
The multiplexed output shows connection messages from your apps.
Sending a getViewHierarchy command via curl to your app returns a JSON tree.

🧪 Testing

This project uses Jest for testing.

Run All Tests

To run the tests for all packages, use the following command from the root directory:

pnpm test

Run Tests for a Single Package

To run the tests for a specific package, use the --filter flag:

pnpm --filter @agenteract/react test

Continuous Integration

Tests are run automatically on every push and pull request to the main branch using GitHub Actions.

Integration Testing

Integration tests verify that packages can be installed and used correctly after publication. They use Verdaccio , a lightweight private npm registry running in Docker.

Local testing workflow:

# Start local npm registry
pnpm verdaccio:start

# Build and publish packages
pnpm verdaccio:publish

# Run integration tests
pnpm test:integration

# Clean up
pnpm verdaccio:stop

GitHub Actions: Integration tests run automatically on PRs and pushes to main and release/** branches using Verdaccio as a service container.

Authentication: Uses expect to automate the authentication process. See docs/VERDACCIO_AUTH.md for details.

See docs/INTEGRATION_TESTING.md for complete information.

📱 App Lifecycle Management

Agenteract provides CLI commands to manage app launching, stopping, building, and setup operations across all supported platforms (React/Vite, Expo, Flutter, Swift, Kotlin).

Launch an App

Launch your app on a device or simulator:

npx @agenteract/agents start-app <project> [options]

Options:

--device <id> - Target specific device/simulator ID
--platform <type> - Override platform detection (vite, expo, flutter, xcode, kmp-android, kmp-desktop)
--headless - Launch browser in headless mode (web apps only)
--prebuild - Use an Expo prebuild (native build) instead of Expo Go (Expo projects only)
--launch-only - Skip build and install steps; launch the already-installed app directly

Examples:

# Launch with auto-detected platform and default device
npx @agenteract/agents start-app expo-app

# Launch on specific iOS simulator
npx @agenteract/agents start-app expo-app --device "iPhone 15 Pro"

# Launch Flutter app on Android with explicit platform
npx @agenteract/agents start-app flutter-app --platform flutter --device emulator-5554

# Launch web app in headless mode
npx @agenteract/agents start-app vite-app --headless

# Launch Expo prebuild (native build) instead of Expo Go
npx @agenteract/agents start-app expo-app --prebuild --device "iPhone 15 Pro"

# Launch without rebuilding or reinstalling (app must already be installed)
npx @agenteract/agents start-app expo-app --launch-only --device "iPhone 15 Pro"

Stop an App

Stop a running application:

npx @agenteract/agents stop-app <project> [options]

Options:

--device <id> - Target specific device ID
--force - Force stop (Android: force-stop instead of stop, Desktop: SIGKILL instead of SIGTERM)
--prebuild - Target an Expo prebuild app instead of Expo Go (Expo projects only)

Examples:

# Stop app gracefully
npx @agenteract/agents stop-app expo-app

# Force stop on specific device
npx @agenteract/agents stop-app expo-app --device emulator-5554 --force

# Stop Expo prebuild app
npx @agenteract/agents stop-app expo-app --prebuild

Build an App

Build your application for a target platform:

npx @agenteract/agents build <project> [options]

Options:

--platform <type> - Target platform (vite, expo, flutter, xcode, kmp-android, kmp-desktop)
--config <type> - Build configuration: debug (default) or release

Examples:

# Build debug version with auto-detected platform
npx @agenteract/agents build flutter-app

# Build release version for Android
npx @agenteract/agents build flutter-app --platform flutter --config release

# Build Swift iOS app
npx @agenteract/agents build swift-app --platform xcode --config debug

Setup Operations

Perform setup operations like install, reinstall, or clear app data:

npx @agenteract/agents setup <project> <action> [options]

Actions:

install - Install the app on device
reinstall - Uninstall and reinstall the app
clearData - Clear app data/cache (Android) or uninstall (iOS)

Options:

--device <id> - Target specific device ID
--platform <type> - Override platform detection

Examples:

# Install app on default device
npx @agenteract/agents setup expo-app install

# Reinstall on specific device
npx @agenteract/agents setup flutter-app reinstall --device emulator-5554

# Clear app data (Android) or uninstall (iOS)
npx @agenteract/agents setup expo-app clearData --platform expo

Lifecycle Configuration

Add optional lifecycle configuration to your agenteract.config.js:

export default {
  projects: [
    {
      name: 'expo-app',
      path: './examples/expo-app',
      devServer: { command: 'npx expo start', port: 8790 },
      lifecycle: {
        bundleId: {
          ios: 'com.example.expoapp',
          android: 'com.example.expoapp'
        },
        mainActivity: 'com.example.expoapp.MainActivity', // Android only
        launchTimeout: 30000, // ms, default: 30000
        requiresInstall: false // default: false
      }
    }
  ]
};

Configuration Options:

bundleId - App bundle identifier (auto-detected from app.json, Info.plist, build.gradle if not specified)
mainActivity - Android main activity class (auto-detected from AndroidManifest.xml if not specified)
launchTimeout - Maximum time to wait for app launch in milliseconds
requiresInstall - Whether setup operations should install before launching

Device Management

Agenteract automatically detects and manages devices.

Default Device Selection:

Explicit --device flag takes highest priority
Falls back to default device from .agenteract-runtime.json
For iOS: Uses booted simulator or first available
For Android: Uses first device from adb devices

Set Default Device:

# Device info is automatically saved when you use --device flag
npx @agenteract/agents start-app expo-app --device "iPhone 15 Pro"

# Future launches will use this device by default
npx @agenteract/agents start-app expo-app

Default device configuration is stored in .agenteract-runtime.json (should not be committed to SCM).

Platform Detection

Agenteract auto-detects your platform by scanning for marker files:

Vite: vite.config.ts, vite.config.js
Expo: app.json with expo key
Flutter: pubspec.yaml
Xcode (Swift/Objective-C): Package.swift, .xcodeproj
KMP Android: build.gradle.kts with Kotlin/Android
KMP Desktop: build.gradle.kts with Compose Desktop

You can override detection with the --platform flag.