# @ui-tars/sdk Guide (Experimental)
[](https://www.npmjs.com/package/@ui-tars/sdk) [](https://app.codecov.io/gh/bytedance/UI-TARS-desktop/components/ui_tars_sdk)
## Overview
`@ui-tars/sdk` is a powerful cross-platform(ANY device/platform) toolkit for building GUI automation agents.
It provides a flexible framework to create agents that can interact with graphical user interfaces through various operators. It supports running on both **Node.js** and the **Web Browser**
```mermaid
classDiagram
class GUIAgent~T extends Operator~ {
+model: UITarsModel
+operator: T
+signal: AbortSignal
+onData
+run()
}
class UITarsModel {
+invoke()
}
class Operator {
<<interface>>
+screenshot()
+execute()
}
class NutJSOperator {
+screenshot()
+execute()
}
class WebOperator {
+screenshot()
+execute()
}
class MobileOperator {
+screenshot()
+execute()
}
GUIAgent --> UITarsModel
GUIAgent ..> Operator
Operator <|.. NutJSOperator
Operator <|.. WebOperator
Operator <|.. MobileOperator
```
## Try it out
```bash
npx @ui-tars/cli start
```
Input your UI-TARS Model Service Config(`baseURL`, `apiKey`, `model`), then you can control your computer with CLI.
```
Need to install the following packages:
Ok to proceed? (y) y
│
◆ Input your instruction
│ _ Open Chrome
└
```
## Agent Execution Process
```mermaid
sequenceDiagram
participant user as User
participant guiAgent as GUI Agent
participant model as UI-TARS Model
participant operator as Operator
user -->> guiAgent: "`instruction` + <br /> `Operator.MANUAL.ACTION_SPACES`"
activate user
activate guiAgent
loop status !== StatusEnum.RUNNING
guiAgent ->> operator: screenshot()
activate operator
operator -->> guiAgent: base64, Physical screen size
deactivate operator
guiAgent ->> model: instruction + actionSpaces + screenshots.slice(-5)
model -->> guiAgent: `prediction`: click(start_box='(27,496)')
guiAgent -->> user: prediction, next action
guiAgent ->> operator: execute(prediction)
activate operator
operator -->> guiAgent: success
deactivate operator
end
deactivate guiAgent
deactivate user
```
### Basic Usage
Basic usage is largely derived from package `@ui-tars/sdk`, here's a basic example of using the SDK:
> Note: Using `nut-js`(cross-platform computer control tool) as the operator, you can also use or customize other operators. NutJS operator that supports common desktop automation actions:
> - Mouse actions: click, double click, right click, drag, hover
> - Keyboard input: typing, hotkeys
> - Scrolling
> - Screenshot capture
```ts
import { GUIAgent } from '@ui-tars/sdk';
import { NutJSOperator } from '@ui-tars/operator-nut-js';
const guiAgent = new GUIAgent({
model: {
baseURL: config.baseURL,
apiKey: config.apiKey,
model: config.model,
},
operator: new NutJSOperator(),
onData: ({ data }) => {
console.log(data)
},
onError: ({ data, error }) => {
console.error(error, data);
},
});
await guiAgent.run('send "hello world" to x.com');
```
### Handling Abort Signals
You can abort the agent by passing a `AbortSignal` to the GUIAgent `signal` option.
```ts
const abortController = new AbortController();
const guiAgent = new GUIAgent({
// ... other config
signal: abortController.signal,
});
// ctrl/cmd + c to cancel operation
process.on('SIGINT', () => {
abortController.abort();
});
```
## Configuration Options
The `GUIAgent` constructor accepts the following configuration options:
- `model`: Model configuration(OpenAI-compatible API) or custom model instance
- `baseURL`: API endpoint URL
- `apiKey`: API authentication key
- `model`: Model name to use
- more options see [OpenAI API](https://platform.openai.com/docs/guides/vision/uploading-base-64-encoded-images)
- `operator`: Instance of an operator class that implements the required interface
- `signal`: AbortController signal for canceling operations
- `onData`: Callback for receiving agent data/status updates
- `data.conversations` is an array of objects, **IMPORTANT: is delta, not the whole conversation history**, each object contains:
- `from`: The role of the message, it can be one of the following:
- `human`: Human message
- `gpt`: Agent response
- `screenshotBase64`: Screenshot base64
- `value`: The content of the message
- `data.status` is the current status of the agent, it can be one of the following:
- `StatusEnum.INIT`: Initial state
- `StatusEnum.RUNNING`: Agent is actively executing
- `StatusEnum.END`: Operation completed
- `StatusEnum.MAX_LOOP`: Maximum loop count reached
- `onError`: Callback for error handling
- `systemPrompt`: Optional custom system prompt
- `maxLoopCount`: Maximum number of interaction loops (default: 25)
### Status flow
```mermaid
stateDiagram-v2
[*] --> INIT
INIT --> RUNNING
RUNNING --> RUNNING: Execute Actions
RUNNING --> END: Task Complete
RUNNING --> MAX_LOOP: Loop Limit Reached
END --> [*]
MAX_LOOP --> [*]
```
## Advanced Usage
### Operator Interface
When implementing a custom operator, you need to implement two core methods: `screenshot()` and `execute()`.
#### Initialize
`npm init` to create a new operator package, configuration is as follows:
```json
{
"name": "your-operator-tool",
"version": "1.0.0",
"main": "./dist/index.js",
"module": "./dist/index.mjs",
"types": "./dist/index.d.ts",
"scripts": {
"dev": "rslib build --watch",
"prepare": "npm run build",
"build": "rsbuild",
"test": "vitest"
},
"files": [
"dist"
],
"publishConfig": {
"access": "public",
"registry": "https://registry.npmjs.org"
},
"dependencies": {
"jimp": "^1.6.0"
},
"peerDependencies": {
"@ui-tars/sdk": "^1.2.0-beta.17"
},
"devDependencies": {
"@ui-tars/sdk": "^1.2.0-beta.17",
"@rslib/core": "^0.5.4",
"typescript": "^5.7.2",
"vitest": "^3.0.2"
}
}
```
#### screenshot()
This method captures the current screen state and returns a `ScreenshotOutput`:
```typescript
interface ScreenshotOutput {
// Base64 encoded image string
base64: string;
// Device pixel ratio (DPR)
scaleFactor: number;
}
```
#### execute()
This method performs actions based on model predictions. It receives an `ExecuteParams` object:
```typescript
interface ExecuteParams {
/** Raw prediction string from the model */
prediction: string;
/** Parsed prediction object */
parsedPrediction: {
action_type: string;
action_inputs: Record<string, any>;
reflection: string | null;
thought: string;
};
/** Device Physical Resolution */
screenWidth: number;
/** Device Physical Resolution */
screenHeight: number;
/** Device DPR */
scaleFactor: number;
/** model coordinates scaling factor [widthFactor, heightFactor] */
factors: Factors;
}
```
Advanced sdk usage is largely derived from package `@ui-tars/sdk/core`, you can create custom operators by extending the base `Operator` class:
```typescript
import {
Operator,
type ScreenshotOutput,
type ExecuteParams
type ExecuteOutput,
} from '@ui-tars/sdk/core';
import { Jimp } from 'jimp';
export class CustomOperator extends Operator {
// Define the action spaces and description for UI-TARS System Prompt splice
static MANUAL = {
ACTION_SPACES: [
'click(start_box="") # click on the element at the specified coordinates',
'type(content="") # type the specified content into the current input field',
'scroll(direction="") # scroll the page in the specified direction',
'finished() # finish the task',
// ...more_actions
],
};
public async screenshot(): Promise<ScreenshotOutput> {
// Implement screenshot functionality
const base64 = 'base64-encoded-image';
const buffer = Buffer.from(base64, 'base64');
const image = await sharp(buffer).toBuffer();
return {
base64: 'base64-encoded-image',
scaleFactor: 1
};
}
async execute(params: ExecuteParams): Promise<ExecuteOutput> {
const { parsedPrediction, screenWidth, screenHeight, scaleFactor } = params;
// Implement action execution logic
// if click action, get coordinates from parsedPrediction
const [startX, startY] = parsedPrediction?.action_inputs?.start_coords || '';
if (parsedPrediction?.action_type === 'finished') {
// finish the GUIAgent task
return { status: StatusEnum.END };
}
}
}
```
Required methods:
- `screenshot()`: Captures the current screen state
- `execute()`: Performs the requested action based on model predictions
Optional static properties:
- `MANUAL`: Define the action spaces and description for UI-TARS Model understanding
- `ACTION_SPACES`: Define the action spaces and description for UI-TARS Model understanding
Loaded into `GUIAgent`:
```ts
const guiAgent = new GUIAgent({
// ... other config
systemPrompt: `
// ... other system prompt
${CustomOperator.MANUAL.ACTION_SPACES.join('\n')}
`,
operator: new CustomOperator(),
});
```
### Custom Model Implementation
You can implement custom model logic by extending the `UITarsModel` class:
```typescript
class CustomUITarsModel extends UITarsModel {
constructor(modelConfig: { model: string }) {
super(modelConfig);
}
async invoke(params: any) {
// Implement custom model logic
return {
prediction: 'action description',
parsedPredictions: [{
action_type: 'click',
action_inputs: { /* ... */ },
reflection: null,
thought: 'reasoning'
}]
};
}
}
const agent = new GUIAgent({
model: new CustomUITarsModel({ model: 'custom-model' }),
// ... other config
});
```
> Note: However, it is not recommended to implement a custom model because it contains a lot of data processing logic (including image transformations, scaling factors, etc.).
### Planning
You can combine planning/reasoning models (such as OpenAI-o1, DeepSeek-R1) to implement complex GUIAgent logic for planning, reasoning, and execution:
```ts
const guiAgent = new GUIAgent({
// ... other config
});
const planningList = await reasoningModel.invoke({
conversations: [
{
role: 'user',
content: 'buy a ticket from beijing to shanghai',
}
]
})
/**
* [
* 'open chrome',
* 'open trip.com',
* 'click "search" button',
* 'select "beijing" in "from" input',
* 'select "shanghai" in "to" input',
* 'click "search" button',
* ]
*/
for (const planning of planningList) {
await guiAgent.run(planning);
}
```
UI-TARS-desktop - docs archive 1.0 sdk
10723 characters
> [!WARNING]
> This document has been archived.
# @ui-tars/sdk Guide (Experimental)
## Overview
`@ui-tars/sdk` is a powerful cross-platform(ANY device/platform) toolkit for building GUI automation agents.
It provides a flexible framework to create agents that can interact with graphical user interfaces through various operators. It supports running on both **Node.js** and the **Web Browser**
```mermaid
classDiagram
class GUIAgent~T extends Operator~ {
+model: UITarsModel
+operator: T
+signal: AbortSignal
+onData
+run()
}
class UITarsModel {
+invoke()
}
class Operator {
<<interface>>
+screenshot()
+execute()
}
class NutJSOperator {
+screenshot()
+execute()
}
class WebOperator {
+screenshot()
+execute()
}
class MobileOperator {
+screenshot()
+execute()
}
GUIAgent --> UITarsModel
GUIAgent ..> Operator
Operator <|.. NutJSOperator
Operator <|.. WebOperator
Operator <|.. MobileOperator
```
## Try it out
```bash
npx @ui-tars/cli start
```
Input your UI-TARS Model Service Config(`baseURL`, `apiKey`, `model`), then you can control your computer with CLI.
```
Need to install the following packages:
Ok to proceed? (y) y
│
◆ Input your instruction
│ _ Open Chrome
└
```
## Agent Execution Process
```mermaid
sequenceDiagram
participant user as User
participant guiAgent as GUI Agent
participant model as UI-TARS Model
participant operator as Operator
user -->> guiAgent: "`instruction` + <br /> `Operator.MANUAL.ACTION_SPACES`"
activate user
activate guiAgent
loop status !== StatusEnum.RUNNING
guiAgent ->> operator: screenshot()
activate operator
operator -->> guiAgent: base64, Physical screen size
deactivate operator
guiAgent ->> model: instruction + actionSpaces + screenshots.slice(-5)
model -->> guiAgent: `prediction`: click(start_box='(27,496)')
guiAgent -->> user: prediction, next action
guiAgent ->> operator: execute(prediction)
activate operator
operator -->> guiAgent: success
deactivate operator
end
deactivate guiAgent
deactivate user
```
### Basic Usage
Basic usage is largely derived from package `@ui-tars/sdk`, here's a basic example of using the SDK:
> Note: Using `nut-js`(cross-platform computer control tool) as the operator, you can also use or customize other operators. NutJS operator that supports common desktop automation actions:
> - Mouse actions: click, double click, right click, drag, hover
> - Keyboard input: typing, hotkeys
> - Scrolling
> - Screenshot capture
```ts
import { GUIAgent } from '@ui-tars/sdk';
import { NutJSOperator } from '@ui-tars/operator-nut-js';
const guiAgent = new GUIAgent({
model: {
baseURL: config.baseURL,
apiKey: config.apiKey,
model: config.model,
},
operator: new NutJSOperator(),
onData: ({ data }) => {
console.log(data)
},
onError: ({ data, error }) => {
console.error(error, data);
},
});
await guiAgent.run('send "hello world" to x.com');
```
### Handling Abort Signals
You can abort the agent by passing a `AbortSignal` to the GUIAgent `signal` option.
```ts
const abortController = new AbortController();
const guiAgent = new GUIAgent({
// ... other config
signal: abortController.signal,
});
// ctrl/cmd + c to cancel operation
process.on('SIGINT', () => {
abortController.abort();
});
```
## Configuration Options
The `GUIAgent` constructor accepts the following configuration options:
- `model`: Model configuration(OpenAI-compatible API) or custom model instance
- `baseURL`: API endpoint URL
- `apiKey`: API authentication key
- `model`: Model name to use
- more options see [OpenAI API](https://platform.openai.com/docs/guides/vision/uploading-base-64-encoded-images)
- `operator`: Instance of an operator class that implements the required interface
- `signal`: AbortController signal for canceling operations
- `onData`: Callback for receiving agent data/status updates
- `data.conversations` is an array of objects, **IMPORTANT: is delta, not the whole conversation history**, each object contains:
- `from`: The role of the message, it can be one of the following:
- `human`: Human message
- `gpt`: Agent response
- `screenshotBase64`: Screenshot base64
- `value`: The content of the message
- `data.status` is the current status of the agent, it can be one of the following:
- `StatusEnum.INIT`: Initial state
- `StatusEnum.RUNNING`: Agent is actively executing
- `StatusEnum.END`: Operation completed
- `StatusEnum.MAX_LOOP`: Maximum loop count reached
- `onError`: Callback for error handling
- `systemPrompt`: Optional custom system prompt
- `maxLoopCount`: Maximum number of interaction loops (default: 25)
### Status flow
```mermaid
stateDiagram-v2
[*] --> INIT
INIT --> RUNNING
RUNNING --> RUNNING: Execute Actions
RUNNING --> END: Task Complete
RUNNING --> MAX_LOOP: Loop Limit Reached
END --> [*]
MAX_LOOP --> [*]
```
## Advanced Usage
### Operator Interface
When implementing a custom operator, you need to implement two core methods: `screenshot()` and `execute()`.
#### Initialize
`npm init` to create a new operator package, configuration is as follows:
```json
{
"name": "your-operator-tool",
"version": "1.0.0",
"main": "./dist/index.js",
"module": "./dist/index.mjs",
"types": "./dist/index.d.ts",
"scripts": {
"dev": "rslib build --watch",
"prepare": "npm run build",
"build": "rsbuild",
"test": "vitest"
},
"files": [
"dist"
],
"publishConfig": {
"access": "public",
"registry": "https://registry.npmjs.org"
},
"dependencies": {
"jimp": "^1.6.0"
},
"peerDependencies": {
"@ui-tars/sdk": "^1.2.0-beta.17"
},
"devDependencies": {
"@ui-tars/sdk": "^1.2.0-beta.17",
"@rslib/core": "^0.5.4",
"typescript": "^5.7.2",
"vitest": "^3.0.2"
}
}
```
#### screenshot()
This method captures the current screen state and returns a `ScreenshotOutput`:
```typescript
interface ScreenshotOutput {
// Base64 encoded image string
base64: string;
// Device pixel ratio (DPR)
scaleFactor: number;
}
```
#### execute()
This method performs actions based on model predictions. It receives an `ExecuteParams` object:
```typescript
interface ExecuteParams {
/** Raw prediction string from the model */
prediction: string;
/** Parsed prediction object */
parsedPrediction: {
action_type: string;
action_inputs: Record<string, any>;
reflection: string | null;
thought: string;
};
/** Device Physical Resolution */
screenWidth: number;
/** Device Physical Resolution */
screenHeight: number;
/** Device DPR */
scaleFactor: number;
/** model coordinates scaling factor [widthFactor, heightFactor] */
factors: Factors;
}
```
Advanced sdk usage is largely derived from package `@ui-tars/sdk/core`, you can create custom operators by extending the base `Operator` class:
```typescript
import {
Operator,
type ScreenshotOutput,
type ExecuteParams
type ExecuteOutput,
} from '@ui-tars/sdk/core';
import { Jimp } from 'jimp';
export class CustomOperator extends Operator {
// Define the action spaces and description for UI-TARS System Prompt splice
static MANUAL = {
ACTION_SPACES: [
'click(start_box="") # click on the element at the specified coordinates',
'type(content="") # type the specified content into the current input field',
'scroll(direction="") # scroll the page in the specified direction',
'finished() # finish the task',
// ...more_actions
],
};
public async screenshot(): Promise<ScreenshotOutput> {
// Implement screenshot functionality
const base64 = 'base64-encoded-image';
const buffer = Buffer.from(base64, 'base64');
const image = await sharp(buffer).toBuffer();
return {
base64: 'base64-encoded-image',
scaleFactor: 1
};
}
async execute(params: ExecuteParams): Promise<ExecuteOutput> {
const { parsedPrediction, screenWidth, screenHeight, scaleFactor } = params;
// Implement action execution logic
// if click action, get coordinates from parsedPrediction
const [startX, startY] = parsedPrediction?.action_inputs?.start_coords || '';
if (parsedPrediction?.action_type === 'finished') {
// finish the GUIAgent task
return { status: StatusEnum.END };
}
}
}
```
Required methods:
- `screenshot()`: Captures the current screen state
- `execute()`: Performs the requested action based on model predictions
Optional static properties:
- `MANUAL`: Define the action spaces and description for UI-TARS Model understanding
- `ACTION_SPACES`: Define the action spaces and description for UI-TARS Model understanding
Loaded into `GUIAgent`:
```ts
const guiAgent = new GUIAgent({
// ... other config
systemPrompt: `
// ... other system prompt
${CustomOperator.MANUAL.ACTION_SPACES.join('\n')}
`,
operator: new CustomOperator(),
});
```
### Custom Model Implementation
You can implement custom model logic by extending the `UITarsModel` class:
```typescript
class CustomUITarsModel extends UITarsModel {
constructor(modelConfig: { model: string }) {
super(modelConfig);
}
async invoke(params: any) {
// Implement custom model logic
return {
prediction: 'action description',
parsedPredictions: [{
action_type: 'click',
action_inputs: { /* ... */ },
reflection: null,
thought: 'reasoning'
}]
};
}
}
const agent = new GUIAgent({
model: new CustomUITarsModel({ model: 'custom-model' }),
// ... other config
});
```
> Note: However, it is not recommended to implement a custom model because it contains a lot of data processing logic (including image transformations, scaling factors, etc.).
### Planning
You can combine planning/reasoning models (such as OpenAI-o1, DeepSeek-R1) to implement complex GUIAgent logic for planning, reasoning, and execution:
```ts
const guiAgent = new GUIAgent({
// ... other config
});
const planningList = await reasoningModel.invoke({
conversations: [
{
role: 'user',
content: 'buy a ticket from beijing to shanghai',
}
]
})
/**
* [
* 'open chrome',
* 'open trip.com',
* 'click "search" button',
* 'select "beijing" in "from" input',
* 'select "shanghai" in "to" input',
* 'click "search" button',
* ]
*/
for (const planning of planningList) {
await guiAgent.run(planning);
}
```
# @agent-tars/core
<b>Agent TARS</b> is a general multimodal AI Agent stack, it brings the power of GUI Agent and Vision into your terminal, computer, browser and product. <br>
It primarily ships with a <a href="https://agent-tars.com/guide/basic/cli.html" target="_blank">CLI</a> and <a href="https://agent-tars.com/guide/basic/web-ui.html" target="_blank">Web UI</a> for usage. It aims to provide a workflow that is closer to human-like task completion through cutting-edge multimodal LLMs and seamless integration with various real-world <a href="https://agent-tars.com/guide/basic/mcp.html" target="_blank">MCP</a> tools.
📣 **Just released**: Agent TARS Beta - check out our [announcement blog post](https://agent-tars.com/beta)!
https://github.com/user-attachments/assets/772b0eef-aef7-4ab9-8cb0-9611820539d8
<br>
<table>
<thead>
<tr>
<th width="50%" align="center">Booking Hotel</th>
<th width="50%" align="center">Generate Chart with extra MCP Servers</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center">
<video src="https://github.com/user-attachments/assets/c9489936-afdc-4d12-adda-d4b90d2a869d" width="50%"></video>
</td>
<td align="center">
<video src="https://github.com/user-attachments/assets/a9fd72d0-01bb-4233-aa27-ca95194bbce9" width="50%"></video>
</td>
</tr>
<tr>
<td align="left">
<b>Instruction:</b> <i>I am in Los Angeles from September 1st to September 6th, with a budget of $5,000. Please help me book a Ritz-Carlton hotel closest to the airport on booking.com and compile a transportation guide for me</i>
</td>
<td align="left">
<b>Instruction:</b> <i>Draw me a chart of Hangzhou's weather for one month</i>
</td>
</tr>
</tbody>
</table>
For more use cases, please check out [#842](https://github.com/bytedance/UI-TARS-desktop/issues/842).
## Overview
`@agent-tars/core` is the core implementation of Agent TARS, built on top of the Tarko Agent framework. It provides a comprehensive multimodal AI agent with advanced browser automation, filesystem operations, and intelligent search capabilities.
### Core Features
- 🖱️ **One-Click Out-of-the-box CLI** - Supports both **headful** [Web UI](https://agent-tars.com/guide/basic/web-ui.html) and **headless** [server](https://agent-tars.com/guide/advanced/server.html)) [execution](https://agent-tars.com/guide/basic/cli.html).
- 🌐 **Hybrid Browser Agent** - Control browsers using [GUI Agent](https://agent-tars.com/guide/basic/browser.html#visual-grounding), [DOM](https://agent-tars.com/guide/basic/browser.html#dom), or a hybrid strategy.
- 🔄 **Event Stream** - Protocol-driven Event Stream drives [Context Engineering](https://agent-tars.com/beta#context-engineering) and [Agent UI](https://agent-tars.com/blog/2025-06-25-introducing-agent-tars-beta.html#easy-to-build-applications).
- 🧰 **MCP Integration** - The kernel is built on MCP and also supports mounting [MCP Servers](https://agent-tars.com/guide/basic/mcp.html) to connect to real-world tools.
### Quick Start
```bash
# Luanch with `npx`.
npx @agent-tars/cli@latest
# Install globally, required Node.js >= 22
npm install @agent-tars/cli@latest -g
# Run with your preferred model provider
agent-tars --provider volcengine --model doubao-1-5-thinking-vision-pro-250428 --apiKey your-api-key
agent-tars --provider anthropic --model claude-3-7-sonnet-latest --apiKey your-api-key
```
Visit the comprehensive [Quick Start](https://agent-tars.com/guide/get-started/quick-start.html) guide for detailed setup instructions.
## Quick Start
### Installation
```bash
npm install @agent-tars/core
```
### Running Agent TARS
Agent TARS can be started in multiple ways:
#### Option 1: Using @agent-tars/cli (Recommended)
```bash
# Install globally
npm install -g @agent-tars/cli
# Run Agent TARS
agent-tars
# Or use directly via npx
npx @agent-tars/cli
```
#### Option 2: Using @tarko/agent-cli
```bash
# Install globally
npm install -g @tarko/agent-cli
# Run Agent TARS through tarko CLI
tarko run agent-tars
# Or use directly via npx
npx @tarko/agent-cli run agent-tars
```
#### Option 3: Programmatic Usage
### Basic Usage
```typescript
import { AgentTARS } from '@agent-tars/core';
// Create an agent instance
const agent = new AgentTARS({
model: {
provider: 'openai',
model: 'gpt-4',
apiKey: process.env.OPENAI_API_KEY,
},
workspace: './workspace',
browser: {
headless: false,
control: 'hybrid',
},
});
// Initialize and run
await agent.initialize();
const result = await agent.run('Search for the latest AI research papers');
console.log(result);
```
## Configuration
### AgentTARSOptions
```typescript
interface AgentTARSOptions {
// Model configuration
model?: {
provider: 'openai' | 'anthropic' | 'doubao';
model: string;
apiKey: string;
};
// Browser settings
browser?: {
headless?: boolean;
control?: 'dom' | 'visual-grounding' | 'hybrid';
cdpEndpoint?: string;
};
// Search configuration
search?: {
provider: 'browser_search' | 'tavily';
count?: number;
apiKey?: string;
};
// Workspace settings
workspace?: string;
// MCP implementation
mcpImpl?: 'in-memory' | 'stdio';
}
```
### Browser Control Modes
- **`dom`**: Direct DOM manipulation (fastest, most reliable)
- **`visual-grounding`**: Vision-based interaction (most flexible)
- **`hybrid`**: Combines both approaches (recommended)
## Advanced Usage
### Custom Instructions
```typescript
const agent = new AgentTARS({
instructions: `
You are a specialized research assistant.
Focus on academic papers and technical documentation.
Always provide citations and sources.
`,
// ... other options
});
```
### Browser State Management
```typescript
// Get browser control information
const browserInfo = agent.getBrowserControlInfo();
console.log(`Mode: ${browserInfo.mode}`);
console.log(`Tools: ${browserInfo.tools.join(', ')}`);
// Access browser manager
const browserManager = agent.getBrowserManager();
if (browserManager) {
const isAlive = await browserManager.isBrowserAlive();
console.log(`Browser status: ${isAlive ? 'alive' : 'dead'}`);
}
```
### Workspace Operations
```typescript
// Get current workspace
const workspace = agent.getWorkingDirectory();
console.log(`Working in: ${workspace}`);
// All file operations are automatically scoped to workspace
const result = await agent.run('Create a summary.md file with today\'s findings');
```
## Error Handling
```typescript
try {
await agent.initialize();
const result = await agent.run('Your task here');
} catch (error) {
console.error('Agent error:', error);
} finally {
// Always cleanup
await agent.cleanup();
}
```
## API Reference
### Core Methods
- `initialize()`: Initialize the agent and all components
- `run(message)`: Execute a task with the given message
- `cleanup()`: Clean up all resources
- `getWorkingDirectory()`: Get current workspace path
- `getBrowserControlInfo()`: Get browser control status
- `getBrowserManager()`: Access browser manager instance
### Events
The agent emits events through the event stream:
```typescript
agent.eventStream.subscribe((event) => {
if (event.type === 'tool_result') {
console.log(`Tool ${event.name} completed`);
}
});
```
## Resources
- 📄 [Blog Post](https://agent-tars.com/beta)
- 🐦 [Release Announcement on Twitter](https://x.com/_ulivz/status/1938009759413899384)
- 🐦 [Official Twitter](https://x.com/agent_tars)
- 💬 [Discord Community](https://discord.gg/HnKcSBgTVx)
- 💬 [飞书交流群](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=deen76f4-ea3c-4964-93a3-78f126f39651)
- 🚀 [Quick Start](https://agent-tars.com/quick-start)
- 💻 [CLI Documentation](https://agent-tars.com/guide/basic/cli.html)
- 🖥️ [Web UI Guide](https://agent-tars.com/guide/basic/web-ui.html)
- 📁 [Workspace Documentation](https://agent-tars.com/guide/basic/workspace.html)
- 🔌 [MCP Documentation](https://agent-tars.com/guide/basic/mcp.html)
## Features
- 🌐 **Advanced Browser Control**: Multiple control strategies (DOM, Visual, Hybrid)
- 📁 **Safe Filesystem Operations**: Workspace-scoped file management
- 🔍 **Intelligent Search**: Integration with multiple search providers
- 🔧 **MCP Integration**: Built-in Model Context Protocol support
- 📸 **Visual Understanding**: Screenshot-based browser interaction
- 🛡️ **Safety First**: Path validation and workspace isolation
## What's Changed
See Full [CHANGELOG](https://github.com/bytedance/UI-TARS-desktop/blob/main/multimodal/CHANGELOG.md)
## Contributing
See [CONTRIBUTING.md](../../CONTRIBUTING.md) for development guidelines.
## License
Apache-2.0 - See [LICENSE](../../LICENSE) for details.
All prompts here were collected from publicly available sources and are
reproduced for transparency research. Browse the
browser automation agents category, the
full gallery of 400+ products, or read the
paper behind the AISPA standard.