Screen Actions

Actions

Screen content, screenshots, and node interactions

Access these methods through agent.actions. Get screen content, take screenshots, and interact with UI nodes.

`screenContent()`

TypeScript

screenContent(): Promise<AndroidNode>

Gets the accessibility tree of the currently focused window. Returns an AndroidNode representing the root of the UI hierarchy.

Returns

Promise<AndroidNode>Root node of the accessibility tree

Examples

TypeScript

const screen = await agent.actions.screenContent();

// Find elements
const button = screen.findTextOne("Submit");
const allButtons = screen.filterAdvanced(f => f.isButton());
const input = screen.findAdvanced(f => f.isEditText().isEditable());

`allScreensContent()`

TypeScript

allScreensContent(): Promise<AndroidNode[]>

Gets the accessibility trees from all visible windows (useful for dialogs, overlays).

Returns

Promise<AndroidNode[]>Array of root nodes for each window

Examples

TypeScript

const screens = await agent.actions.allScreensContent();
for (const screen of screens) {
  const dialog = screen.findTextOne("OK");
  if (dialog) break;
}

`screenshot()`

TypeScript

screenshot(maxWidth: number, maxHeight: number, quality: number, cropX1?: number, cropY1?: number, cropX2?: number, cropY2?: number): Promise<{screenshot: string | null, compressedWidth: number, compressedHeight: number, originalWidth: number, originalHeight: number}>

Takes a screenshot with optional scaling and cropping.

Parameters

Name	Type	Description
`maxWidth`	`number`	Maximum width to scale to
`maxHeight`	`number`	Maximum height to scale to
`quality`	`number`	JPEG quality (1-100)
`cropX1?`	`number`	Crop region left
`cropY1?`	`number`	Crop region top
`cropX2?`	`number`	Crop region right
`cropY2?`	`number`	Crop region bottom

Returns

{screenshot, compressedWidth, compressedHeight, originalWidth, originalHeight}Screenshot data as base64 string with dimensions

Examples

Full screenshot

TypeScript

const result = await agent.actions.screenshot(1080, 1920, 80);

Cropped screenshot

TypeScript

const result = await agent.actions.screenshot(500, 500, 90, 100, 100, 600, 600);

`nodeAction()`

TypeScript

nodeAction(node: AndroidNode | object, actionInt: number, data?: object, fieldsToIgnore?: string[]): Promise<{actionPerformed: boolean}>

Performs an accessibility action on a node.

Parameters

Name	Type	Description
`node`	`AndroidNode \| object`	The node to perform action on
`actionInt`	`number`	Action constant (use agent.constants.ACTION_*)
`data?`	`object`	Additional action data
`fieldsToIgnore?`	`string[]`	Node fields to ignore when matching

Returns

{actionPerformed: boolean}Whether the action was successfully performed

Examples

Using node.performAction (preferred)

TypeScript

const screen = await agent.actions.screenContent();
const button = screen.findTextOne("Submit");

if (button) {
  const result = await button.performAction(agent.constants.ACTION_CLICK);
  console.log("Clicked:", result.actionPerformed);
}

Using agent.actions.nodeAction (legacy)

TypeScript

const result = await agent.actions.nodeAction(
  button,
  agent.constants.ACTION_CLICK
);

`showNotification()`

TypeScript

showNotification(title: string, message: string): Promise<void>

Shows a system notification.

Parameters

Name	Type	Description
`title`	`string`	Notification title
`message`	`string`	Notification message

Returns

Promise<void>Resolves when notification is shown

Examples

TypeScript

await agent.actions.showNotification("Task Complete", "Your automation has finished.");

`speak()`

TypeScript

speak(text: string, delayMs?: number): Promise<void>

Reads text aloud using the device's text-to-speech (TextToSpeech) engine.

Parameters

Name	Type	Description
`text`	`string`	The text to speak
`delayMs?`	`number`	Delay in milliseconds before speaking

Returns

Promise<void>Resolves once the utterance has been queued to the TTS engine

Examples

TypeScript

await agent.actions.speak("Automation finished");

Speak after a delay

TypeScript

await agent.actions.speak("Heads up", 1000);

`displayDialog()`

TypeScript

displayDialog(text: string, heading: string, overlayOtherApps?: boolean, delayMs?: number): Promise<void>

Displays a simple dialog to the device user, reusing the in-app-message system (the same overlay/dialog shown by push notifications). With overlayOtherApps=true the dialog is drawn over whatever app is in the foreground (requires the accessibility service); otherwise it is shown the next time the app is opened.

Parameters

Name	Type	Description
`text`	`string`	The dialog body text
`heading`	`string`	Heading shown above the body (pass an empty string for none)
`overlayOtherApps?`	`boolean`	Show immediately as an overlay over other apps (default false)
`delayMs?`	`number`	Delay in milliseconds before showing the dialog

Returns

Promise<void>Resolves once the dialog has been dispatched

Examples

TypeScript

await agent.actions.displayDialog("Please keep the screen on", "Notice");

Overlay over the current app

TypeScript

await agent.actions.displayDialog("Do not touch", "Automation running", true);