Uso de la herramienta para el uso de computadoras para agentes (versión preliminar)

Captura de pantalla de la inicialización para la ejecución de herramientas de uso informático

En el ejemplo de código siguiente se muestra cómo crear una versión del agente con la herramienta de uso del equipo, enviar una solicitud inicial con una captura de pantalla y realizar varias iteraciones para completar una tarea.

from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import PromptAgentDefinition, ComputerUsePreviewTool

# Import shared helper functions
from computer_use_util import (
    SearchState,
    load_screenshot_assets,
    handle_computer_action_and_take_screenshot,
    print_final_output,
)

"""Main function to demonstrate Computer Use Agent functionality."""
# Initialize state machine
current_state = SearchState.INITIAL

# Load screenshot assets
try:
    screenshots = load_screenshot_assets()
    print("Successfully loaded screenshot assets")
except FileNotFoundError:
    print("Failed to load required screenshot assets. Use the maintained SDK sample on GitHub to get the helper file and images.")
    exit(1)

Creación de una versión del agente con la herramienta

# Format: "https://resource_name.ai.azure.com/api/projects/project_name"
PROJECT_ENDPOINT = "your_project_endpoint"

project = AIProjectClient(
    endpoint=PROJECT_ENDPOINT,
    credential=DefaultAzureCredential(),
)

computer_use_tool = ComputerUsePreviewTool(display_width=1026, display_height=769, environment="windows")

agent = project.agents.create_version(
    agent_name="ComputerUseAgent",
    definition=PromptAgentDefinition(
        model="computer-use-preview",
        instructions="""
        You are a computer automation assistant. 

        Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
        """,
        tools=[computer_use_tool],
    ),
    description="Computer automation agent with screen interaction capabilities.",
)
print(f"Agent created (id: {agent.id}, name: {agent.name})")

Iteración de la herramienta para procesar la captura de pantalla y realizar el paso siguiente

openai = project.get_openai_client()

# Initial request with screenshot - start with Bing search page
response = openai.responses.create(
    input=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
                },
                {
                    "type": "input_image",
                    "image_url": screenshots["browser_search"]["url"],
                    "detail": "high",
                },  # Start with Bing search page
            ],
        }
    ],
    extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
    truncation="auto",
)

print(f"Initial response received (ID: {response.id})")

Realizar varias iteraciones

Asegúrese de revisar cada iteración y acción. En el ejemplo de código siguiente se muestra una solicitud de API básica. Después de enviar la solicitud de API inicial, realice un bucle donde el código de la aplicación lleve a cabo la acción especificada. Envíe una captura de pantalla con cada turno para que el modelo pueda evaluar el estado actualizado del entorno. El ejemplo incluye un número máximo de iteraciones para evitar bucles infinitos, pero puede ajustarlo según sea necesario.

max_iterations = 10  # Allow enough iterations for completion
iteration = 0

while True:
    if iteration >= max_iterations:
        print(f"\nReached maximum iterations ({max_iterations}). Stopping.")
        break

    iteration += 1
    print(f"\n--- Iteration {iteration} ---")

    # Check for computer calls in the response
    computer_calls = [item for item in response.output if item.type == "computer_call"]

    if not computer_calls:
        print_final_output(response)
        break

    # Process the first computer call
    computer_call = computer_calls[0]
    action = computer_call.action
    call_id = computer_call.call_id

    # Handle the action and get the screenshot info
    screenshot_info, current_state = handle_computer_action_and_take_screenshot(action, current_state, screenshots)

    # Regular response with just the screenshot
    response = openai.responses.create(
        previous_response_id=response.id,
        input=[
            {
                "call_id": call_id,
                "type": "computer_call_output",
                "output": {
                    "type": "computer_screenshot",
                    "image_url": screenshot_info["url"],
                },
            }
        ],
        extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
        truncation="auto",
    )

    print(f"Iteration {iteration}: response received (ID: {response.id})")

Limpieza

project.agents.delete_version(agent_name=agent.name, agent_version=agent.version)
print("Agent deleted")

Salida prevista

En el ejemplo siguiente se muestra la salida esperada al ejecutar el ejemplo de código anterior:

Successfully loaded screenshot assets
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
    Click at (512, 384) - Simulating click on UI element
    -> Assuming click on Search button when search field was populated, displaying results.
    -> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted

Ejemplo para el uso de un agente con la herramienta Uso de computadoras

En el ejemplo de código de C# siguiente se muestra cómo crear una versión del agente con la herramienta de uso del equipo, enviar una solicitud inicial con una captura de pantalla y realizar varias iteraciones para completar una tarea. Para permitir que el agente use la herramienta de uso del equipo, use ResponseTool.CreateComputerTool() al configurar las herramientas del agente. En este ejemplo se usa código sincrónico. Para el uso asincrónico, consulte el ejemplo sample code en el Azure SDK para .NET repositorio en GitHub.

using System;
using System.Runtime.CompilerServices;
using Azure.AI.Projects;
using Azure.AI.Projects.OpenAI;
using Azure.Identity;

class ComputerUseDemo
{
    // Format: "https://resource_name.ai.azure.com/api/projects/project_name"
    private const string ProjectEndpoint = "your_project_endpoint";

    // Read image files using `ReadImageFile` method.
    private static BinaryData ReadImageFile(string name, [CallerFilePath] string pth = "")
    {
        var dirName = Path.GetDirectoryName(pth) ?? "";
        return new BinaryData(File.ReadAllBytes(Path.Combine(dirName, name)));
    }

    // Create a helper method to parse the ComputerTool outputs and to respond
    // to Agents queries with new screenshots. Note that throughout
    // this sample the media type for image is set. Agents support `image/jpeg`,
    // `image/png`, `image/gif` and `image/webp` media types.
    private static string ProcessComputerUseCall(ComputerCallResponseItem item, string oldScreenshot)
    {
        string currentScreenshot = "browser_search";
        switch (item.Action.Kind)
        {
            case ComputerCallActionKind.Type:
                Console.WriteLine($"  Typing text \"{item.Action.TypeText}\" - Simulating keyboard input");
                currentScreenshot = "search_typed";
                break;
            case ComputerCallActionKind.KeyPress:
                HashSet<string> codes = new(item.Action.KeyPressKeyCodes);
                if (codes.Contains("Return") || codes.Contains("ENTER"))
                {
                    // If we have typed the value to the search field, go to search results.
                    if (string.Equals(oldScreenshot, "search_typed"))
                    {
                        Console.WriteLine("  -> Detected ENTER key press, when search field was populated, displaying results.");
                        currentScreenshot = "search_results";
                    }
                    else
                    {
                        Console.WriteLine("  -> Detected ENTER key press, on results or unpopulated search, do nothing.");
                        currentScreenshot = oldScreenshot;
                    }
                }
                else
                {
                    Console.WriteLine($"  Key press: {item.Action.KeyPressKeyCodes.Aggregate("", (agg, next) => agg + "+" + next)} - Simulating key combination");
                }
                break;
            case ComputerCallActionKind.Click:
                Console.WriteLine($"  Click at ({item.Action.ClickCoordinates.Value.X}, {item.Action.ClickCoordinates.Value.Y}) - Simulating click on UI element");
                if (string.Equals(oldScreenshot, "search_typed"))
                {
                    Console.WriteLine("  -> Assuming click on Search button when search field was populated, displaying results.");
                    currentScreenshot = "search_results";
                }
                else
                {
                    Console.WriteLine("  -> Assuming click on Search on results or when search was not populated, do nothing.");
                    currentScreenshot = oldScreenshot;
                }
                break;
            case ComputerCallActionKind.Drag:
                string pathStr = item.Action.DragPath.ToArray().Select(p => $"{p.X}, {p.Y}").Aggregate("", (agg, next) => $"{agg} -> {next}");
                Console.WriteLine($"  Drag path: {pathStr} - Simulating drag operation");
                break;
            case ComputerCallActionKind.Scroll:
                Console.WriteLine($"  Scroll at ({item.Action.ScrollCoordinates.Value.X}, {item.Action.ScrollCoordinates.Value.Y}) - Simulating scroll action");
                break;
            case ComputerCallActionKind.Screenshot:
                Console.WriteLine("  Taking screenshot - Capturing current screen state");
                break;
            default:
                break;
        }
        Console.WriteLine($"  -> Action processed: {item.Action.Kind}");

        return currentScreenshot;
    }

    public static void Main()
    {
        // Create project client
        AIProjectClient projectClient = new(endpoint: new Uri(ProjectEndpoint), tokenProvider: new DefaultAzureCredential());

        // Read in three example screenshots and place them into a dictionary.
        Dictionary<string, BinaryData> screenshots = new() {
            { "browser_search", ReadImageFile("Assets/cua_browser_search.png")},
            { "search_typed", ReadImageFile("Assets/cua_search_typed.png")},
            { "search_results", ReadImageFile("Assets/cua_search_results.png")},
        };

        // Create a PromptAgentDefinition with ComputerTool.
        PromptAgentDefinition agentDefinition = new(model: "computer-use-preview")
        {
            Instructions = "You are a computer automation assistant.\n\n" +
                            "Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.",
            Tools = {
                ResponseTool.CreateComputerTool(
                    environment: new ComputerToolEnvironment("windows"),
                    displayWidth: 1026,
                    displayHeight: 769
                ),
            }
        };
        AgentVersion agentVersion = projectClient.Agents.CreateAgentVersion(
            agentName: "myAgent",
            options: new(agentDefinition)
        );
        // Create an `ResponseResult` using `ResponseItem`, containing two `ResponseContentPart`:
        // one with the image and another with the text. In the loop, request Agent
        // while it is continuing to browse web. Finally, print the tool output message.
        ProjectResponsesClient responseClient = projectClient.OpenAI.GetProjectResponsesClientForAgent(agentVersion.Name);
        CreateResponseOptions responseOptions = new()
        {
            TruncationMode = ResponseTruncationMode.Auto,
            InputItems =
            {
                ResponseItem.CreateUserMessageItem(
                [
                    ResponseContentPart.CreateInputTextPart("I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete."),
                    ResponseContentPart.CreateInputImagePart(imageBytes: screenshots["browser_search"], imageBytesMediaType: "image/png", imageDetailLevel: ResponseImageDetailLevel.High)
                ]),
            },
        };
        bool computerUseCalled = false;
        string currentScreenshot = "browser_search";
        int limitIteration = 10;
        ResponseResult response;
        do
        {
            response = responseClient.CreateResponse(responseOptions);
            computerUseCalled = false;
            responseOptions.InputItems.Clear();
            responseOptions.PreviousResponseId = response.Id;
            foreach (ResponseItem responseItem in response.OutputItems)
            {
                responseOptions.InputItems.Add(responseItem);
                if (responseItem is ComputerCallResponseItem computerCall)
                {
                    currentScreenshot = ProcessComputerUseCall(computerCall, currentScreenshot);
                    responseOptions.InputItems.Add(ResponseItem.CreateComputerCallOutputItem(callId: computerCall.CallId, output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageBytes: screenshots[currentScreenshot], screenshotImageBytesMediaType: "image/png")));
                    computerUseCalled = true;
                }
            }
            limitIteration--;
        } while (computerUseCalled && limitIteration > 0);
        Console.WriteLine(response.GetOutputText());

        // Clean up resources by deleting Agent.
        projectClient.Agents.DeleteAgentVersion(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
    }
}

Salida prevista

En el ejemplo siguiente se muestra la salida esperada al ejecutar el ejemplo de código anterior:

Agent created (id: ..., name: myAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: Type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
  Click at (512, 384) - Simulating click on UI element
  -> Assuming click on Search button when search field was populated, displaying results.
  -> Action processed: Click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted

Ejemplo para el uso de un agente con la herramienta Uso de computadoras

En el ejemplo de código typeScript siguiente se muestra cómo crear una versión del agente con la herramienta de uso del equipo, enviar una solicitud inicial con una captura de pantalla y realizar varias iteraciones para completar una tarea. Para obtener un ejemplo de JavaScript, consulte el código sample en el repositorio Azure SDK para JavaScript en GitHub.

import { DefaultAzureCredential } from "@azure/identity";
import { AIProjectClient } from "@azure/ai-projects";
import {
  SearchState,
  loadScreenshotAssets,
  handleComputerActionAndTakeScreenshot,
  printFinalOutput,
  type ComputerAction,
} from "./computerUseUtil.js";

// Format: "https://resource_name.ai.azure.com/api/projects/project_name"
const PROJECT_ENDPOINT = "your_project_endpoint";

export async function main(): Promise<void> {
  // Initialize state machine
  let currentState = SearchState.INITIAL;

  // Load screenshot assets
  const screenshots = loadScreenshotAssets();
  console.log("Successfully loaded screenshot assets");

  // Create AI Project client
  const project = new AIProjectClient(PROJECT_ENDPOINT, new DefaultAzureCredential());
  const openai = project.getOpenAIClient();

  console.log("Creating Computer Use Agent...");
  const agent = await project.agents.createVersion("ComputerUseAgent", {
    kind: "prompt" as const,
    model: "computer-use-preview",
    instructions: `
You are a computer automation assistant.

Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
    `.trim(),
    tools: [
      {
        type: "computer_use_preview",
        display_width: 1026,
        display_height: 769,
        environment: "windows" as const,
      },
    ],
  });
  console.log(`Agent created (id: ${agent.id}, name: ${agent.name}, version: ${agent.version})`);

  // Initial request with screenshot - start with Bing search page
  console.log(
    "Starting computer automation session (initial screenshot: cua_browser_search.png)...",
  );
  let response = await openai.responses.create(
    {
      input: [
        {
          role: "user" as const,
          content: [
            {
              type: "input_text",
              text: "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
            },
            {
              type: "input_image",
              image_url: screenshots.browser_search.url,
              detail: "high",
            },
          ],
        },
      ],
      truncation: "auto",
    },
    {
      body: { agent: { name: agent.name, type: "agent_reference" } },
    },
  );

  console.log(`Initial response received (ID: ${response.id})`);

  // Main interaction loop with deterministic completion
  const maxIterations = 10; // Allow enough iterations for completion
  let iteration = 0;

  while (iteration < maxIterations) {
    iteration++;
    console.log(`\n--- Iteration ${iteration} ---`);

    // Check for computer calls in the response
    const computerCalls = response.output.filter((item) => item.type === "computer_call");

    if (computerCalls.length === 0) {
      printFinalOutput({
        output: response.output,
        status: response.status ?? "",
      });
      break;
    }

    // Process the first computer call
    const computerCall = computerCalls[0];
    const action: ComputerAction = computerCall.action;
    const callId: string = computerCall.call_id;

    console.log(`Processing computer call (ID: ${callId})`);

    // Handle the action and get the screenshot info
    const [screenshotInfo, updatedState] = handleComputerActionAndTakeScreenshot(
      action,
      currentState,
      screenshots,
    );
    currentState = updatedState;

    console.log(`Sending action result back to agent (using ${screenshotInfo.filename})...`);
    // Regular response with just the screenshot
    response = await openai.responses.create(
      {
        previous_response_id: response.id,
        input: [
          {
            call_id: callId,
            type: "computer_call_output",
            output: {
              type: "computer_screenshot",
              image_url: screenshotInfo.url,
            },
          },
        ],
        truncation: "auto",
      },
      {
        body: { agent: { name: agent.name, type: "agent_reference" } },
      },
    );

    console.log(`Follow-up response received (ID: ${response.id})`);
  }

  if (iteration >= maxIterations) {
    console.log(`\nReached maximum iterations (${maxIterations}). Stopping.`);
  }

  // Clean up resources
  console.log("\nCleaning up...");
  await project.agents.deleteVersion(agent.name, agent.version);
  console.log("Agent deleted");

  console.log("\nComputer Use Agent sample completed!");
}

main().catch((err) => {
  console.error("The sample encountered an error:", err);
});

Salida prevista

En el ejemplo siguiente se muestra la salida esperada al ejecutar el ejemplo de código anterior:

Successfully loaded screenshot assets
Creating Computer Use Agent...
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
  Typing text "OpenAI news" - Simulating keyboard input
  -> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
    Click at (512, 384) - Simulating click on UI element
    -> Assuming click on Search button when search field was populated, displaying results.
    -> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Cleaning up...
Agent deleted
Computer Use Agent sample completed!

Uso del ordenador en un agente de Java

Agregue la dependencia a pom.xml:

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-ai-agents</artifactId>
    <version>2.0.0-beta.1</version>
</dependency>

Crear un agente de uso de computadora

import com.azure.ai.agents.AgentsClient;
import com.azure.ai.agents.AgentsClientBuilder;
import com.azure.ai.agents.ResponsesClient;
import com.azure.ai.agents.models.*;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.openai.models.responses.Response;
import com.openai.models.responses.ResponseCreateParams;

import java.util.Collections;

public class ComputerUseExample {
    // Format: "https://resource_name.ai.azure.com/api/projects/project_name"
    private static final String PROJECT_ENDPOINT = "your_project_endpoint";

    public static void main(String[] args) {

        AgentsClientBuilder builder = new AgentsClientBuilder()
            .credential(new DefaultAzureCredentialBuilder().build())
            .endpoint(PROJECT_ENDPOINT);

        AgentsClient agentsClient = builder.buildAgentsClient();
        ResponsesClient responsesClient = builder.buildResponsesClient();

        // Create computer use tool
        ComputerUsePreviewTool tool = new ComputerUsePreviewTool(
            ComputerEnvironment.WINDOWS,
            1024,
            768
        );

        // Create agent with computer use tool
        PromptAgentDefinition agentDefinition = new PromptAgentDefinition("computer-use-preview")
            .setInstructions("You are a computer automation assistant.")
            .setTools(Collections.singletonList(tool));

        AgentVersionDetails agent = agentsClient.createAgentVersion("computer-use-agent", agentDefinition);
        System.out.printf("Agent created: %s (version %s)%n", agent.getName(), agent.getVersion());

        // Create a response with initial screenshot
        AgentReference agentReference = new AgentReference(agent.getName())
            .setVersion(agent.getVersion());

        Response response = responsesClient.createWithAgent(
            agentReference,
            ResponseCreateParams.builder()
                .input("Open the browser and navigate to microsoft.com"));

        System.out.println("Response: " + response.output());

        // The response will contain computer_call items with actions
        // to execute. Process each action, take screenshots, and
        // send results back using responsesClient.createWithAgent()
        // with the previousResponseId and computer call output.

        // Clean up
        agentsClient.deleteAgentVersion(agent.getName(), agent.getVersion());
    }
}

Para obtener un bucle completo de uso del equipo con el control de captura de pantalla, consulte el ejemplo ComputerUseSync.java.

Uso del ordenador con la API REST

Obtener un token de acceso:

export AGENT_TOKEN=$(az account get-access-token --scope "https://ai.azure.com/.default" --query accessToken -o tsv)

Creación de un agente con uso de equipo

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/agents?api-version=v1" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "name": "computer-use-agent",
    "definition": {
      "kind": "prompt",
      "model": "computer-use-preview",
      "instructions": "You are a computer automation assistant.",
      "tools": [
        {
          "type": "computer_use_preview",
          "environment": "windows",
          "display_width": 1024,
          "display_height": 768
        }
      ]
    }
  }'

Genere una respuesta

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
    "input": "Open the browser and navigate to microsoft.com"
  }'

La respuesta incluye computer_call elementos de salida con acciones que se van a ejecutar. Procese cada acción individual, capture capturas de pantalla y envíe de vuelta los resultados mediante el punto de conexión de respuestas con previous_response_id.

Enviar resultados de la acción con captura de pantalla

Después de ejecutar la acción del equipo (por ejemplo, haga clic o escriba), capture una captura de pantalla y vuelva a enviarla:

curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $AGENT_TOKEN" \
  -d '{
    "agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
    "previous_response_id": "<RESPONSE_ID>",
    "input": [
      {
        "type": "computer_call_output",
        "call_id": "<CALL_ID>",
        "output": {
          "type": "computer_screenshot",
          "image_url": "data:image/png;base64,<BASE64_SCREENSHOT>"
        }
      }
    ]
  }'

Reemplace <RESPONSE_ID>, <CALL_ID>y por <BASE64_SCREENSHOT> los valores de la respuesta anterior. Repita este ciclo hasta que el modelo devuelva una respuesta de texto en lugar de .computer_call

Limpieza

curl -X DELETE "$FOUNDRY_PROJECT_ENDPOINT/agents/computer-use-agent?api-version=v1" \
  -H "Authorization: Bearer $AGENT_TOKEN"