How does an Agent
use Slack?

I started with a task that needs a real saved change

• The agent must find Alex's launch-review message and change its time from 14:00 to 15:00 UTC.
• Typing the new time is not enough: it must save the edit without changing other messages.

• A static mockup cannot test persistence. A full Slack backend adds services these tasks do not need.
• Fictional data · Task abbreviated here · Try the workspace without a key · Prior-work review.

The model proposes an action; the harness executes it

• The harness accepts only supported actions, so a model response cannot execute arbitrary code.
• After Save, it reads the updated page so the model can check whether the edit appeared.
• Action, time, and spending limits stop a stuck agent from making model calls indefinitely.

• The example shows browser control. The API path executes application operations instead.

The model can use four interfaces to the same app

• A browser edit requires finding the editor, entering text, and saving the change.
• The actor API can request that edit directly, so the workspace changes without a browser click.
• The same grader checks the result, but the model uses different information and actions to reach it.

• Page JSON describes page controls, not the underlying database. Actor API access exposes broader structured data.
• The harness uses a shared JSON action protocol, not each provider's native computer-use protocol.

Success means the requested change was saved

• A model can type the right text and click Cancel, so the screen alone cannot prove success.
• The grader is code that checks the stored messages, topics, and other workspace data.

• The grader checks the final state. It does not penalize a wrong action that the agent fully undoes.
• Task definitions and result checks.

I tested the success rule on the same edit

• Alex's original message must change from 14:00 to 15:00, with no other changes.

• This helps distinguish software errors from a model taking the wrong action.
• Passing these tests does not prove the task instructions are clear.

• Software verification · Broader workflow tests cover 84 valid and 2,583 invalid states across seven seeds.

Each run starts with its own copy of the task

• Even with one model, each attempt needs fresh data so it cannot inherit completed work.
• Each run gets its own SQLite database and browser session, so its actions change only its own workspace.

• Tested: write to A, check B is unchanged; close A, check B still works. Cost: separate files and browser setup.
• The actor and control servers share a process. Separate data is not isolation from host crashes or malicious code.

What makes one run take longer?

• A fresh workspace makes runs independent, but each extra decision adds another model request.
• In the live viewer, the workspace waits during the model request and updates after the action executes.

• A faster reset helps between runs, not between decisions. Local M5 Max / Node 24 medians exclude model calls and are not a full-run estimate.
• Samples and setup: 100 stored sessions; screenshots from 10 sequential browser contexts, not 100 concurrent agents.

Each run consumes memory and storage

• The browser stays open while the model decides, so a slow run also holds memory for longer.

• More simultaneous runs need more browser memory.
• More saved runs need more storage.

• 48 KiB is fixture disk size, not memory per run. Full browser memory and recording size need separate measurements.
• Resource choices and measurements · Experiment runs close their own browsers. Browser reuse is untested.

A queue would limit how many browsers run at once

• Because each active run holds a browser in memory, starting every visitor immediately could exhaust the machine.
• I would queue new runs until a worker is free, while keeping each active run's data separate.

• Concurrent capacity is unmeasured. Test 1, 4, 8, and 16 workers for throughput, memory, provider limits, and crash cleanup.
• Use separate processes or containers for stronger containment. Browser pooling trades lower startup cost for shared failure risk.

The reusable harness made model comparisons possible

• Once runs were repeatable, I wanted to compare how different models completed the same tasks.
• I expanded the assignment into a benchmark prototype with two separate comparisons.
• The first compares models using accessibility controls, and the second compares four interfaces.

Comparison 1: 17 models, using accessibility controls

• Each model receives named page controls and returns one JSON action per request.
• All 306 runs use the same starting data, recent-four history, and no site guide.
• This tests model performance under one interface, not the best interface for each model.

• Seed 1042 · 4,096 output tokens per request · identical graders · provider-default reasoning.
• I kept 105 earlier runs and added 201 new runs without replacing failures.
• Node 24 setup and API key instructions · Exact routes, limits, and run history. Provider routing and collection time were not controlled.

Why was the highest score 14/18?

• GPT-6.1 Sol and GPT-6 Astra each passed 14 tasks and missed the same four.

• On the last three tasks, Sol reached 40 actions and Astra hit the $5 estimated-allowance limit. That is not billed cost.
• Failure analysis and traces. Across all models, 15/18 tasks passed at least once. One attempt per model and task; no grades changed.

What the early interface tests taught me

• Early interface tests recorded successful, incorrect and interrupted runs.
• They used different task and model combinations, so their pooled scores could not compare interfaces.
• For the next study, I gave every model the same six tasks through all four interfaces.

• Early development traces are separate evidence, not a third benchmark or part of the later scores.
• Later study: randomized interface order · fixed seed, history, reasoning, and limits · $25 total · no retries. Full design and controls.

The next version needs validated, unseen task variants

• Ask someone outside the project to solve each task from its instructions.
• Compare their interpretation with the expected result.
• Set run limits with both short and long tasks.
• Use new task variants and repeat runs to measure reliability.

• Detailed engineering review · Remaining validation work. These are next steps, not completed results.

Demo and design questions

• Which workflows should the next version test?
• Should the score include wrong actions that the agent later undoes?
• How many agents must run at once, and what should each run cost?

• Relay can reset, execute actions, and return a reward. No reinforcement learning training has been run.
• Code and setup · Requirements and remaining work.