A benchmark and RL environment for API clients

Beam reverse-engineers APIs from recorded traffic. Give it a server, and it tests its own guesses.

It scores 99.8% when the recording shows everything and 49.1% when one required header looks optional in the recording. Give it the missing piece and it recovers to 96.0 to 98.9%.

With five test requests at high effort, its pass rate goes from 49.7% → 95.7% (mean of 3 runs).

Dataset Code Pre-registration

84-second tour, captioned

01 The task

A recording in, a client out, hidden tests on a server the model never saw.

Recording GET /v2/invoices?page=2 200 application/json
A HAR file: the requests and responses of one real or invented API, plus a goal that names the methods to write.
Model Beam, GPT-OSS, Qwen
Reads the files and writes one Python file. No LLM judge anywhere in the loop.
client.py class Client: def list_invoices(...)
Must work for any host, key and data, so replaying the recording does not pass.
Hidden tests fresh server new data, keys, host
The score is the share of hidden tests that pass.

Boxes show an example of each file type, not a real task.

1
Build

Goal and recording in, client.py out, in one call. Tier B is 50 invented stateful APIs, each built with six switches. Tier A is 27 tasks from real public API specs. The leaderboard uses Tier B.
- Login type
- Paging
- Token expiry
- Version check
- Hidden header
- Response envelope
Investigate

The same 50 tasks, but the model may send test requests to a practice server before it writes. The control arm has the tool removed (no probes). The other arm allows up to 5 probes.
Repair

An old client, the error it now throws and a short new recording. Five of the ten kinds of change are silent: the server still answers 200, but the data looks different. 64 tasks in 7 families. One call, no tools, no retries. Two arms: the base task, and the same task with a contract hint in the goal text.

02 Leaderboard

Seven settings, five benchmarks, one grader.

Pass rate is the share of hidden tests passed, averaged over tasks. Hover or focus a cell for its 95% confidence interval. Click a column to sort.

03 Effort

Effort helps, but less than the missing piece, and it costs about four times the tokens.

04 What extra information buys

Telling the model the missing piece moves the score more than any effort setting.

Each row is one setting. The hollow dot is the score without the extra information, the filled dot is with it.

05 Beam in detail

Where the extra information lands, by effort.

Light bars are the score with the piece missing. Dark bars are the score with it supplied. Low and medium are one run per task; high is the mean of three runs in cases 1 and 3 and in case 2 with 5 probes.

06 What we found

Six findings, each tested before we looked.

Each card is a pre-registered hypothesis (see PREREG.md) or a headline result. Numbers are pass rates on hidden tests; brackets are 95% bootstrap intervals over tasks.

SupportedH16

Beam fails to notice a silent API change more than it fails to fix it.

No hint46.0%
With hint87.7%

Pass rate, %. Beam, high effort, mean of 3 runs, 64 tasks.

+41.7 ptswith the hint, high effort

We tested: Case 3: the same 64 broken clients, with and without a one-line hint that names what changed.
Result: +41.7 pts [33.5, 49.8] at high effort
Why: Told what changed, Beam repairs most clients. Not told, it keeps trusting the old client and does not check it against the new traffic.

SupportedH22

The hint works because of what it says, not because it is extra text.

No hint31.6%
Placebo hint32.1%
Real hint71.6%

Pass rate, %. Beam, low effort, 1 run, 64 tasks.

+39.5 ptsreal hint over placebo, low effort

We tested: A placebo hint of the same length that says nothing about the change.
Result: Placebo: +0.5 pts [−6.7, +7.4]. Real hint over placebo: +39.5 pts [+30.0, +49.2].
Why: The gain comes from the information about the change. Extra words alone do nothing.

RejectedH17

Pre-registered ~~Effort does not fix noticing.~~

Thinking longer helps Beam notice, but slowly.

Low effort31.6%
High effort46.0%

Pass rate, %. Beam, no hint, 64 tasks. Low is 1 run, high is the mean of 3.

+14.4 ptshigh over low effort, no hint

We tested: Case 3 with no hint, low effort against high effort.
Result: +14.4 pts [8.3, 20.8] for 4.4× the output tokens
Why: At high effort Beam checks more of the recording against the client and catches some changes. A one-line hint gives about three times the gain at no extra cost.

SupportedH10

When the recording hides a header, the model lacks evidence, not instructions.

As built49.1%
With hint96.0%
With evidence98.9%

Pass rate, %. Beam, high effort, 25 tasks. As built is the mean of 3 runs, the others 1 run.

+49.7 ptsevidence over as built, high effort

We tested: Case 1 tasks where the recording hides a required header (n = 25): add a hint, or add the missing evidence.
Result: Hint +46.9 pts, evidence +49.7 pts.
Why: Beam cannot guess a header it never saw. Give it the evidence and the gap closes.

ResultCase 2

At high effort, Beam runs its own experiments.

Low effort4/25 tasks16.0%
Medium effort10/25 tasks40.0%
High effort17/25 tasks68.0%

Share of the 25 hidden-header tasks where Beam tested the header, %. Beam, up to 5 probes, 1 run.

17/25tasks where Beam tested the header, high effort

We tested: Case 2, 25 tasks whose recording hides a required header. Beam may send 5 requests to a practice server.
Result: Tested the header in 4/25, 10/25, 17/25 tasks; pass rate on these tasks 50.3% → 78.3% → 99.4%. On all 50 Case 2 tasks at high effort: 49.7% with no probes, 95.7% with five (mean of 3 runs).
Why: With more effort Beam forms a guess, designs a request to test it, and writes the client from the answer.

SupportedH19

More changes per release are harder to repair.

Single changen = 1497.6%
3 changesn = 1076.7%
4 changesn = 875.0%

Full-repair share, %. Beam, hint arm, high effort, mean of 3 runs.

−22.6 ptsfull repair, four changes against one

We tested: Case 3 with the hint, high effort, grouped by how many changes a release has.
Result: Full repair falls from 97.6% with one change to 76.7% with three and 75.0% with four.
Why: Each change is one more thing to find. One miss fails the client.

Not settled: H20, H21 and H23 are inconclusive at this sample size; H18 was not tested. Full table in Method.

07 Watch an episode

Watch Beam run an experiment.

Three real Case 2 episodes. The recording carries a header that the server secretly requires. With five probes at high effort, Beam tests it on the practice server and writes the client from the answer. With one probe, it guesses.

08 Explore a task

What the model sees.

Only the visible files are shown: the goal, the requests in the recording and one short excerpt. Hidden tests, credentials for grading and reference clients are not published.

09 Method

Hidden tests, no judge, hypotheses written first.

Grading

Each task starts a new fake server with new data, a new key and a new host. The submitted client.py runs against it. A hidden test suite counts what passes. A client that copies the recording fails. Missing submissions count as failures. API errors are excluded and stay visible in the raw files.

How Beam was used

Model Beam-1.0-501B-A23B through Reflection's OpenAI-compatible API, with reasoning_effort set to low, medium or high on every call.
Cases 1 and 3 are one call with no tools. Case 2 is multi-turn tool calling: http_request to the practice server and submit_client to end. The reasoning_content of each assistant turn is sent back in the next request.
Beam at high effort in cases 1 and 3, and in case 2 with 5 probes, is the mean of 3 runs per task. Everything else is one run.

Comparing models

GPT-OSS-120B and Qwen3.5-397B-A17B ran through the Hugging Face router (provider Novita), one run per cell.
Their output was capped at 32,768 tokens per call. Beam's runs used 65,536.
Qwen was run with thinking on. It has no effort levels, so it is never labelled low, medium or high and is not equivalent to any Beam effort.
Intervals are percentile bootstraps over tasks ( resamples).

Pre-registered hypotheses

Written in PREREG.md before the runs. Verdicts come from the analysis summaries. H18 was not tested because the xhigh effort level was dropped.

Pre-registered hypotheses and their verdicts
ID	Claim	Verdict	Evidence
H10	The hidden-header failure is about evidence, not prompting	supported	hint - as built +46.9 [+30.1, +63.0] pts; evidence - as built +49.7 [+33.3, +66.1] pts (n=25); larger: evidence; hint arm 96.0%; headline is a prompting effect
H16	Beam fails to notice a silent interface break more than it fails to fix it	supported	low: hint - base +40.0 pts [95% CI +29.4, +50.1] (n=64); high (mean of 3 runs): hint - base +41.7 pts [95% CI +33.5, +49.8] (n=64)
H17	Effort does not fix noticing	rejected	base, high (mean of 3 runs) - low +14.4 pts [95% CI +8.3, +20.8] (n=64); output high/low = 4.41x
H18	Effort has diminishing returns on repair (H14 on v3; per arm)	not tested	not tested: xhigh dropped 2026-10-27, see PREREG.md deviations
H19	More changes per release are harder (H15 on v3)	supported	full-repair share, hint, high (mean of 3 runs): single 97.6% (n=14), release3 76.7% (n=10), release4 75.0% (n=8)
H20	Missing or wrong side information costs more than more code (k = 3 families, hint arm, high)	inconclusive	pass rate, hint, high (mean of 3 runs): partial - release3 -27.7 pts [95% CI -56.6, -1.1] (n=18) (n=8 vs 10); full-repair -30.8 pts; misleading - release3 +6.9 pts [95% CI -3.3, +16.6] (n=18) (n=8 vs 10); full-repair +6.7 pts; terse - release3 +8.1 pts [95% CI -2.8, +18.2] (n=18) (n=8 vs 10); full-repair +15.0 pts (terse is shown, not used)
H21	A layered SDK client is harder to repair than a compact one (k = 3)	inconclusive	pass rate, high (mean of 3 runs): base: sdk - release3 +58.6 pts [95% CI +36.7, +76.6] (n=18) (n=8 vs 10); full-repair +68.3 pts; hint: sdk - release3 -3.4 pts [95% CI -28.0, +15.0] (n=18) (n=8 vs 10); full-repair +6.7 pts
H22	The hint effect is not a generic prompt effect	supported	placebo - base +0.5 [-6.7, +7.4] pts (n=64); hint - placebo +39.5 [+30.0, +49.2] pts (n=64)
H23	A warning alone gives a large part of the hint effect	inconclusive	notice - base +18.7 [+10.0, +27.6] pts (n=64)

10 Use it

Grade a client in three commands. Train against it as an environment.

Install

git clone https://github.com/MohitGoyal09/har2client && cd har2client/har2client
uv sync

Grade a reference client

No API key and no model call. Build one Tier B task, then grade any client.py you wrote or a model returned. The repository README also shows how to grade a task's own reference client, which scores pass_rate=1.000.

uv run python scripts/build_tierb.py --out runs/quickstart/tierB --seed 1 --limit 1 --configs runs/quickstart/tierb_configs.jsonl
T=runs/quickstart/tierB/tierb_00_invoices
uv run har2client grade $T client.py

As an RL environment

An episode is observe, then submit one client. The reward is the pass rate on a fresh grading instance. The environments are built and tested. No training run is reported.

from har2client.env.env import TrafficToClientEnv
from har2client.env.repair_env import RepairEnv


def main() -> None:
    # Case 1: observed traffic -> client
    env = TrafficToClientEnv(source="browser_har", split="train", curriculum="L0", compact=True, seed=0)
    obs = env.reset()  # task_id, goal, observation (the recording), observation_format, probe_budget, source
    out = env.step({"type": "submit", "code": env.reference_code()})  # replace with your model's code
    print("case 1 reward:", out["reward"], "done:", out["done"])  # 1.0 for the reference client
    env.close()

    # Case 3: repair a client after the API changed
    repair = RepairEnv(split="train", curriculum="R0", seed=0)
    obs = repair.reset()  # task_id, goal, client_v1, failure, observation, observation_format
    out = repair.step({"type": "submit", "code": obs["client_v1"]})  # a no-op patch: fails some tests
    print("case 3 reward:", out["reward"], "regression:", out["info"]["regression_rate"])
    repair.close()


if __name__ == "__main__":  # the grader uses spawned worker processes: keep this guard
    main()

Cite

@software{har2client2026,
  title   = {HAR2Client: a benchmark for turning recorded HTTP traffic into working API clients},
  author  = {Goyal, Mohit and Bhattacharya, Ananya},
  year    = {2026},
  version = {0.1},
  url     = {https://github.com/MohitGoyal09/har2client},
  license = {Apache-2.0}
}

Licences

Code: Apache-2.0. Data and results: CC-BY-4.0. Tier A tasks keep the licence of the public API spec they come from; the dataset card lists it per task.

Authors

Mohit Goyal and Ananya Bhattacharya.

Entry for the Reflection x Hugging Face "Build with Beam" challenge.