Repository navigation
test: run acp-tck against v1 and v2 SDK agents - #147
Conversation
Add STDIO agents built on the stable and v2 Agent runtimes as systems under test for https://github.com/agentclientprotocol/acp-tck (pinned at fa00ccd), with setup/run scripts modelled on Koog's A2A TCK setup. The CI workflow is manual-only until the reported failures are fixed.
EugeneTheDev
changed the title
| @@ -0,0 +1,25 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
It would be better to manage acp-tck through pyproject.toml and uv.lock, and run it with uv run --locked. Currently, changing COMMIT_HASH leaves existing local checkouts untouched, so local runs and CI can use different TCK revisions. This would also remove the custom clone/setup logic and lock transitive dependencies.
There was a problem hiding this comment.
Good point, I haven't though about using uv like this here. Will do
| * A stdin read blocked on another path cannot be interrupted, so exit explicitly once the | ||
| * transport is closed instead of waiting for every coroutine to unwind. | ||
| */ | ||
| exitProcess(0) |
There was a problem hiding this comment.
Looks like this exitProcess(0) is unreachable when the transport closes while stdin is still open
There was a problem hiding this comment.
Also, I think nothing can actually close the transport that way today: System.out is a PrintStream, so it swallows the IOException, which means a dead stdout looks like a successful write and the agent keeps answering nobody
…d of blocking, move exitProcess inside runBlocking
2 participants
Adds conformance testing with acp-tck, set up like Koog's A2A TCK tests.
:test-tck:acp-tck-agent-v1and:test-tck:acp-tck-agent-v2: STDIO agents on the stable and v2Agentruntimes that advertise every capability the suites check.test-tck/setup_tck.shclones acp-tck at a pinned commit;test-tck/run_tck.sh <v1|v2>builds the agent and runs the suite.acpTck.ymlruns both suites and uploads the JSON reports. It is manual-only for now, because the SDK doesn't pass yet.Current results against acp-tck
3d46e21, not fixed in this PR:session/loadreplays history after the response (LOAD-002, capability);session/newwithoutcwdis answered with-32700instead of-32602(ERROR-002, advisory); the agent never connects to the stdio MCP server it is given (SESSION-004, advisory).initializewith versions other than 2 returns-32602instead of selecting a version or-32022withrequested/supporteddata (INIT-003/202/205);session/resumesends nostate_updatesnapshot before responding (RESUME-206), accepts an unsupportedreplayFromcursor (RESUME-207), and answers a malformedreplayFromwith-32603/-32700instead of-32602(RESUME-208);session/closeresponds before the cancelled idlestate_update(CLOSE-203).