main · last commit
13 days ago ·
7g0stsfu
ah-tqc Runner: an errored-final-turn pi run (exit 0, zero tokens) finalizes as success
Past Stand
bd reopen ah-tqc
| Created by | Eugene Blikh |
| Owner | bigbes@gmail.com |
| Created | 2026-07-19T05:38:11Z |
| Started | 2026-08-04T23:24:53Z |
| Updated | 2026-08-04T23:46:00Z |
| Closed | 2026-08-04T23:46:00Z |
Live incident 2026-07-19 (task 4, Q&A smoke): pi --mode json made exactly one model call, the litellm proxy 403'd it (key not allowed for model 'coder'), pi recorded stopReason:error with zero usage and EXITED 0 in ~0.87s. The daemon trusts the exit code: outcome=success, card advanced to in_review, Q&A delivery then correctly reported 'no answer.json' — a broken model config masquerades as a successful run. Fix direction: Status/finalize (or ahub-run) should inspect the tail of events.jsonl — a run whose final assistant turn has stopReason:error (or whose agent_end follows zero completed tool/text turns) should finalize as outcome=error regardless of exit code. Second finding to fold in: /api/v1/status shows cost_usd=0 for ALL runs including real multi-minute ones (tasks 1-2), so the usage.cost.total accumulation from events.jsonl appears broken on pi 0.73.x — re-verify the event shape and fix the cost parse.
FINDING 1 (errored-final-turn ⇒ false success) FIXED in 665e805. Root cause confirmed as filed: applyExit mapped exit.json's code straight to the §9 outcome, and pi exits 0 even when its only assistant turn was rejected by the provider (stopReason 'error', zero usage). Fix: ParseEvents now also records the LAST assistant message_end's stopReason/errorMessage plus an assistant-message count, exposed as EventStreamInfo.Errored(); applyExit takes the parsed stream and maps exit 0 + Errored() to domain.OutcomeError while still recording the true exit code on the run row. The stream is POSITIVE evidence only — an absent, unreadable, or assistant-turn-less events.jsonl leaves the exit code's verdict alone, so a genuine success can never be flipped by a missing file. Both Status completion branches were factored into (*PiZellij).finalizeFromExit so the ah-2ef re-read and the ah-wka grace-clear can no longer drift apart between them; it WARNs with the provider's rejection text whenever it overrules a zero exit. SPEC §9 'Status resolution' updated to match. Re-verified against a REAL pi 0.82.1 capture that the pinned message_end shape (message.usage.cost.total, message.stopReason, message.errorMessage) is unchanged since the 0.70.2 fixtures. FINDING 2 (cost_usd=0 on every live run) STILL OPEN. Ruled out: the parser. A real pi 0.82.1 run against the direct deepseek provider produces message.usage.cost.total exactly where ParseEvents reads it, and the summation is correct (see ah-1cx.4). Leading hypothesis: agent-1 runs pi through a CUSTOM 'litellm' provider (models are named litellm/<model>), and pi prices a response from its own per-provider model registry — a custom OpenAI-compatible provider has no pricing metadata, so every cost field comes back 0. That would make cost_usd=0 pi's behaviour, not our bug, and the fix would be to source cost from LiteLLM instead. NEEDS LIVE EVIDENCE from agent-1 (an events.jsonl from a real run: is message.usage.cost.total literally 0, or is the usage block shaped differently under the litellm provider?) — the ssh probe is blocked by the local permission classifier, so this needs an operator '!' handoff.
ah-1cx
— Hardening: post-audit bug sweep (2026-07-20)
parent-child
open
Nothing depends on this issue.
| id | ah-tqc |
| content_hash | 0ec5f3f94c113689a7606ace2c0a5475ee68df56a367f991d8d4967233d03efe |
| title | Runner: an errored-final-turn pi run (exit 0, zero tokens) finalizes as success |
| description | Live incident 2026-07-19 (task 4, Q&A smoke): pi --mode json made exactly one model call, the litellm proxy 403'd it (key not allowed for model 'coder'), pi recorded stopReason:error with zero usage and EXITED 0 in ~0.87s. The daemon trusts the exit code: outcome=success, card advanced to in_review, Q&A delivery then correctly reported 'no answer.json' — a broken model config masquerades as a successful run. Fix direction: Status/finalize (or ahub-run) should inspect the tail of events.jsonl — a run whose final assistant turn has stopReason:error (or whose agent_end follows zero completed tool/text turns) should finalize as outcome=error regardless of exit code. Second finding to fold in: /api/v1/status shows cost_usd=0 for ALL runs including real multi-minute ones (tasks 1-2), so the usage.cost.total accumulation from events.jsonl appears broken on pi 0.73.x — re-verify the event shape and fix the cost parse. |
| design | |
| acceptance_criteria | |
| notes | FINDING 1 (errored-final-turn ⇒ false success) FIXED in 665e805. Root cause confirmed as filed: applyExit mapped exit.json's code straight to the §9 outcome, and pi exits 0 even when its only assistant turn was rejected by the provider (stopReason 'error', zero usage). Fix: ParseEvents now also records the LAST assistant message_end's stopReason/errorMessage plus an assistant-message count, exposed as EventStreamInfo.Errored(); applyExit takes the parsed stream and maps exit 0 + Errored() to domain.OutcomeError while still recording the true exit code on the run row. The stream is POSITIVE evidence only — an absent, unreadable, or assistant-turn-less events.jsonl leaves the exit code's verdict alone, so a genuine success can never be flipped by a missing file. Both Status completion branches were factored into (*PiZellij).finalizeFromExit so the ah-2ef re-read and the ah-wka grace-clear can no longer drift apart between them; it WARNs with the provider's rejection text whenever it overrules a zero exit. SPEC §9 'Status resolution' updated to match. Re-verified against a REAL pi 0.82.1 capture that the pinned message_end shape (message.usage.cost.total, message.stopReason, message.errorMessage) is unchanged since the 0.70.2 fixtures. FINDING 2 (cost_usd=0 on every live run) STILL OPEN. Ruled out: the parser. A real pi 0.82.1 run against the direct deepseek provider produces message.usage.cost.total exactly where ParseEvents reads it, and the summation is correct (see ah-1cx.4). Leading hypothesis: agent-1 runs pi through a CUSTOM 'litellm' provider (models are named litellm/<model>), and pi prices a response from its own per-provider model registry — a custom OpenAI-compatible provider has no pricing metadata, so every cost field comes back 0. That would make cost_usd=0 pi's behaviour, not our bug, and the fix would be to source cost from LiteLLM instead. NEEDS LIVE EVIDENCE from agent-1 (an events.jsonl from a real run: is message.usage.cost.total literally 0, or is the usage block shaped differently under the litellm provider?) — the ssh probe is blocked by the local permission classifier, so this needs an operator '!' handoff. |
| status | closed |
| priority | 2 |
| issue_type | bug |
| assignee | Eugene Blikh |
| estimated_minutes | NULL |
| created_at | 2026-07-19T05:38:11Z |
| created_by | Eugene Blikh |
| owner | bigbes@gmail.com |
| updated_at | 2026-08-04T23:46:00Z |
| closed_at | 2026-08-04T23:46:00Z |
| closed_by_session | |
| external_ref | NULL |
| spec_id | |
| compaction_level | 0 |
| compacted_at | NULL |
| compacted_at_commit | NULL |
| original_size | NULL |
| sender | |
| ephemeral | 0 |
| wisp_type | |
| pinned | 0 |
| is_template | 0 |
| mol_type | |
| work_type | |
| source_system | |
| metadata | �{} |
| source_repo | |
| close_reason | BOTH findings resolved. Finding 1 (errored-final-turn ⇒ false success): fixed in 665e805, then COMPLETED in f11ef03 after an adversarial review disassembled the installed pi 0.82.x and found the first pass implemented only half of pi's actual contract. Errored() is now two-armed: (a) the final assistant turn's stopReason is one pi itself calls unusable — 'error' OR 'aborted', since pi's own predicate at print-mode.js:105 is 'error || aborted' and that block lives inside 'if (mode === "text")', which is precisely why json mode exits 0 on both; (b) SessionID != "" && AssistantMessages == 0, the arm the bead named but the first pass never implemented (AssistantMessages was being computed for exactly this and never consulted). Arm (b) closes an EMPIRICALLY REPRODUCED hole: print-mode.js:94 guards the whole run with 'if (initialMessage)', so a blank or whitespace-only rendered PROMPT.md — which prepareTaskDir writes verbatim, unchecked, with nothing in reconcile guarding it either — makes pi do nothing and exit 0 with a one-line stream, sending the card to In Review, publishing an empty review branch and firing the Q&A answer lane. The SessionID != "" gate is kept and heavily commented because it separates 'file present and parsed, zero turns' (evidence) from 'nothing to parse' (ignorance) — the conflation the original positive-evidence-only rationale made. finalizeFromExit's warn log gained a stream_verdict attribute so an operator can tell a provider rejection (pi's text in pi_error) from a did-nothing run. SPEC §9 and domain.Outcome's enum comment (c8083c2) both updated. Finding 2 (cost_usd=0 on every live run) was SPLIT OUT to its own bead — the parser is ruled out, and the leading hypothesis is that pi has no pricing metadata for the custom 'litellm' provider; it needs live evidence from agent-1. Review follow-ups also split out: the failure comment renders 'exit code 0' and drops the provider's rejection text; nothing rejects an empty prompt at the source. |
| event_kind | |
| actor | |
| target | |
| payload | |
| await_type | |
| await_id | |
| timeout_ns | 0 |
| waiters | |
| hook_bead | |
| role_bead | |
| agent_state | |
| last_activity | NULL |
| role_type | |
| rig | |
| due_at | NULL |
| defer_until | NULL |
| no_history | 0 |
| started_at | 2026-08-04T23:24:53Z |
| is_blocked | 0 |
| issue_id | ah-tqc |
| label | milestone:v0.1.0 |
| id | 2e48a09e-41ee-516e-9c68-894bb87425a5 |
| issue_id | ah-tqc |
| type | parent-child |
| created_at | 2026-07-20T02:37:56Z |
| created_by | Eugene Blikh |
| metadata | �{} |
| thread_id | |
| depends_on_issue_id | ah-1cx |
| depends_on_wisp_id | NULL |
| depends_on_external | NULL |
| id | 019f78e1-caf4-7c77-932b-5cb2986a4897 |
| issue_id | ah-tqc |
| event_type | created |
| actor | Eugene Blikh |
| old_value | |
| new_value | |
| comment | NULL |
| created_at | 2026-07-19T08:38:11Z |
| id | 019f7cda-2a1a-7e2c-a94c-8b2b6c8d721b |
| issue_id | ah-tqc |
| event_type | label_added |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Added label: milestone:hardening |
| created_at | 2026-07-20T03:08:20Z |
| id | 019f7cdb-f215-7933-8cae-30d938e2ca2a |
| issue_id | ah-tqc |
| event_type | label_removed |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Removed label: milestone:hardening |
| created_at | 2026-07-20T03:10:16Z |
| id | 019f7cdc-faec-7bc7-9566-f0de60ad90bf |
| issue_id | ah-tqc |
| event_type | label_added |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Added label: milestone:v0.1.0 |
| created_at | 2026-07-20T03:11:24Z |
| id | 019fcf18-22d8-7dda-83c2-527acdc041f0 |
| issue_id | ah-tqc |
| event_type | claimed |
| actor | Eugene Blikh |
| old_value | {"id":"ah-tqc","title":"Runner: an errored-final-turn pi run (exit 0, zero tokens) finalizes as success","description":"Live incident 2026-07-19 (task 4, Q\u0026A smoke): pi --mode json made exactly one model call, the litellm proxy 403'd it (key not allowed for model 'coder'), pi recorded stopReason:error with zero usage and EXITED 0 in ~0.87s. The daemon trusts the exit code: outcome=success, card advanced to in_review, Q\u0026A delivery then correctly reported 'no answer.json' — a broken model config masquerades as a successful run. Fix direction: Status/finalize (or ahub-run) should inspect the tail of events.jsonl — a run whose final assistant turn has stopReason:error (or whose agent_end follows zero completed tool/text turns) should finalize as outcome=error regardless of exit code. Second finding to fold in: /api/v1/status shows cost_usd=0 for ALL runs including real multi-minute ones (tasks 1-2), so the usage.cost.total accumulation from events.jsonl appears broken on pi 0.73.x — re-verify the event shape and fix the cost parse.","status":"open","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-07-19T05:38:11Z","created_by":"Eugene Blikh","updated_at":"2026-07-19T05:38:11Z","labels":["milestone:v0.1.0"]} |
| new_value | {"assignee":"Eugene Blikh","status":"in_progress"} |
| comment | NULL |
| created_at | 2026-08-05T02:24:53Z |
| id | 019fcf1e-9c53-7036-9f0d-4a4bfa72b31a |
| issue_id | ah-tqc |
| event_type | updated |
| actor | Eugene Blikh |
| old_value | {"id":"ah-tqc","title":"Runner: an errored-final-turn pi run (exit 0, zero tokens) finalizes as success","description":"Live incident 2026-07-19 (task 4, Q\u0026A smoke): pi --mode json made exactly one model call, the litellm proxy 403'd it (key not allowed for model 'coder'), pi recorded stopReason:error with zero usage and EXITED 0 in ~0.87s. The daemon trusts the exit code: outcome=success, card advanced to in_review, Q\u0026A delivery then correctly reported 'no answer.json' — a broken model config masquerades as a successful run. Fix direction: Status/finalize (or ahub-run) should inspect the tail of events.jsonl — a run whose final assistant turn has stopReason:error (or whose agent_end follows zero completed tool/text turns) should finalize as outcome=error regardless of exit code. Second finding to fold in: /api/v1/status shows cost_usd=0 for ALL runs including real multi-minute ones (tasks 1-2), so the usage.cost.total accumulation from events.jsonl appears broken on pi 0.73.x — re-verify the event shape and fix the cost parse.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-07-19T05:38:11Z","created_by":"Eugene Blikh","updated_at":"2026-08-04T23:24:53Z","started_at":"2026-08-04T23:24:53Z","labels":["milestone:v0.1.0"]} |
| new_value | {"notes":"FINDING 1 (errored-final-turn ⇒ false success) FIXED in 665e805. Root cause confirmed as filed: applyExit mapped exit.json's code straight to the §9 outcome, and pi exits 0 even when its only assistant turn was rejected by the provider (stopReason 'error', zero usage). Fix: ParseEvents now also records the LAST assistant message_end's stopReason/errorMessage plus an assistant-message count, exposed as EventStreamInfo.Errored(); applyExit takes the parsed stream and maps exit 0 + Errored() to domain.OutcomeError while still recording the true exit code on the run row. The stream is POSITIVE evidence only — an absent, unreadable, or assistant-turn-less events.jsonl leaves the exit code's verdict alone, so a genuine success can never be flipped by a missing file. Both Status completion branches were factored into (*PiZellij).finalizeFromExit so the ah-2ef re-read and the ah-wka grace-clear can no longer drift apart between them; it WARNs with the provider's rejection text whenever it overrules a zero exit. SPEC §9 'Status resolution' updated to match. Re-verified against a REAL pi 0.82.1 capture that the pinned message_end shape (message.usage.cost.total, message.stopReason, message.errorMessage) is unchanged since the 0.70.2 fixtures.\n\nFINDING 2 (cost_usd=0 on every live run) STILL OPEN. Ruled out: the parser. A real pi 0.82.1 run against the direct deepseek provider produces message.usage.cost.total exactly where ParseEvents reads it, and the summation is correct (see ah-1cx.4). Leading hypothesis: agent-1 runs pi through a CUSTOM 'litellm' provider (models are named litellm/\u003cmodel\u003e), and pi prices a response from its own per-provider model registry — a custom OpenAI-compatible provider has no pricing metadata, so every cost field comes back 0. That would make cost_usd=0 pi's behaviour, not our bug, and the fix would be to source cost from LiteLLM instead. NEEDS LIVE EVIDENCE from agent-1 (an events.jsonl from a real run: is message.usage.cost.total literally 0, or is the usage block shaped differently under the litellm provider?) — the ssh probe is blocked by the local permission classifier, so this needs an operator '!' handoff."} |
| comment | NULL |
| created_at | 2026-08-05T02:31:57Z |
| id | 019fcf2b-7862-7b00-86f6-5efca769ccda |
| issue_id | ah-tqc |
| event_type | closed |
| actor | Eugene Blikh |
| old_value | |
| new_value | BOTH findings resolved. Finding 1 (errored-final-turn ⇒ false success): fixed in 665e805, then COMPLETED in f11ef03 after an adversarial review disassembled the installed pi 0.82.x and found the first pass implemented only half of pi's actual contract. Errored() is now two-armed: (a) the final assistant turn's stopReason is one pi itself calls unusable — 'error' OR 'aborted', since pi's own predicate at print-mode.js:105 is 'error || aborted' and that block lives inside 'if (mode === "text")', which is precisely why json mode exits 0 on both; (b) SessionID != "" && AssistantMessages == 0, the arm the bead named but the first pass never implemented (AssistantMessages was being computed for exactly this and never consulted). Arm (b) closes an EMPIRICALLY REPRODUCED hole: print-mode.js:94 guards the whole run with 'if (initialMessage)', so a blank or whitespace-only rendered PROMPT.md — which prepareTaskDir writes verbatim, unchecked, with nothing in reconcile guarding it either — makes pi do nothing and exit 0 with a one-line stream, sending the card to In Review, publishing an empty review branch and firing the Q&A answer lane. The SessionID != "" gate is kept and heavily commented because it separates 'file present and parsed, zero turns' (evidence) from 'nothing to parse' (ignorance) — the conflation the original positive-evidence-only rationale made. finalizeFromExit's warn log gained a stream_verdict attribute so an operator can tell a provider rejection (pi's text in pi_error) from a did-nothing run. SPEC §9 and domain.Outcome's enum comment (c8083c2) both updated. Finding 2 (cost_usd=0 on every live run) was SPLIT OUT to its own bead — the parser is ruled out, and the leading hypothesis is that pi has no pricing metadata for the custom 'litellm' provider; it needs live evidence from agent-1. Review follow-ups also split out: the failure comment renders 'exit code 0' and drops the provider's rejection text; nothing rejects an empty prompt at the source. |
| comment | NULL |
| created_at | 2026-08-05T02:46:00Z |
No comments.
Close reason