~bigbes/agents-dev · parade

main · last commit 13 days ago · 7g0stsfu

← Back to the parade

ah-tqc Runner: an errored-final-turn pi run (exit 0, zero tokens) finalizes as success Past Stand

status: closed P2 bug @Eugene Blikh milestone:v0.1.0
bd reopen ah-tqc
Created byEugene Blikh
Ownerbigbes@gmail.com
Created2026-07-19T05:38:11Z
Started2026-08-04T23:24:53Z
Updated2026-08-04T23:46:00Z
Closed2026-08-04T23:46:00Z
Description
Live incident 2026-07-19 (task 4, Q&A smoke): pi --mode json made exactly one model call, the litellm proxy 403'd it (key not allowed for model 'coder'), pi recorded stopReason:error with zero usage and EXITED 0 in ~0.87s. The daemon trusts the exit code: outcome=success, card advanced to in_review, Q&A delivery then correctly reported 'no answer.json' — a broken model config masquerades as a successful run. Fix direction: Status/finalize (or ahub-run) should inspect the tail of events.jsonl — a run whose final assistant turn has stopReason:error (or whose agent_end follows zero completed tool/text turns) should finalize as outcome=error regardless of exit code. Second finding to fold in: /api/v1/status shows cost_usd=0 for ALL runs including real multi-minute ones (tasks 1-2), so the usage.cost.total accumulation from events.jsonl appears broken on pi 0.73.x — re-verify the event shape and fix the cost parse.
Notes
FINDING 1 (errored-final-turn ⇒ false success) FIXED in 665e805. Root cause confirmed as filed: applyExit mapped exit.json's code straight to the §9 outcome, and pi exits 0 even when its only assistant turn was rejected by the provider (stopReason 'error', zero usage). Fix: ParseEvents now also records the LAST assistant message_end's stopReason/errorMessage plus an assistant-message count, exposed as EventStreamInfo.Errored(); applyExit takes the parsed stream and maps exit 0 + Errored() to domain.OutcomeError while still recording the true exit code on the run row. The stream is POSITIVE evidence only — an absent, unreadable, or assistant-turn-less events.jsonl leaves the exit code's verdict alone, so a genuine success can never be flipped by a missing file. Both Status completion branches were factored into (*PiZellij).finalizeFromExit so the ah-2ef re-read and the ah-wka grace-clear can no longer drift apart between them; it WARNs with the provider's rejection text whenever it overrules a zero exit. SPEC §9 'Status resolution' updated to match. Re-verified against a REAL pi 0.82.1 capture that the pinned message_end shape (message.usage.cost.total, message.stopReason, message.errorMessage) is unchanged since the 0.70.2 fixtures.

FINDING 2 (cost_usd=0 on every live run) STILL OPEN. Ruled out: the parser. A real pi 0.82.1 run against the direct deepseek provider produces message.usage.cost.total exactly where ParseEvents reads it, and the summation is correct (see ah-1cx.4). Leading hypothesis: agent-1 runs pi through a CUSTOM 'litellm' provider (models are named litellm/<model>), and pi prices a response from its own per-provider model registry — a custom OpenAI-compatible provider has no pricing metadata, so every cost field comes back 0. That would make cost_usd=0 pi's behaviour, not our bug, and the fix would be to source cost from LiteLLM instead. NEEDS LIVE EVIDENCE from agent-1 (an events.jsonl from a real run: is message.usage.cost.total literally 0, or is the usage block shaped differently under the litellm provider?) — the ssh probe is blocked by the local permission classifier, so this needs an operator '!' handoff.

Depends on

  • ah-1cx — Hardening: post-audit bug sweep (2026-07-20) parent-child open

Depended on by

Nothing depends on this issue.

No comments.

Close reason

BOTH findings resolved. Finding 1 (errored-final-turn ⇒ false success): fixed in 665e805, then COMPLETED in f11ef03 after an adversarial review disassembled the installed pi 0.82.x and found the first pass implemented only half of pi's actual contract. Errored() is now two-armed: (a) the final assistant turn's stopReason is one pi itself calls unusable — 'error' OR 'aborted', since pi's own predicate at print-mode.js:105 is 'error || aborted' and that block lives inside 'if (mode === "text")', which is precisely why json mode exits 0 on both; (b) SessionID != "" && AssistantMessages == 0, the arm the bead named but the first pass never implemented (AssistantMessages was being computed for exactly this and never consulted). Arm (b) closes an EMPIRICALLY REPRODUCED hole: print-mode.js:94 guards the whole run with 'if (initialMessage)', so a blank or whitespace-only rendered PROMPT.md — which prepareTaskDir writes verbatim, unchecked, with nothing in reconcile guarding it either — makes pi do nothing and exit 0 with a one-line stream, sending the card to In Review, publishing an empty review branch and firing the Q&A answer lane. The SessionID != "" gate is kept and heavily commented because it separates 'file present and parsed, zero turns' (evidence) from 'nothing to parse' (ignorance) — the conflation the original positive-evidence-only rationale made. finalizeFromExit's warn log gained a stream_verdict attribute so an operator can tell a provider rejection (pi's text in pi_error) from a did-nothing run. SPEC §9 and domain.Outcome's enum comment (c8083c2) both updated. Finding 2 (cost_usd=0 on every live run) was SPLIT OUT to its own bead — the parser is ruled out, and the leading hypothesis is that pi has no pricing metadata for the custom 'litellm' provider; it needs live evidence from agent-1. Review follow-ups also split out: the failure comment renders 'exit code 0' and drops the provider's rejection text; nothing rejects an empty prompt at the source.
  • Eugene Blikh created the issue · 2026-07-19T08:38:11Z
  • Eugene Blikh added under epic ah-1cx · 2026-07-20T02:37:56Z
  • Eugene Blikh added label milestone:hardening · 2026-07-20T03:08:20Z
  • Eugene Blikh removed label milestone:hardening · 2026-07-20T03:10:16Z
  • Eugene Blikh added label milestone:v0.1.0 · 2026-07-20T03:11:24Z
  • Eugene Blikh claimed · 2026-08-05T02:24:53Z
  • Eugene Blikh updated notes to FINDING 1 (errored-final-turn ⇒ false success) FIXED in 665e805. Root cause confirmed as filed: applyExit mapped exit.json's code straight to the §9 outcome, and pi exits 0 even when its only assistant turn was rejected by the provider (stopReason 'error', zero usage). Fix: ParseEvents now also records the LAST assistant message_end's stopReason/errorMessage plus an assistant-message count, exposed as EventStreamInfo.Errored(); applyExit takes the parsed stream and maps exit 0 + Errored() to domain.OutcomeError while still recording the true exit code on the run row. The stream is POSITIVE evidence only — an absent, unreadable, or assistant-turn-less events.jsonl leaves the exit code's verdict alone, so a genuine success can never be flipped by a missing file. Both Status completion branches were factored into (*PiZellij).finalizeFromExit so the ah-2ef re-read and the ah-wka grace-clear can no longer drift apart between them; it WARNs with the provider's rejection text whenever it overrules a zero exit. SPEC §9 'Status resolution' updated to match. Re-verified against a REAL pi 0.82.1 capture that the pinned message_end shape (message.usage.cost.total, message.stopReason, message.errorMessage) is unchanged since the 0.70.2 fixtures. FINDING 2 (cost_usd=0 on every live run) STILL OPEN. Ruled out: the parser. A real pi 0.82.1 run against the direct deepseek provider produces message.usage.cost.total exactly where ParseEvents reads it, and the summation is correct (see ah-1cx.4). Leading hypothesis: agent-1 runs pi through a CUSTOM 'litellm' provider (models are named litellm/<model>), and pi prices a response from its own per-provider model registry — a custom OpenAI-compatible provider has no pricing metadata, so every cost field comes back 0. That would make cost_usd=0 pi's behaviour, not our bug, and the fix would be to source cost from LiteLLM instead. NEEDS LIVE EVIDENCE from agent-1 (an events.jsonl from a real run: is message.usage.cost.total literally 0, or is the usage block shaped differently under the litellm provider?) — the ssh probe is blocked by the local permission classifier, so this needs an operator '!' handoff. · 2026-08-05T02:31:57Z
  • Eugene Blikh closed the issue · 2026-08-05T02:46:00Z
    BOTH findings resolved. Finding 1 (errored-final-turn ⇒ false success): fixed in 665e805, then COMPLETED in f11ef03 after an adversarial review disassembled the installed pi 0.82.x and found the first pass implemented only half of pi's actual contract. Errored() is now two-armed: (a) the final assistant turn's stopReason is one pi itself calls unusable — 'error' OR 'aborted', since pi's own predicate at print-mode.js:105 is 'error || aborted' and that block lives inside 'if (mode === "text")', which is precisely why json mode exits 0 on both; (b) SessionID != "" && AssistantMessages == 0, the arm the bead named but the first pass never implemented (AssistantMessages was being computed for exactly this and never consulted). Arm (b) closes an EMPIRICALLY REPRODUCED hole: print-mode.js:94 guards the whole run with 'if (initialMessage)', so a blank or whitespace-only rendered PROMPT.md — which prepareTaskDir writes verbatim, unchecked, with nothing in reconcile guarding it either — makes pi do nothing and exit 0 with a one-line stream, sending the card to In Review, publishing an empty review branch and firing the Q&A answer lane. The SessionID != "" gate is kept and heavily commented because it separates 'file present and parsed, zero turns' (evidence) from 'nothing to parse' (ignorance) — the conflation the original positive-evidence-only rationale made. finalizeFromExit's warn log gained a stream_verdict attribute so an operator can tell a provider rejection (pi's text in pi_error) from a did-nothing run. SPEC §9 and domain.Outcome's enum comment (c8083c2) both updated. Finding 2 (cost_usd=0 on every live run) was SPLIT OUT to its own bead — the parser is ruled out, and the leading hypothesis is that pi has no pricing metadata for the custom 'litellm' provider; it needs live evidence from agent-1. Review follow-ups also split out: the failure comment renders 'exit code 0' and drops the provider's rejection text; nothing rejects an empty prompt at the source.
Stored rows — what this pane was built from, as read
issues 1 row
id ah-tqc
content_hash 0ec5f3f94c113689a7606ace2c0a5475ee68df56a367f991d8d4967233d03efe
title Runner: an errored-final-turn pi run (exit 0, zero tokens) finalizes as success
description Live incident 2026-07-19 (task 4, Q&A smoke): pi --mode json made exactly one model call, the litellm proxy 403'd it (key not allowed for model 'coder'), pi recorded stopReason:error with zero usage and EXITED 0 in ~0.87s. The daemon trusts the exit code: outcome=success, card advanced to in_review, Q&A delivery then correctly reported 'no answer.json' — a broken model config masquerades as a successful run. Fix direction: Status/finalize (or ahub-run) should inspect the tail of events.jsonl — a run whose final assistant turn has stopReason:error (or whose agent_end follows zero completed tool/text turns) should finalize as outcome=error regardless of exit code. Second finding to fold in: /api/v1/status shows cost_usd=0 for ALL runs including real multi-minute ones (tasks 1-2), so the usage.cost.total accumulation from events.jsonl appears broken on pi 0.73.x — re-verify the event shape and fix the cost parse.
design
acceptance_criteria
notes FINDING 1 (errored-final-turn ⇒ false success) FIXED in 665e805. Root cause confirmed as filed: applyExit mapped exit.json's code straight to the §9 outcome, and pi exits 0 even when its only assistant turn was rejected by the provider (stopReason 'error', zero usage). Fix: ParseEvents now also records the LAST assistant message_end's stopReason/errorMessage plus an assistant-message count, exposed as EventStreamInfo.Errored(); applyExit takes the parsed stream and maps exit 0 + Errored() to domain.OutcomeError while still recording the true exit code on the run row. The stream is POSITIVE evidence only — an absent, unreadable, or assistant-turn-less events.jsonl leaves the exit code's verdict alone, so a genuine success can never be flipped by a missing file. Both Status completion branches were factored into (*PiZellij).finalizeFromExit so the ah-2ef re-read and the ah-wka grace-clear can no longer drift apart between them; it WARNs with the provider's rejection text whenever it overrules a zero exit. SPEC §9 'Status resolution' updated to match. Re-verified against a REAL pi 0.82.1 capture that the pinned message_end shape (message.usage.cost.total, message.stopReason, message.errorMessage) is unchanged since the 0.70.2 fixtures. FINDING 2 (cost_usd=0 on every live run) STILL OPEN. Ruled out: the parser. A real pi 0.82.1 run against the direct deepseek provider produces message.usage.cost.total exactly where ParseEvents reads it, and the summation is correct (see ah-1cx.4). Leading hypothesis: agent-1 runs pi through a CUSTOM 'litellm' provider (models are named litellm/<model>), and pi prices a response from its own per-provider model registry — a custom OpenAI-compatible provider has no pricing metadata, so every cost field comes back 0. That would make cost_usd=0 pi's behaviour, not our bug, and the fix would be to source cost from LiteLLM instead. NEEDS LIVE EVIDENCE from agent-1 (an events.jsonl from a real run: is message.usage.cost.total literally 0, or is the usage block shaped differently under the litellm provider?) — the ssh probe is blocked by the local permission classifier, so this needs an operator '!' handoff.
status closed
priority 2
issue_type bug
assignee Eugene Blikh
estimated_minutes NULL
created_at 2026-07-19T05:38:11Z
created_by Eugene Blikh
owner bigbes@gmail.com
updated_at 2026-08-04T23:46:00Z
closed_at 2026-08-04T23:46:00Z
closed_by_session
external_ref NULL
spec_id
compaction_level 0
compacted_at NULL
compacted_at_commit NULL
original_size NULL
sender
ephemeral 0
wisp_type
pinned 0
is_template 0
mol_type
work_type
source_system
metadata �{}
source_repo
close_reason BOTH findings resolved. Finding 1 (errored-final-turn ⇒ false success): fixed in 665e805, then COMPLETED in f11ef03 after an adversarial review disassembled the installed pi 0.82.x and found the first pass implemented only half of pi's actual contract. Errored() is now two-armed: (a) the final assistant turn's stopReason is one pi itself calls unusable — 'error' OR 'aborted', since pi's own predicate at print-mode.js:105 is 'error || aborted' and that block lives inside 'if (mode === "text")', which is precisely why json mode exits 0 on both; (b) SessionID != "" && AssistantMessages == 0, the arm the bead named but the first pass never implemented (AssistantMessages was being computed for exactly this and never consulted). Arm (b) closes an EMPIRICALLY REPRODUCED hole: print-mode.js:94 guards the whole run with 'if (initialMessage)', so a blank or whitespace-only rendered PROMPT.md — which prepareTaskDir writes verbatim, unchecked, with nothing in reconcile guarding it either — makes pi do nothing and exit 0 with a one-line stream, sending the card to In Review, publishing an empty review branch and firing the Q&A answer lane. The SessionID != "" gate is kept and heavily commented because it separates 'file present and parsed, zero turns' (evidence) from 'nothing to parse' (ignorance) — the conflation the original positive-evidence-only rationale made. finalizeFromExit's warn log gained a stream_verdict attribute so an operator can tell a provider rejection (pi's text in pi_error) from a did-nothing run. SPEC §9 and domain.Outcome's enum comment (c8083c2) both updated. Finding 2 (cost_usd=0 on every live run) was SPLIT OUT to its own bead — the parser is ruled out, and the leading hypothesis is that pi has no pricing metadata for the custom 'litellm' provider; it needs live evidence from agent-1. Review follow-ups also split out: the failure comment renders 'exit code 0' and drops the provider's rejection text; nothing rejects an empty prompt at the source.
event_kind
actor
target
payload
await_type
await_id
timeout_ns 0
waiters
hook_bead
role_bead
agent_state
last_activity NULL
role_type
rig
due_at NULL
defer_until NULL
no_history 0
started_at 2026-08-04T23:24:53Z
is_blocked 0
labels 1 row
issue_id ah-tqc
label milestone:v0.1.0
dependencies 1 row
id 2e48a09e-41ee-516e-9c68-894bb87425a5
issue_id ah-tqc
type parent-child
created_at 2026-07-20T02:37:56Z
created_by Eugene Blikh
metadata �{}
thread_id
depends_on_issue_id ah-1cx
depends_on_wisp_id NULL
depends_on_external NULL
events 7 rows
id 019f78e1-caf4-7c77-932b-5cb2986a4897
issue_id ah-tqc
event_type created
actor Eugene Blikh
old_value
new_value
comment NULL
created_at 2026-07-19T08:38:11Z
id 019f7cda-2a1a-7e2c-a94c-8b2b6c8d721b
issue_id ah-tqc
event_type label_added
actor Eugene Blikh
old_value NULL
new_value NULL
comment Added label: milestone:hardening
created_at 2026-07-20T03:08:20Z
id 019f7cdb-f215-7933-8cae-30d938e2ca2a
issue_id ah-tqc
event_type label_removed
actor Eugene Blikh
old_value NULL
new_value NULL
comment Removed label: milestone:hardening
created_at 2026-07-20T03:10:16Z
id 019f7cdc-faec-7bc7-9566-f0de60ad90bf
issue_id ah-tqc
event_type label_added
actor Eugene Blikh
old_value NULL
new_value NULL
comment Added label: milestone:v0.1.0
created_at 2026-07-20T03:11:24Z
id 019fcf18-22d8-7dda-83c2-527acdc041f0
issue_id ah-tqc
event_type claimed
actor Eugene Blikh
old_value {"id":"ah-tqc","title":"Runner: an errored-final-turn pi run (exit 0, zero tokens) finalizes as success","description":"Live incident 2026-07-19 (task 4, Q\u0026A smoke): pi --mode json made exactly one model call, the litellm proxy 403'd it (key not allowed for model 'coder'), pi recorded stopReason:error with zero usage and EXITED 0 in ~0.87s. The daemon trusts the exit code: outcome=success, card advanced to in_review, Q\u0026A delivery then correctly reported 'no answer.json' — a broken model config masquerades as a successful run. Fix direction: Status/finalize (or ahub-run) should inspect the tail of events.jsonl — a run whose final assistant turn has stopReason:error (or whose agent_end follows zero completed tool/text turns) should finalize as outcome=error regardless of exit code. Second finding to fold in: /api/v1/status shows cost_usd=0 for ALL runs including real multi-minute ones (tasks 1-2), so the usage.cost.total accumulation from events.jsonl appears broken on pi 0.73.x — re-verify the event shape and fix the cost parse.","status":"open","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-07-19T05:38:11Z","created_by":"Eugene Blikh","updated_at":"2026-07-19T05:38:11Z","labels":["milestone:v0.1.0"]}
new_value {"assignee":"Eugene Blikh","status":"in_progress"}
comment NULL
created_at 2026-08-05T02:24:53Z
id 019fcf1e-9c53-7036-9f0d-4a4bfa72b31a
issue_id ah-tqc
event_type updated
actor Eugene Blikh
old_value {"id":"ah-tqc","title":"Runner: an errored-final-turn pi run (exit 0, zero tokens) finalizes as success","description":"Live incident 2026-07-19 (task 4, Q\u0026A smoke): pi --mode json made exactly one model call, the litellm proxy 403'd it (key not allowed for model 'coder'), pi recorded stopReason:error with zero usage and EXITED 0 in ~0.87s. The daemon trusts the exit code: outcome=success, card advanced to in_review, Q\u0026A delivery then correctly reported 'no answer.json' — a broken model config masquerades as a successful run. Fix direction: Status/finalize (or ahub-run) should inspect the tail of events.jsonl — a run whose final assistant turn has stopReason:error (or whose agent_end follows zero completed tool/text turns) should finalize as outcome=error regardless of exit code. Second finding to fold in: /api/v1/status shows cost_usd=0 for ALL runs including real multi-minute ones (tasks 1-2), so the usage.cost.total accumulation from events.jsonl appears broken on pi 0.73.x — re-verify the event shape and fix the cost parse.","status":"in_progress","priority":2,"issue_type":"bug","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-07-19T05:38:11Z","created_by":"Eugene Blikh","updated_at":"2026-08-04T23:24:53Z","started_at":"2026-08-04T23:24:53Z","labels":["milestone:v0.1.0"]}
new_value {"notes":"FINDING 1 (errored-final-turn ⇒ false success) FIXED in 665e805. Root cause confirmed as filed: applyExit mapped exit.json's code straight to the §9 outcome, and pi exits 0 even when its only assistant turn was rejected by the provider (stopReason 'error', zero usage). Fix: ParseEvents now also records the LAST assistant message_end's stopReason/errorMessage plus an assistant-message count, exposed as EventStreamInfo.Errored(); applyExit takes the parsed stream and maps exit 0 + Errored() to domain.OutcomeError while still recording the true exit code on the run row. The stream is POSITIVE evidence only — an absent, unreadable, or assistant-turn-less events.jsonl leaves the exit code's verdict alone, so a genuine success can never be flipped by a missing file. Both Status completion branches were factored into (*PiZellij).finalizeFromExit so the ah-2ef re-read and the ah-wka grace-clear can no longer drift apart between them; it WARNs with the provider's rejection text whenever it overrules a zero exit. SPEC §9 'Status resolution' updated to match. Re-verified against a REAL pi 0.82.1 capture that the pinned message_end shape (message.usage.cost.total, message.stopReason, message.errorMessage) is unchanged since the 0.70.2 fixtures.\n\nFINDING 2 (cost_usd=0 on every live run) STILL OPEN. Ruled out: the parser. A real pi 0.82.1 run against the direct deepseek provider produces message.usage.cost.total exactly where ParseEvents reads it, and the summation is correct (see ah-1cx.4). Leading hypothesis: agent-1 runs pi through a CUSTOM 'litellm' provider (models are named litellm/\u003cmodel\u003e), and pi prices a response from its own per-provider model registry — a custom OpenAI-compatible provider has no pricing metadata, so every cost field comes back 0. That would make cost_usd=0 pi's behaviour, not our bug, and the fix would be to source cost from LiteLLM instead. NEEDS LIVE EVIDENCE from agent-1 (an events.jsonl from a real run: is message.usage.cost.total literally 0, or is the usage block shaped differently under the litellm provider?) — the ssh probe is blocked by the local permission classifier, so this needs an operator '!' handoff."}
comment NULL
created_at 2026-08-05T02:31:57Z
id 019fcf2b-7862-7b00-86f6-5efca769ccda
issue_id ah-tqc
event_type closed
actor Eugene Blikh
old_value
new_value BOTH findings resolved. Finding 1 (errored-final-turn ⇒ false success): fixed in 665e805, then COMPLETED in f11ef03 after an adversarial review disassembled the installed pi 0.82.x and found the first pass implemented only half of pi's actual contract. Errored() is now two-armed: (a) the final assistant turn's stopReason is one pi itself calls unusable — 'error' OR 'aborted', since pi's own predicate at print-mode.js:105 is 'error || aborted' and that block lives inside 'if (mode === "text")', which is precisely why json mode exits 0 on both; (b) SessionID != "" && AssistantMessages == 0, the arm the bead named but the first pass never implemented (AssistantMessages was being computed for exactly this and never consulted). Arm (b) closes an EMPIRICALLY REPRODUCED hole: print-mode.js:94 guards the whole run with 'if (initialMessage)', so a blank or whitespace-only rendered PROMPT.md — which prepareTaskDir writes verbatim, unchecked, with nothing in reconcile guarding it either — makes pi do nothing and exit 0 with a one-line stream, sending the card to In Review, publishing an empty review branch and firing the Q&A answer lane. The SessionID != "" gate is kept and heavily commented because it separates 'file present and parsed, zero turns' (evidence) from 'nothing to parse' (ignorance) — the conflation the original positive-evidence-only rationale made. finalizeFromExit's warn log gained a stream_verdict attribute so an operator can tell a provider rejection (pi's text in pi_error) from a did-nothing run. SPEC §9 and domain.Outcome's enum comment (c8083c2) both updated. Finding 2 (cost_usd=0 on every live run) was SPLIT OUT to its own bead — the parser is ruled out, and the leading hypothesis is that pi has no pricing metadata for the custom 'litellm' provider; it needs live evidence from agent-1. Review follow-ups also split out: the failure comment renders 'exit code 0' and drops the provider's rejection text; nothing rejects an empty prompt at the source.
comment NULL
created_at 2026-08-05T02:46:00Z