main · last commit
13 days ago ·
7g0stsfu
ah-1cx.4 runner: verify F9 cost model — is pi message cost.total per-message or cumulative?
Past Stand
bd reopen ah-1cx.4
| Created by | Eugene Blikh |
| Owner | bigbes@gmail.com |
| Created | 2026-07-19T23:37:39Z |
| Started | 2026-08-04T23:32:44Z |
| Updated | 2026-08-04T23:46:00Z |
| Closed | 2026-08-04T23:46:00Z |
costFrom/ParseEvents SUM message.usage.cost.total across all assistant message_end events (events.go:120-128), but no captured multi-message fixture proves the field is per-message; testdata/pi-events-two-messages.jsonl is hand-built (TODO from closed ah-nyl.11). If cost.total is actually cumulative, every multi-turn run's reported cost is inflated and both costFrom and the fixture must be re-pinned to take the last message's value. Capture a real multi-assistant-message pi --mode json run and settle it. Distinct from sibling ah-tqc (errored-turn-as-success + cost_usd parse); marker-sweep audit 2026-07-20.
ah-1cx
— Hardening: post-audit bug sweep (2026-07-20)
parent-child
open
Nothing depends on this issue.
| id | ah-1cx.4 |
| content_hash | 4cc6b8f0ddcf69ba72aad626c3b1b374102663f9bf71080ea5d8e6a01106fca9 |
| title | runner: verify F9 cost model — is pi message cost.total per-message or cumulative? |
| description | costFrom/ParseEvents SUM message.usage.cost.total across all assistant message_end events (events.go:120-128), but no captured multi-message fixture proves the field is per-message; testdata/pi-events-two-messages.jsonl is hand-built (TODO from closed ah-nyl.11). If cost.total is actually cumulative, every multi-turn run's reported cost is inflated and both costFrom and the fixture must be re-pinned to take the last message's value. Capture a real multi-assistant-message pi --mode json run and settle it. Distinct from sibling ah-tqc (errored-turn-as-success + cost_usd parse); marker-sweep audit 2026-07-20. |
| design | |
| acceptance_criteria | |
| notes | |
| status | closed |
| priority | 3 |
| issue_type | bug |
| assignee | NULL |
| estimated_minutes | NULL |
| created_at | 2026-07-19T23:37:39Z |
| created_by | Eugene Blikh |
| owner | bigbes@gmail.com |
| updated_at | 2026-08-04T23:46:00Z |
| closed_at | 2026-08-04T23:46:00Z |
| closed_by_session | |
| external_ref | NULL |
| spec_id | |
| compaction_level | 0 |
| compacted_at | NULL |
| compacted_at_commit | NULL |
| original_size | NULL |
| sender | |
| ephemeral | 0 |
| wisp_type | |
| pinned | 0 |
| is_template | 0 |
| mol_type | |
| work_type | |
| source_system | |
| metadata | �{} |
| source_repo | |
| close_reason | SETTLED in f11ef03: message.usage.cost.total is PER-MESSAGE, not cumulative, so ParseEvents's summation is correct. TODO(ah-nyl.11) removed from both costFrom's doc comment and TestParseEventsTwoAssistantMessagesSumsPerMessageCost. Evidence is a REAL two-assistant-message, one-tool-call run captured from pi 0.82.1 (deepseek/deepseek-v4-flash, exit 0), committed as testdata/pi-events-multi-real.jsonl; the 123 message_update lines (95% of the raw 166 KB, ignored by the parser) were stripped and the trimmed 20-line fixture verified to parse byte-for-byte identically to the raw capture. The proof recorded in costFrom is ARITHMETIC rather than assertive: one flat set of unit rates ($0.14 / $0.28 / $0.0028 per Mtok for input / output / cacheRead) explains each message's cost from that message's OWN token counts alone — msg1 97*1.4e-7=0.00001358 and msg2 85*1.4e-7=0.0000119; msg1 70*2.8e-7=0.0000196 and msg2 79*2.8e-7=0.00002212 — so msg2's total contains no part of msg1's, which a cumulative field could not manage. TestParseEventsRealMultiMessageCapture pins session id, CostUSD == 0.0000374808+0.0000386792, AssistantMessages == 2, FinalStopReason == 'stop' (the first turn ended in toolUse). The hand-built pi-events-two-messages.jsonl and its test are KEPT but re-framed as the pi 0.70.2 shape pin with round numbers. Two incidental findings now documented in the events.go header: turn_end repeats its turn's usage verbatim (so the not-double-counted rule is right), and 0.82.1 adds a trailing agent_settled event plus assistant message_start events that carry a stopReason and a zeroed usage block — the latter would skew AssistantMessages/FinalStopReason if the message_end type check were ever loosened. |
| event_kind | |
| actor | |
| target | |
| payload | |
| await_type | |
| await_id | |
| timeout_ns | 0 |
| waiters | |
| hook_bead | |
| role_bead | |
| agent_state | |
| last_activity | NULL |
| role_type | |
| rig | |
| due_at | NULL |
| defer_until | NULL |
| no_history | 0 |
| started_at | 2026-08-04T23:32:44Z |
| is_blocked | 0 |
| issue_id | ah-1cx.4 |
| label | milestone:v0.1.0 |
| id | 2560dba7-7e91-58ca-b298-c4c2546377ba |
| issue_id | ah-1cx.4 |
| type | parent-child |
| created_at | 2026-07-20T02:37:39Z |
| created_by | Eugene Blikh |
| metadata | �{} |
| thread_id | |
| depends_on_issue_id | ah-1cx |
| depends_on_wisp_id | NULL |
| depends_on_external | NULL |
| id | 019f7cbe-12dc-77f7-ba0a-4b9a1e0ce934 |
| issue_id | ah-1cx.4 |
| event_type | created |
| actor | Eugene Blikh |
| old_value | |
| new_value | |
| comment | NULL |
| created_at | 2026-07-20T02:37:39Z |
| id | 019f7cda-2a1b-7829-95c9-cc8f441a7e81 |
| issue_id | ah-1cx.4 |
| event_type | label_added |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Added label: milestone:hardening |
| created_at | 2026-07-20T03:08:20Z |
| id | 019f7cdb-ee09-78cd-b47a-9be030625646 |
| issue_id | ah-1cx.4 |
| event_type | label_removed |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Removed label: milestone:hardening |
| created_at | 2026-07-20T03:10:15Z |
| id | 019f7cdc-faea-7395-aa29-adb82f4d563a |
| issue_id | ah-1cx.4 |
| event_type | label_added |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Added label: milestone:v0.1.0 |
| created_at | 2026-07-20T03:11:24Z |
| id | 019fcf1f-5070-7036-98e4-42f415b30492 |
| issue_id | ah-1cx.4 |
| event_type | status_changed |
| actor | Eugene Blikh |
| old_value | {"id":"ah-1cx.4","title":"runner: verify F9 cost model — is pi message cost.total per-message or cumulative?","description":"costFrom/ParseEvents SUM message.usage.cost.total across all assistant message_end events (events.go:120-128), but no captured multi-message fixture proves the field is per-message; testdata/pi-events-two-messages.jsonl is hand-built (TODO from closed ah-nyl.11). If cost.total is actually cumulative, every multi-turn run's reported cost is inflated and both costFrom and the fixture must be re-pinned to take the last message's value. Capture a real multi-assistant-message pi --mode json run and settle it. Distinct from sibling ah-tqc (errored-turn-as-success + cost_usd parse); marker-sweep audit 2026-07-20.","status":"open","priority":3,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-07-19T23:37:39Z","created_by":"Eugene Blikh","updated_at":"2026-07-19T23:37:39Z","labels":["milestone:v0.1.0"]} |
| new_value | {"status":"in_progress"} |
| comment | NULL |
| created_at | 2026-08-05T02:32:43Z |
| id | 019fcf2b-766e-7d52-b56c-ab27e2e3af7c |
| issue_id | ah-1cx.4 |
| event_type | closed |
| actor | Eugene Blikh |
| old_value | |
| new_value | SETTLED in f11ef03: message.usage.cost.total is PER-MESSAGE, not cumulative, so ParseEvents's summation is correct. TODO(ah-nyl.11) removed from both costFrom's doc comment and TestParseEventsTwoAssistantMessagesSumsPerMessageCost. Evidence is a REAL two-assistant-message, one-tool-call run captured from pi 0.82.1 (deepseek/deepseek-v4-flash, exit 0), committed as testdata/pi-events-multi-real.jsonl; the 123 message_update lines (95% of the raw 166 KB, ignored by the parser) were stripped and the trimmed 20-line fixture verified to parse byte-for-byte identically to the raw capture. The proof recorded in costFrom is ARITHMETIC rather than assertive: one flat set of unit rates ($0.14 / $0.28 / $0.0028 per Mtok for input / output / cacheRead) explains each message's cost from that message's OWN token counts alone — msg1 97*1.4e-7=0.00001358 and msg2 85*1.4e-7=0.0000119; msg1 70*2.8e-7=0.0000196 and msg2 79*2.8e-7=0.00002212 — so msg2's total contains no part of msg1's, which a cumulative field could not manage. TestParseEventsRealMultiMessageCapture pins session id, CostUSD == 0.0000374808+0.0000386792, AssistantMessages == 2, FinalStopReason == 'stop' (the first turn ended in toolUse). The hand-built pi-events-two-messages.jsonl and its test are KEPT but re-framed as the pi 0.70.2 shape pin with round numbers. Two incidental findings now documented in the events.go header: turn_end repeats its turn's usage verbatim (so the not-double-counted rule is right), and 0.82.1 adds a trailing agent_settled event plus assistant message_start events that carry a stopReason and a zeroed usage block — the latter would skew AssistantMessages/FinalStopReason if the message_end type check were ever loosened. |
| comment | NULL |
| created_at | 2026-08-05T02:45:59Z |
No comments.
Close reason