main · last commit
13 days ago ·
7g0stsfu
ah-1cx.8 [bug] cost_usd is 0 for every live run — pi prices from its own registry, the litellm provider has none
Lined Up
bd update ah-1cx.8 --claim
bd close ah-1cx.8
| Created by | Eugene Blikh |
| Owner | bigbes@gmail.com |
| Created | 2026-08-04T23:32:21Z |
| Updated | 2026-08-04T23:55:28Z |
Split out of ah-tqc's second finding so that bead can close on its first finding. /api/v1/status reports cost_usd=0 for ALL runs on agent-1, including real multi-minute ones (observed on tasks 1-2 during the 2026-07-19 Q&A smoke). The run row's CostUSD comes from ParseEvents summing message.usage.cost.total over assistant message_end events. RULED OUT: the parser. A real pi 0.82.1 capture against the DIRECT deepseek provider (deepseek/deepseek-v4-flash) puts cost.total exactly where ParseEvents reads it, and the summation is arithmetically correct (proven per-message, not cumulative — see ah-1cx.4). LEADING HYPOTHESIS: pi computes cost itself from a per-provider model pricing registry. agent-1 runs every role through a CUSTOM OpenAI-compatible provider named 'litellm' (role models are 'litellm/<model>'), and a custom provider carries no pricing metadata, so pi emits cost{input:0,output:0,total:0} while token counts are still real. If that holds, nothing in agent-hub is broken and the fix is to stop trusting pi for cost: either derive it from token counts against a locally configured price table, or read spend from the LiteLLM proxy (it tracks per-key spend), or drop the field from the status API rather than reporting a confident zero.
Settle the hypothesis BEFORE writing code. Needed evidence, from agent-1 (the local ssh probe is blocked by the permission classifier — needs an operator '!' handoff): 1. A real events.jsonl from a completed run: jq -c 'select(.type=="message_end" and .message.role=="assistant") | .message.usage' over the archived stream. Are totalTokens non-zero while every cost.* is 0? That confirms the hypothesis outright. 2. The pi provider config on the box (how the 'litellm' provider is declared) — does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is config-only, on the box, with no code change at all. Only if both come back negative does this become a parser/shape bug.
Either (a) pi is configured/patched so cost.total is real for the litellm provider and a live run reports a non-zero cost_usd, or (b) cost is sourced from somewhere trustworthy, or (c) the field is explicitly documented+surfaced as unavailable rather than a silent 0. In all three cases SPEC and the status API description must match what is actually reported.
OPERATOR HANDOFF — the ssh probe from this dev session is blocked by the local permission classifier, so these need a '!' handoff or a human at a terminal. Run on agent-1 (or via ssh from the mac): 1. Find a completed run's event stream and look at what pi actually reported: ssh agent-1.lab.internal sudo -u agenthub bash -lc 'ls -t /opt/agent-hub/work/*/task-*/.task/events.jsonl 2>/dev/null | head -3' sudo -u agenthub bash -lc 'jq -c "select(.type==\"message_end\" and .message.role==\"assistant\") | .message.usage" <PATH>' The decisive question: are input/output/totalTokens NON-ZERO while every cost.* is 0? That confirms the hypothesis outright — pi is reporting real token counts and no price, i.e. it has no pricing metadata for the custom 'litellm' provider. (If the worktrees are already pruned, the archive lane keeps tarballs — check archive_dir, default <work_root>/archive, for task-<id>.tar.gz.) 2. Look at how the litellm provider is declared in pi's config on the box: sudo -u agenthub bash -lc 'ls ~/.pi; grep -rn litellm ~/.pi 2>/dev/null | head' Does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is CONFIG-ONLY, on the box, with no code change in agents-dev at all — and this bead closes as 'not our bug, fixed in pi config'. 3. Cross-check what LiteLLM itself thinks the spend was, since it tracks per-virtual-key spend. If pi cannot be taught the prices, LiteLLM is the trustworthy source and the fix becomes 'read cost from LiteLLM, not from the event stream'. Only if step 1 comes back with cost fields that are populated-but-mis-shaped does this become a parser bug — and in that case capture the events.jsonl and add it as a testdata fixture, the way ah-1cx.4 did for the 0.82.1 shape. For reference, a REAL pi 0.82.1 capture against the DIRECT deepseek provider (local mac, 2026-08-05) has cost exactly where ParseEvents reads it: message.usage.cost = {input, output, cacheRead, cacheWrite, total}, all non-zero, and the arithmetic checks out against flat per-Mtok rates. So the parser is not the problem on a provider pi has prices for.
ah-1cx
— Hardening: post-audit bug sweep (2026-07-20)
parent-child
open
Nothing depends on this issue.
| id | ah-1cx.8 |
| content_hash | f9af778a32a0fc2ac9d0d9bf63b5430905203491ae53324cebc0c91faecff187 |
| title | [bug] cost_usd is 0 for every live run — pi prices from its own registry, the litellm provider has none |
| description | Split out of ah-tqc's second finding so that bead can close on its first finding. /api/v1/status reports cost_usd=0 for ALL runs on agent-1, including real multi-minute ones (observed on tasks 1-2 during the 2026-07-19 Q&A smoke). The run row's CostUSD comes from ParseEvents summing message.usage.cost.total over assistant message_end events. RULED OUT: the parser. A real pi 0.82.1 capture against the DIRECT deepseek provider (deepseek/deepseek-v4-flash) puts cost.total exactly where ParseEvents reads it, and the summation is arithmetically correct (proven per-message, not cumulative — see ah-1cx.4). LEADING HYPOTHESIS: pi computes cost itself from a per-provider model pricing registry. agent-1 runs every role through a CUSTOM OpenAI-compatible provider named 'litellm' (role models are 'litellm/<model>'), and a custom provider carries no pricing metadata, so pi emits cost{input:0,output:0,total:0} while token counts are still real. If that holds, nothing in agent-hub is broken and the fix is to stop trusting pi for cost: either derive it from token counts against a locally configured price table, or read spend from the LiteLLM proxy (it tracks per-key spend), or drop the field from the status API rather than reporting a confident zero. |
| design | Settle the hypothesis BEFORE writing code. Needed evidence, from agent-1 (the local ssh probe is blocked by the permission classifier — needs an operator '!' handoff): 1. A real events.jsonl from a completed run: jq -c 'select(.type=="message_end" and .message.role=="assistant") | .message.usage' over the archived stream. Are totalTokens non-zero while every cost.* is 0? That confirms the hypothesis outright. 2. The pi provider config on the box (how the 'litellm' provider is declared) — does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is config-only, on the box, with no code change at all. Only if both come back negative does this become a parser/shape bug. |
| acceptance_criteria | Either (a) pi is configured/patched so cost.total is real for the litellm provider and a live run reports a non-zero cost_usd, or (b) cost is sourced from somewhere trustworthy, or (c) the field is explicitly documented+surfaced as unavailable rather than a silent 0. In all three cases SPEC and the status API description must match what is actually reported. |
| notes | OPERATOR HANDOFF — the ssh probe from this dev session is blocked by the local permission classifier, so these need a '!' handoff or a human at a terminal. Run on agent-1 (or via ssh from the mac): 1. Find a completed run's event stream and look at what pi actually reported: ssh agent-1.lab.internal sudo -u agenthub bash -lc 'ls -t /opt/agent-hub/work/*/task-*/.task/events.jsonl 2>/dev/null | head -3' sudo -u agenthub bash -lc 'jq -c "select(.type==\"message_end\" and .message.role==\"assistant\") | .message.usage" <PATH>' The decisive question: are input/output/totalTokens NON-ZERO while every cost.* is 0? That confirms the hypothesis outright — pi is reporting real token counts and no price, i.e. it has no pricing metadata for the custom 'litellm' provider. (If the worktrees are already pruned, the archive lane keeps tarballs — check archive_dir, default <work_root>/archive, for task-<id>.tar.gz.) 2. Look at how the litellm provider is declared in pi's config on the box: sudo -u agenthub bash -lc 'ls ~/.pi; grep -rn litellm ~/.pi 2>/dev/null | head' Does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is CONFIG-ONLY, on the box, with no code change in agents-dev at all — and this bead closes as 'not our bug, fixed in pi config'. 3. Cross-check what LiteLLM itself thinks the spend was, since it tracks per-virtual-key spend. If pi cannot be taught the prices, LiteLLM is the trustworthy source and the fix becomes 'read cost from LiteLLM, not from the event stream'. Only if step 1 comes back with cost fields that are populated-but-mis-shaped does this become a parser bug — and in that case capture the events.jsonl and add it as a testdata fixture, the way ah-1cx.4 did for the 0.82.1 shape. For reference, a REAL pi 0.82.1 capture against the DIRECT deepseek provider (local mac, 2026-08-05) has cost exactly where ParseEvents reads it: message.usage.cost = {input, output, cacheRead, cacheWrite, total}, all non-zero, and the arithmetic checks out against flat per-Mtok rates. So the parser is not the problem on a provider pi has prices for. |
| status | open |
| priority | 3 |
| issue_type | bug |
| assignee | NULL |
| estimated_minutes | NULL |
| created_at | 2026-08-04T23:32:21Z |
| created_by | Eugene Blikh |
| owner | bigbes@gmail.com |
| updated_at | 2026-08-04T23:55:28Z |
| closed_at | NULL |
| closed_by_session | |
| external_ref | NULL |
| spec_id | |
| compaction_level | 0 |
| compacted_at | NULL |
| compacted_at_commit | NULL |
| original_size | NULL |
| sender | |
| ephemeral | 0 |
| wisp_type | |
| pinned | 0 |
| is_template | 0 |
| mol_type | |
| work_type | |
| source_system | |
| metadata | �{} |
| source_repo | |
| close_reason | |
| event_kind | |
| actor | |
| target | |
| payload | |
| await_type | |
| await_id | |
| timeout_ns | 0 |
| waiters | |
| hook_bead | |
| role_bead | |
| agent_state | |
| last_activity | NULL |
| role_type | |
| rig | |
| due_at | NULL |
| defer_until | NULL |
| no_history | 0 |
| started_at | NULL |
| is_blocked | 0 |
| issue_id | ah-1cx.8 |
| label | milestone:v0.1.0 |
| id | e50673a4-3c25-53b6-bd66-8c89812cca1a |
| issue_id | ah-1cx.8 |
| type | parent-child |
| created_at | 2026-08-05T02:32:20Z |
| created_by | Eugene Blikh |
| metadata | �{} |
| thread_id | |
| depends_on_issue_id | ah-1cx |
| depends_on_wisp_id | NULL |
| depends_on_external | NULL |
| id | 019fcf1e-f613-790c-8f99-35fe20082165 |
| issue_id | ah-1cx.8 |
| event_type | created |
| actor | Eugene Blikh |
| old_value | |
| new_value | |
| comment | NULL |
| created_at | 2026-08-05T02:32:20Z |
| id | 019fcf1f-4d39-7ee8-b1e9-4a14bb842846 |
| issue_id | ah-1cx.8 |
| event_type | label_added |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Added label: milestone:v0.1.0 |
| created_at | 2026-08-05T02:32:42Z |
| id | 019fcf34-1ff9-72f9-8ade-890847c38d63 |
| issue_id | ah-1cx.8 |
| event_type | updated |
| actor | Eugene Blikh |
| old_value | {"id":"ah-1cx.8","title":"[bug] cost_usd is 0 for every live run — pi prices from its own registry, the litellm provider has none","description":"Split out of ah-tqc's second finding so that bead can close on its first finding.\n\n/api/v1/status reports cost_usd=0 for ALL runs on agent-1, including real multi-minute ones (observed on tasks 1-2 during the 2026-07-19 Q\u0026A smoke). The run row's CostUSD comes from ParseEvents summing message.usage.cost.total over assistant message_end events.\n\nRULED OUT: the parser. A real pi 0.82.1 capture against the DIRECT deepseek provider (deepseek/deepseek-v4-flash) puts cost.total exactly where ParseEvents reads it, and the summation is arithmetically correct (proven per-message, not cumulative — see ah-1cx.4).\n\nLEADING HYPOTHESIS: pi computes cost itself from a per-provider model pricing registry. agent-1 runs every role through a CUSTOM OpenAI-compatible provider named 'litellm' (role models are 'litellm/\u003cmodel\u003e'), and a custom provider carries no pricing metadata, so pi emits cost{input:0,output:0,total:0} while token counts are still real. If that holds, nothing in agent-hub is broken and the fix is to stop trusting pi for cost: either derive it from token counts against a locally configured price table, or read spend from the LiteLLM proxy (it tracks per-key spend), or drop the field from the status API rather than reporting a confident zero.","design":"Settle the hypothesis BEFORE writing code. Needed evidence, from agent-1 (the local ssh probe is blocked by the permission classifier — needs an operator '!' handoff):\n\n1. A real events.jsonl from a completed run: jq -c 'select(.type==\"message_end\" and .message.role==\"assistant\") | .message.usage' over the archived stream. Are totalTokens non-zero while every cost.* is 0? That confirms the hypothesis outright.\n2. The pi provider config on the box (how the 'litellm' provider is declared) — does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is config-only, on the box, with no code change at all.\n\nOnly if both come back negative does this become a parser/shape bug.","acceptance_criteria":"Either (a) pi is configured/patched so cost.total is real for the litellm provider and a live run reports a non-zero cost_usd, or (b) cost is sourced from somewhere trustworthy, or (c) the field is explicitly documented+surfaced as unavailable rather than a silent 0. In all three cases SPEC and the status API description must match what is actually reported.","status":"open","priority":3,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-08-04T23:32:21Z","created_by":"Eugene Blikh","updated_at":"2026-08-04T23:32:21Z","labels":["milestone:v0.1.0"]} |
| new_value | {"notes":"OPERATOR HANDOFF — the ssh probe from this dev session is blocked by the local permission classifier, so these need a '!' handoff or a human at a terminal. Run on agent-1 (or via ssh from the mac):\n\n1. Find a completed run's event stream and look at what pi actually reported:\n ssh agent-1.lab.internal\n sudo -u agenthub bash -lc 'ls -t /opt/agent-hub/work/*/task-*/.task/events.jsonl 2\u003e/dev/null | head -3'\n sudo -u agenthub bash -lc 'jq -c \"select(.type==\\\"message_end\\\" and .message.role==\\\"assistant\\\") | .message.usage\" \u003cPATH\u003e'\n The decisive question: are input/output/totalTokens NON-ZERO while every cost.* is 0? That confirms the hypothesis outright — pi is reporting real token counts and no price, i.e. it has no pricing metadata for the custom 'litellm' provider.\n (If the worktrees are already pruned, the archive lane keeps tarballs — check archive_dir, default \u003cwork_root\u003e/archive, for task-\u003cid\u003e.tar.gz.)\n\n2. Look at how the litellm provider is declared in pi's config on the box:\n sudo -u agenthub bash -lc 'ls ~/.pi; grep -rn litellm ~/.pi 2\u003e/dev/null | head'\n Does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is CONFIG-ONLY, on the box, with no code change in agents-dev at all — and this bead closes as 'not our bug, fixed in pi config'.\n\n3. Cross-check what LiteLLM itself thinks the spend was, since it tracks per-virtual-key spend. If pi cannot be taught the prices, LiteLLM is the trustworthy source and the fix becomes 'read cost from LiteLLM, not from the event stream'.\n\nOnly if step 1 comes back with cost fields that are populated-but-mis-shaped does this become a parser bug — and in that case capture the events.jsonl and add it as a testdata fixture, the way ah-1cx.4 did for the 0.82.1 shape.\n\nFor reference, a REAL pi 0.82.1 capture against the DIRECT deepseek provider (local mac, 2026-08-05) has cost exactly where ParseEvents reads it: message.usage.cost = {input, output, cacheRead, cacheWrite, total}, all non-zero, and the arithmetic checks out against flat per-Mtok rates. So the parser is not the problem on a provider pi has prices for."} |
| comment | NULL |
| created_at | 2026-08-05T02:55:27Z |
No comments.