~bigbes/agents-dev · parade

main · last commit 13 days ago · 7g0stsfu

← Back to the parade

ah-1cx.8 [bug] cost_usd is 0 for every live run — pi prices from its own registry, the litellm provider has none Lined Up

status: open P3 bug milestone:v0.1.0
bd update ah-1cx.8 --claim bd close ah-1cx.8
Created byEugene Blikh
Ownerbigbes@gmail.com
Created2026-08-04T23:32:21Z
Updated2026-08-04T23:55:28Z
Description
Split out of ah-tqc's second finding so that bead can close on its first finding.

/api/v1/status reports cost_usd=0 for ALL runs on agent-1, including real multi-minute ones (observed on tasks 1-2 during the 2026-07-19 Q&A smoke). The run row's CostUSD comes from ParseEvents summing message.usage.cost.total over assistant message_end events.

RULED OUT: the parser. A real pi 0.82.1 capture against the DIRECT deepseek provider (deepseek/deepseek-v4-flash) puts cost.total exactly where ParseEvents reads it, and the summation is arithmetically correct (proven per-message, not cumulative — see ah-1cx.4).

LEADING HYPOTHESIS: pi computes cost itself from a per-provider model pricing registry. agent-1 runs every role through a CUSTOM OpenAI-compatible provider named 'litellm' (role models are 'litellm/<model>'), and a custom provider carries no pricing metadata, so pi emits cost{input:0,output:0,total:0} while token counts are still real. If that holds, nothing in agent-hub is broken and the fix is to stop trusting pi for cost: either derive it from token counts against a locally configured price table, or read spend from the LiteLLM proxy (it tracks per-key spend), or drop the field from the status API rather than reporting a confident zero.
Design
Settle the hypothesis BEFORE writing code. Needed evidence, from agent-1 (the local ssh probe is blocked by the permission classifier — needs an operator '!' handoff):

1. A real events.jsonl from a completed run: jq -c 'select(.type=="message_end" and .message.role=="assistant") | .message.usage' over the archived stream. Are totalTokens non-zero while every cost.* is 0? That confirms the hypothesis outright.
2. The pi provider config on the box (how the 'litellm' provider is declared) — does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is config-only, on the box, with no code change at all.

Only if both come back negative does this become a parser/shape bug.
Acceptance criteria
Either (a) pi is configured/patched so cost.total is real for the litellm provider and a live run reports a non-zero cost_usd, or (b) cost is sourced from somewhere trustworthy, or (c) the field is explicitly documented+surfaced as unavailable rather than a silent 0. In all three cases SPEC and the status API description must match what is actually reported.
Notes
OPERATOR HANDOFF — the ssh probe from this dev session is blocked by the local permission classifier, so these need a '!' handoff or a human at a terminal. Run on agent-1 (or via ssh from the mac):

1. Find a completed run's event stream and look at what pi actually reported:
   ssh agent-1.lab.internal
   sudo -u agenthub bash -lc 'ls -t /opt/agent-hub/work/*/task-*/.task/events.jsonl 2>/dev/null | head -3'
   sudo -u agenthub bash -lc 'jq -c "select(.type==\"message_end\" and .message.role==\"assistant\") | .message.usage" <PATH>'
   The decisive question: are input/output/totalTokens NON-ZERO while every cost.* is 0? That confirms the hypothesis outright — pi is reporting real token counts and no price, i.e. it has no pricing metadata for the custom 'litellm' provider.
   (If the worktrees are already pruned, the archive lane keeps tarballs — check archive_dir, default <work_root>/archive, for task-<id>.tar.gz.)

2. Look at how the litellm provider is declared in pi's config on the box:
   sudo -u agenthub bash -lc 'ls ~/.pi; grep -rn litellm ~/.pi 2>/dev/null | head'
   Does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is CONFIG-ONLY, on the box, with no code change in agents-dev at all — and this bead closes as 'not our bug, fixed in pi config'.

3. Cross-check what LiteLLM itself thinks the spend was, since it tracks per-virtual-key spend. If pi cannot be taught the prices, LiteLLM is the trustworthy source and the fix becomes 'read cost from LiteLLM, not from the event stream'.

Only if step 1 comes back with cost fields that are populated-but-mis-shaped does this become a parser bug — and in that case capture the events.jsonl and add it as a testdata fixture, the way ah-1cx.4 did for the 0.82.1 shape.

For reference, a REAL pi 0.82.1 capture against the DIRECT deepseek provider (local mac, 2026-08-05) has cost exactly where ParseEvents reads it: message.usage.cost = {input, output, cacheRead, cacheWrite, total}, all non-zero, and the arithmetic checks out against flat per-Mtok rates. So the parser is not the problem on a provider pi has prices for.

Depends on

  • ah-1cx — Hardening: post-audit bug sweep (2026-07-20) parent-child open

Depended on by

Nothing depends on this issue.

No comments.

  • Eugene Blikh added under epic ah-1cx · 2026-08-05T02:32:20Z
  • Eugene Blikh created the issue · 2026-08-05T02:32:20Z
  • Eugene Blikh added label milestone:v0.1.0 · 2026-08-05T02:32:42Z
  • Eugene Blikh updated notes to OPERATOR HANDOFF — the ssh probe from this dev session is blocked by the local permission classifier, so these need a '!' handoff or a human at a terminal. Run on agent-1 (or via ssh from the mac): 1. Find a completed run's event stream and look at what pi actually reported: ssh agent-1.lab.internal sudo -u agenthub bash -lc 'ls -t /opt/agent-hub/work/*/task-*/.task/events.jsonl 2>/dev/null | head -3' sudo -u agenthub bash -lc 'jq -c "select(.type==\"message_end\" and .message.role==\"assistant\") | .message.usage" <PATH>' The decisive question: are input/output/totalTokens NON-ZERO while every cost.* is 0? That confirms the hypothesis outright — pi is reporting real token counts and no price, i.e. it has no pricing metadata for the custom 'litellm' provider. (If the worktrees are already pruned, the archive lane keeps tarballs — check archive_dir, default <work_root>/archive, for task-<id>.tar.gz.) 2. Look at how the litellm provider is declared in pi's config on the box: sudo -u agenthub bash -lc 'ls ~/.pi; grep -rn litellm ~/.pi 2>/dev/null | head' Does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is CONFIG-ONLY, on the box, with no code change in agents-dev at all — and this bead closes as 'not our bug, fixed in pi config'. 3. Cross-check what LiteLLM itself thinks the spend was, since it tracks per-virtual-key spend. If pi cannot be taught the prices, LiteLLM is the trustworthy source and the fix becomes 'read cost from LiteLLM, not from the event stream'. Only if step 1 comes back with cost fields that are populated-but-mis-shaped does this become a parser bug — and in that case capture the events.jsonl and add it as a testdata fixture, the way ah-1cx.4 did for the 0.82.1 shape. For reference, a REAL pi 0.82.1 capture against the DIRECT deepseek provider (local mac, 2026-08-05) has cost exactly where ParseEvents reads it: message.usage.cost = {input, output, cacheRead, cacheWrite, total}, all non-zero, and the arithmetic checks out against flat per-Mtok rates. So the parser is not the problem on a provider pi has prices for. · 2026-08-05T02:55:27Z
Stored rows — what this pane was built from, as read
issues 1 row
id ah-1cx.8
content_hash f9af778a32a0fc2ac9d0d9bf63b5430905203491ae53324cebc0c91faecff187
title [bug] cost_usd is 0 for every live run — pi prices from its own registry, the litellm provider has none
description Split out of ah-tqc's second finding so that bead can close on its first finding. /api/v1/status reports cost_usd=0 for ALL runs on agent-1, including real multi-minute ones (observed on tasks 1-2 during the 2026-07-19 Q&A smoke). The run row's CostUSD comes from ParseEvents summing message.usage.cost.total over assistant message_end events. RULED OUT: the parser. A real pi 0.82.1 capture against the DIRECT deepseek provider (deepseek/deepseek-v4-flash) puts cost.total exactly where ParseEvents reads it, and the summation is arithmetically correct (proven per-message, not cumulative — see ah-1cx.4). LEADING HYPOTHESIS: pi computes cost itself from a per-provider model pricing registry. agent-1 runs every role through a CUSTOM OpenAI-compatible provider named 'litellm' (role models are 'litellm/<model>'), and a custom provider carries no pricing metadata, so pi emits cost{input:0,output:0,total:0} while token counts are still real. If that holds, nothing in agent-hub is broken and the fix is to stop trusting pi for cost: either derive it from token counts against a locally configured price table, or read spend from the LiteLLM proxy (it tracks per-key spend), or drop the field from the status API rather than reporting a confident zero.
design Settle the hypothesis BEFORE writing code. Needed evidence, from agent-1 (the local ssh probe is blocked by the permission classifier — needs an operator '!' handoff): 1. A real events.jsonl from a completed run: jq -c 'select(.type=="message_end" and .message.role=="assistant") | .message.usage' over the archived stream. Are totalTokens non-zero while every cost.* is 0? That confirms the hypothesis outright. 2. The pi provider config on the box (how the 'litellm' provider is declared) — does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is config-only, on the box, with no code change at all. Only if both come back negative does this become a parser/shape bug.
acceptance_criteria Either (a) pi is configured/patched so cost.total is real for the litellm provider and a live run reports a non-zero cost_usd, or (b) cost is sourced from somewhere trustworthy, or (c) the field is explicitly documented+surfaced as unavailable rather than a silent 0. In all three cases SPEC and the status API description must match what is actually reported.
notes OPERATOR HANDOFF — the ssh probe from this dev session is blocked by the local permission classifier, so these need a '!' handoff or a human at a terminal. Run on agent-1 (or via ssh from the mac): 1. Find a completed run's event stream and look at what pi actually reported: ssh agent-1.lab.internal sudo -u agenthub bash -lc 'ls -t /opt/agent-hub/work/*/task-*/.task/events.jsonl 2>/dev/null | head -3' sudo -u agenthub bash -lc 'jq -c "select(.type==\"message_end\" and .message.role==\"assistant\") | .message.usage" <PATH>' The decisive question: are input/output/totalTokens NON-ZERO while every cost.* is 0? That confirms the hypothesis outright — pi is reporting real token counts and no price, i.e. it has no pricing metadata for the custom 'litellm' provider. (If the worktrees are already pruned, the archive lane keeps tarballs — check archive_dir, default <work_root>/archive, for task-<id>.tar.gz.) 2. Look at how the litellm provider is declared in pi's config on the box: sudo -u agenthub bash -lc 'ls ~/.pi; grep -rn litellm ~/.pi 2>/dev/null | head' Does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is CONFIG-ONLY, on the box, with no code change in agents-dev at all — and this bead closes as 'not our bug, fixed in pi config'. 3. Cross-check what LiteLLM itself thinks the spend was, since it tracks per-virtual-key spend. If pi cannot be taught the prices, LiteLLM is the trustworthy source and the fix becomes 'read cost from LiteLLM, not from the event stream'. Only if step 1 comes back with cost fields that are populated-but-mis-shaped does this become a parser bug — and in that case capture the events.jsonl and add it as a testdata fixture, the way ah-1cx.4 did for the 0.82.1 shape. For reference, a REAL pi 0.82.1 capture against the DIRECT deepseek provider (local mac, 2026-08-05) has cost exactly where ParseEvents reads it: message.usage.cost = {input, output, cacheRead, cacheWrite, total}, all non-zero, and the arithmetic checks out against flat per-Mtok rates. So the parser is not the problem on a provider pi has prices for.
status open
priority 3
issue_type bug
assignee NULL
estimated_minutes NULL
created_at 2026-08-04T23:32:21Z
created_by Eugene Blikh
owner bigbes@gmail.com
updated_at 2026-08-04T23:55:28Z
closed_at NULL
closed_by_session
external_ref NULL
spec_id
compaction_level 0
compacted_at NULL
compacted_at_commit NULL
original_size NULL
sender
ephemeral 0
wisp_type
pinned 0
is_template 0
mol_type
work_type
source_system
metadata �{}
source_repo
close_reason
event_kind
actor
target
payload
await_type
await_id
timeout_ns 0
waiters
hook_bead
role_bead
agent_state
last_activity NULL
role_type
rig
due_at NULL
defer_until NULL
no_history 0
started_at NULL
is_blocked 0
labels 1 row
issue_id ah-1cx.8
label milestone:v0.1.0
dependencies 1 row
id e50673a4-3c25-53b6-bd66-8c89812cca1a
issue_id ah-1cx.8
type parent-child
created_at 2026-08-05T02:32:20Z
created_by Eugene Blikh
metadata �{}
thread_id
depends_on_issue_id ah-1cx
depends_on_wisp_id NULL
depends_on_external NULL
events 3 rows
id 019fcf1e-f613-790c-8f99-35fe20082165
issue_id ah-1cx.8
event_type created
actor Eugene Blikh
old_value
new_value
comment NULL
created_at 2026-08-05T02:32:20Z
id 019fcf1f-4d39-7ee8-b1e9-4a14bb842846
issue_id ah-1cx.8
event_type label_added
actor Eugene Blikh
old_value NULL
new_value NULL
comment Added label: milestone:v0.1.0
created_at 2026-08-05T02:32:42Z
id 019fcf34-1ff9-72f9-8ade-890847c38d63
issue_id ah-1cx.8
event_type updated
actor Eugene Blikh
old_value {"id":"ah-1cx.8","title":"[bug] cost_usd is 0 for every live run — pi prices from its own registry, the litellm provider has none","description":"Split out of ah-tqc's second finding so that bead can close on its first finding.\n\n/api/v1/status reports cost_usd=0 for ALL runs on agent-1, including real multi-minute ones (observed on tasks 1-2 during the 2026-07-19 Q\u0026A smoke). The run row's CostUSD comes from ParseEvents summing message.usage.cost.total over assistant message_end events.\n\nRULED OUT: the parser. A real pi 0.82.1 capture against the DIRECT deepseek provider (deepseek/deepseek-v4-flash) puts cost.total exactly where ParseEvents reads it, and the summation is arithmetically correct (proven per-message, not cumulative — see ah-1cx.4).\n\nLEADING HYPOTHESIS: pi computes cost itself from a per-provider model pricing registry. agent-1 runs every role through a CUSTOM OpenAI-compatible provider named 'litellm' (role models are 'litellm/\u003cmodel\u003e'), and a custom provider carries no pricing metadata, so pi emits cost{input:0,output:0,total:0} while token counts are still real. If that holds, nothing in agent-hub is broken and the fix is to stop trusting pi for cost: either derive it from token counts against a locally configured price table, or read spend from the LiteLLM proxy (it tracks per-key spend), or drop the field from the status API rather than reporting a confident zero.","design":"Settle the hypothesis BEFORE writing code. Needed evidence, from agent-1 (the local ssh probe is blocked by the permission classifier — needs an operator '!' handoff):\n\n1. A real events.jsonl from a completed run: jq -c 'select(.type==\"message_end\" and .message.role==\"assistant\") | .message.usage' over the archived stream. Are totalTokens non-zero while every cost.* is 0? That confirms the hypothesis outright.\n2. The pi provider config on the box (how the 'litellm' provider is declared) — does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is config-only, on the box, with no code change at all.\n\nOnly if both come back negative does this become a parser/shape bug.","acceptance_criteria":"Either (a) pi is configured/patched so cost.total is real for the litellm provider and a live run reports a non-zero cost_usd, or (b) cost is sourced from somewhere trustworthy, or (c) the field is explicitly documented+surfaced as unavailable rather than a silent 0. In all three cases SPEC and the status API description must match what is actually reported.","status":"open","priority":3,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-08-04T23:32:21Z","created_by":"Eugene Blikh","updated_at":"2026-08-04T23:32:21Z","labels":["milestone:v0.1.0"]}
new_value {"notes":"OPERATOR HANDOFF — the ssh probe from this dev session is blocked by the local permission classifier, so these need a '!' handoff or a human at a terminal. Run on agent-1 (or via ssh from the mac):\n\n1. Find a completed run's event stream and look at what pi actually reported:\n ssh agent-1.lab.internal\n sudo -u agenthub bash -lc 'ls -t /opt/agent-hub/work/*/task-*/.task/events.jsonl 2\u003e/dev/null | head -3'\n sudo -u agenthub bash -lc 'jq -c \"select(.type==\\\"message_end\\\" and .message.role==\\\"assistant\\\") | .message.usage\" \u003cPATH\u003e'\n The decisive question: are input/output/totalTokens NON-ZERO while every cost.* is 0? That confirms the hypothesis outright — pi is reporting real token counts and no price, i.e. it has no pricing metadata for the custom 'litellm' provider.\n (If the worktrees are already pruned, the archive lane keeps tarballs — check archive_dir, default \u003cwork_root\u003e/archive, for task-\u003cid\u003e.tar.gz.)\n\n2. Look at how the litellm provider is declared in pi's config on the box:\n sudo -u agenthub bash -lc 'ls ~/.pi; grep -rn litellm ~/.pi 2\u003e/dev/null | head'\n Does pi's config format accept per-model pricing for a custom provider? If it does, the cheapest fix is CONFIG-ONLY, on the box, with no code change in agents-dev at all — and this bead closes as 'not our bug, fixed in pi config'.\n\n3. Cross-check what LiteLLM itself thinks the spend was, since it tracks per-virtual-key spend. If pi cannot be taught the prices, LiteLLM is the trustworthy source and the fix becomes 'read cost from LiteLLM, not from the event stream'.\n\nOnly if step 1 comes back with cost fields that are populated-but-mis-shaped does this become a parser bug — and in that case capture the events.jsonl and add it as a testdata fixture, the way ah-1cx.4 did for the 0.82.1 shape.\n\nFor reference, a REAL pi 0.82.1 capture against the DIRECT deepseek provider (local mac, 2026-08-05) has cost exactly where ParseEvents reads it: message.usage.cost = {input, output, cacheRead, cacheWrite, total}, all non-zero, and the arithmetic checks out against flat per-Mtok rates. So the parser is not the problem on a provider pi has prices for."}
comment NULL
created_at 2026-08-05T02:55:27Z