~bigbes/agents-dev · parade

main · last commit 13 days ago · 7g0stsfu

← Back to the parade

ah-nyl.13 fix: attempt-evidence composition (phantom adopt) + meta wedge + signal escalation Past Stand

status: closed P0 bug
bd reopen ah-nyl.13
Created byEugene Blikh
Ownerbigbes@gmail.com
Created2026-07-13T01:51:43Z
Started2026-07-13T01:51:44Z
Updated2026-07-13T02:26:12Z
Closed2026-07-13T02:26:12Z
Description
Fix-verification review of the fix commits found a CRITICAL composition bug plus follow-ons. Findings below; line refs @ HEAD (80b565b). Scope: internal/ports (ONE additive field), internal/runner, cmd/ahub-run, internal/reconcile.

CORE RULE (fixes G1+G4): a live zellij session is evidence for attempt K ONLY when .task/meta.json's attempt == K (meta is written by Start(K)). The session task-<N> outlives attempts (keep-pane shell after normal exit; create-background default pane), so session-alive alone is NOT evidence.

G1 CRITICAL (reconcile.go:240-252 + runner.go:151-178): re-attempt after a finished run with the session alive: probe Status(task, latest+1) ignores old exit.json (attempt mismatch) and sees the alive session -> Running -> phantom-adopts a run row for attempt K+1 that was never Started -> "timeout" kill ~30min later. Also defeats the interrupted-finalize guard (reconcile.go:486: !st.Running false due to keep-pane shell) so a successful attempt can be republished as "attempt K+1 timeout". FIX in runner.Status: in the session-alive branch read meta.json; meta.attempt == queried attempt -> Running as today; meta.attempt != queried attempt (or meta absent) -> this session is NOT runtime for the queried attempt: report Completed=false, Running=false is not representable... implement as: expose the meta attempt in the status and let Running mean "session alive AND meta matches". Concretely: add field `MetaAttempt int` (0 = unknown/absent) to ports.RunStatus with a doc comment; populate it whenever meta.json is readable; Running=true ONLY when session alive && MetaAttempt == queried attempt; when session alive && MetaAttempt != attempt -> Running=false, Completed=false (a new legitimate "no evidence for this attempt" state — update the ports doc comment for Status accordingly: exactly one of Running/Completed OR neither when the live session belongs to a different attempt). Reconcile: runtimeEvidence stays (Running || Completed-non-crashed) — the neither-state naturally means "no evidence", so claim proceeds with a real Start. Verify the crash-window adopt still works (meta matches -> Running -> adopt).

G4 MAJOR (reconcile.go:478-484 + runner.go:151-158): DB-loss adoption probes attempt 1 while the live runtime is attempt K>1 -> with G1's fix alone this becomes "no evidence" -> wrongly fails the card while pi K runs unsupervised. FIX in reconcile.adoptOrFail: when the probed attempt yields no evidence but Status reports a live session with MetaAttempt M > 0 and M != probed attempt, re-probe/adopt attempt M (adopt the ACTUAL in-flight attempt: create run row at attempt M, StartedAt=now). Regression test: nil latest run + live runtime with meta{attempt:3} -> adopts attempt 3, no kill, no fail.

G3 MAJOR (runner.go:255-263, taskfiles.go:111-128): torn/corrupt meta.json permanently wedges Start (existingRun hard-errors every tick). FIX: write meta.json atomically (same tmp+fsync+rename helper as exit.json) AND treat unparseable meta.json as absent (warn + decline reuse) in both existingRun and the G1 Status path. Regression test: garbage meta.json -> Start proceeds fresh (after prune/reuse logic), Status doesn't error.

G5 MINOR (cmd/ahub-run/main.go:127-136): child ignoring SIGTERM/SIGHUP -> ahub-run waits forever, session killed under it, token burn. FIX: after forwarding the signal, bounded wait (10s) then SIGKILL the child process group; still write exit.json+report. Test with a TERM-ignoring child script.

G6 MINOR (reconcile.go:294-298): persistent Start failure (e.g. pi not on PATH) -> error event appended EVERY tick, unbounded, card stuck in ready. FIX: dedup — skip appending when the task's most recent event has identical kind+payload (cheap: track last event per task in-memory in the Reconciler); AND after 5 consecutive start failures for the same (task, attempt), move the card to failed with a comment (move-then-comment) so the operator sees it. Tests for both.

G7 MINOR (reconcile.go:486-496): dragging an already-finalized card back to in_progress re-runs finalize with moveCard=true -> duplicate result comment per drag. FIX: fire the interrupted-finalize completion branch with moveCard=true only when rec.State == in_progress; for terminal rec.State just align per the §10 terminal rules. Test: drag in_review card to in_progress with dead session -> no duplicate comment, converges.

Constraints: scope exactly internal/ports (RunStatus field + doc), internal/runner, cmd/ahub-run, internal/reconcile. Do NOT touch internal/vikunja (a sibling fixer works there concurrently), internal/store, internal/config, docs/, .beads/, go.mod. Conventions testify+culpa. Build/vet/test -race ONLY ./internal/ports/... ./internal/runner/... ./cmd/ahub-run/... ./internal/reconcile/... ; never ./... . Update the reconcile fakes so an unscripted (task,attempt) Status returns the realistic three-state contract (the zero-value RunStatus masked G1 — make fakes fail loudly on unscripted queries instead). Commits: 'runner: attempt-evidence via meta.json (+atomic meta, signal escalation)' + 'reconcile: adopt actual in-flight attempt, start-failure backoff, refinalize guard'; hooks bypassed; retry on index.lock; no push.

Depends on

  • ah-nyl — Stage 1 MVP: poll-driven happy path parent-child closed

Depended on by

Nothing depends on this issue.

No comments.

Close reason

landed a0c8858+24cb927: meta-aware attempt evidence (three-state Status w/ MetaAttempt), atomic meta + lenient corrupt handling, adopt actual in-flight attempt, SIGKILL escalation, start-failure dedup+escalation, refinalize bounce-back; fakes panic on unscripted queries; six traces pinned
  • Eugene Blikh added under epic ah-nyl · 2026-07-13T04:51:42Z
  • Eugene Blikh created the issue · 2026-07-13T04:51:42Z
  • Eugene Blikh changed status to in_progress · 2026-07-13T04:51:44Z
  • Eugene Blikh closed the issue · 2026-07-13T05:26:11Z
    landed a0c8858+24cb927: meta-aware attempt evidence (three-state Status w/ MetaAttempt), atomic meta + lenient corrupt handling, adopt actual in-flight attempt, SIGKILL escalation, start-failure dedup+escalation, refinalize bounce-back; fakes panic on unscripted queries; six traces pinned
  • Eugene Blikh added label milestone:stage-1 · 2026-07-20T03:08:46Z
  • Eugene Blikh removed label milestone:stage-1 · 2026-07-20T03:10:20Z
Stored rows — what this pane was built from, as read
issues 1 row
id ah-nyl.13
content_hash 7cbeacf33cd82863a1322764cd5c5d84db98099b33682f72598735f7aeccfc46
title fix: attempt-evidence composition (phantom adopt) + meta wedge + signal escalation
description Fix-verification review of the fix commits found a CRITICAL composition bug plus follow-ons. Findings below; line refs @ HEAD (80b565b). Scope: internal/ports (ONE additive field), internal/runner, cmd/ahub-run, internal/reconcile. CORE RULE (fixes G1+G4): a live zellij session is evidence for attempt K ONLY when .task/meta.json's attempt == K (meta is written by Start(K)). The session task-<N> outlives attempts (keep-pane shell after normal exit; create-background default pane), so session-alive alone is NOT evidence. G1 CRITICAL (reconcile.go:240-252 + runner.go:151-178): re-attempt after a finished run with the session alive: probe Status(task, latest+1) ignores old exit.json (attempt mismatch) and sees the alive session -> Running -> phantom-adopts a run row for attempt K+1 that was never Started -> "timeout" kill ~30min later. Also defeats the interrupted-finalize guard (reconcile.go:486: !st.Running false due to keep-pane shell) so a successful attempt can be republished as "attempt K+1 timeout". FIX in runner.Status: in the session-alive branch read meta.json; meta.attempt == queried attempt -> Running as today; meta.attempt != queried attempt (or meta absent) -> this session is NOT runtime for the queried attempt: report Completed=false, Running=false is not representable... implement as: expose the meta attempt in the status and let Running mean "session alive AND meta matches". Concretely: add field `MetaAttempt int` (0 = unknown/absent) to ports.RunStatus with a doc comment; populate it whenever meta.json is readable; Running=true ONLY when session alive && MetaAttempt == queried attempt; when session alive && MetaAttempt != attempt -> Running=false, Completed=false (a new legitimate "no evidence for this attempt" state — update the ports doc comment for Status accordingly: exactly one of Running/Completed OR neither when the live session belongs to a different attempt). Reconcile: runtimeEvidence stays (Running || Completed-non-crashed) — the neither-state naturally means "no evidence", so claim proceeds with a real Start. Verify the crash-window adopt still works (meta matches -> Running -> adopt). G4 MAJOR (reconcile.go:478-484 + runner.go:151-158): DB-loss adoption probes attempt 1 while the live runtime is attempt K>1 -> with G1's fix alone this becomes "no evidence" -> wrongly fails the card while pi K runs unsupervised. FIX in reconcile.adoptOrFail: when the probed attempt yields no evidence but Status reports a live session with MetaAttempt M > 0 and M != probed attempt, re-probe/adopt attempt M (adopt the ACTUAL in-flight attempt: create run row at attempt M, StartedAt=now). Regression test: nil latest run + live runtime with meta{attempt:3} -> adopts attempt 3, no kill, no fail. G3 MAJOR (runner.go:255-263, taskfiles.go:111-128): torn/corrupt meta.json permanently wedges Start (existingRun hard-errors every tick). FIX: write meta.json atomically (same tmp+fsync+rename helper as exit.json) AND treat unparseable meta.json as absent (warn + decline reuse) in both existingRun and the G1 Status path. Regression test: garbage meta.json -> Start proceeds fresh (after prune/reuse logic), Status doesn't error. G5 MINOR (cmd/ahub-run/main.go:127-136): child ignoring SIGTERM/SIGHUP -> ahub-run waits forever, session killed under it, token burn. FIX: after forwarding the signal, bounded wait (10s) then SIGKILL the child process group; still write exit.json+report. Test with a TERM-ignoring child script. G6 MINOR (reconcile.go:294-298): persistent Start failure (e.g. pi not on PATH) -> error event appended EVERY tick, unbounded, card stuck in ready. FIX: dedup — skip appending when the task's most recent event has identical kind+payload (cheap: track last event per task in-memory in the Reconciler); AND after 5 consecutive start failures for the same (task, attempt), move the card to failed with a comment (move-then-comment) so the operator sees it. Tests for both. G7 MINOR (reconcile.go:486-496): dragging an already-finalized card back to in_progress re-runs finalize with moveCard=true -> duplicate result comment per drag. FIX: fire the interrupted-finalize completion branch with moveCard=true only when rec.State == in_progress; for terminal rec.State just align per the §10 terminal rules. Test: drag in_review card to in_progress with dead session -> no duplicate comment, converges. Constraints: scope exactly internal/ports (RunStatus field + doc), internal/runner, cmd/ahub-run, internal/reconcile. Do NOT touch internal/vikunja (a sibling fixer works there concurrently), internal/store, internal/config, docs/, .beads/, go.mod. Conventions testify+culpa. Build/vet/test -race ONLY ./internal/ports/... ./internal/runner/... ./cmd/ahub-run/... ./internal/reconcile/... ; never ./... . Update the reconcile fakes so an unscripted (task,attempt) Status returns the realistic three-state contract (the zero-value RunStatus masked G1 — make fakes fail loudly on unscripted queries instead). Commits: 'runner: attempt-evidence via meta.json (+atomic meta, signal escalation)' + 'reconcile: adopt actual in-flight attempt, start-failure backoff, refinalize guard'; hooks bypassed; retry on index.lock; no push.
design
acceptance_criteria
notes
status closed
priority 0
issue_type bug
assignee NULL
estimated_minutes NULL
created_at 2026-07-13T01:51:43Z
created_by Eugene Blikh
owner bigbes@gmail.com
updated_at 2026-07-13T02:26:12Z
closed_at 2026-07-13T02:26:12Z
closed_by_session
external_ref NULL
spec_id
compaction_level 0
compacted_at NULL
compacted_at_commit NULL
original_size NULL
sender
ephemeral 0
wisp_type
pinned 0
is_template 0
mol_type
work_type
source_system
metadata �{}
source_repo
close_reason landed a0c8858+24cb927: meta-aware attempt evidence (three-state Status w/ MetaAttempt), atomic meta + lenient corrupt handling, adopt actual in-flight attempt, SIGKILL escalation, start-failure dedup+escalation, refinalize bounce-back; fakes panic on unscripted queries; six traces pinned
event_kind
actor
target
payload
await_type
await_id
timeout_ns 0
waiters
hook_bead
role_bead
agent_state
last_activity NULL
role_type
rig
due_at NULL
defer_until NULL
no_history 0
started_at 2026-07-13T01:51:44Z
is_blocked 0
dependencies 1 row
id 4372af3a-00cc-56ee-b979-e83992867124
issue_id ah-nyl.13
type parent-child
created_at 2026-07-13T04:51:42Z
created_by Eugene Blikh
metadata �{}
thread_id
depends_on_issue_id ah-nyl
depends_on_wisp_id NULL
depends_on_external NULL
events 5 rows
id 019f592c-4a40-7e25-a8a3-2e2fb8156823
issue_id ah-nyl.13
event_type created
actor Eugene Blikh
old_value
new_value
comment NULL
created_at 2026-07-13T04:51:42Z
id 019f592c-5028-7626-a05a-c9bd39512fbd
issue_id ah-nyl.13
event_type status_changed
actor Eugene Blikh
old_value {"id":"ah-nyl.13","title":"fix: attempt-evidence composition (phantom adopt) + meta wedge + signal escalation","description":"Fix-verification review of the fix commits found a CRITICAL composition bug plus follow-ons. Findings below; line refs @ HEAD (80b565b). Scope: internal/ports (ONE additive field), internal/runner, cmd/ahub-run, internal/reconcile.\n\nCORE RULE (fixes G1+G4): a live zellij session is evidence for attempt K ONLY when .task/meta.json's attempt == K (meta is written by Start(K)). The session task-\u003cN\u003e outlives attempts (keep-pane shell after normal exit; create-background default pane), so session-alive alone is NOT evidence.\n\nG1 CRITICAL (reconcile.go:240-252 + runner.go:151-178): re-attempt after a finished run with the session alive: probe Status(task, latest+1) ignores old exit.json (attempt mismatch) and sees the alive session -\u003e Running -\u003e phantom-adopts a run row for attempt K+1 that was never Started -\u003e \"timeout\" kill ~30min later. Also defeats the interrupted-finalize guard (reconcile.go:486: !st.Running false due to keep-pane shell) so a successful attempt can be republished as \"attempt K+1 timeout\". FIX in runner.Status: in the session-alive branch read meta.json; meta.attempt == queried attempt -\u003e Running as today; meta.attempt != queried attempt (or meta absent) -\u003e this session is NOT runtime for the queried attempt: report Completed=false, Running=false is not representable... implement as: expose the meta attempt in the status and let Running mean \"session alive AND meta matches\". Concretely: add field `MetaAttempt int` (0 = unknown/absent) to ports.RunStatus with a doc comment; populate it whenever meta.json is readable; Running=true ONLY when session alive \u0026\u0026 MetaAttempt == queried attempt; when session alive \u0026\u0026 MetaAttempt != attempt -\u003e Running=false, Completed=false (a new legitimate \"no evidence for this attempt\" state — update the ports doc comment for Status accordingly: exactly one of Running/Completed OR neither when the live session belongs to a different attempt). Reconcile: runtimeEvidence stays (Running || Completed-non-crashed) — the neither-state naturally means \"no evidence\", so claim proceeds with a real Start. Verify the crash-window adopt still works (meta matches -\u003e Running -\u003e adopt).\n\nG4 MAJOR (reconcile.go:478-484 + runner.go:151-158): DB-loss adoption probes attempt 1 while the live runtime is attempt K\u003e1 -\u003e with G1's fix alone this becomes \"no evidence\" -\u003e wrongly fails the card while pi K runs unsupervised. FIX in reconcile.adoptOrFail: when the probed attempt yields no evidence but Status reports a live session with MetaAttempt M \u003e 0 and M != probed attempt, re-probe/adopt attempt M (adopt the ACTUAL in-flight attempt: create run row at attempt M, StartedAt=now). Regression test: nil latest run + live runtime with meta{attempt:3} -\u003e adopts attempt 3, no kill, no fail.\n\nG3 MAJOR (runner.go:255-263, taskfiles.go:111-128): torn/corrupt meta.json permanently wedges Start (existingRun hard-errors every tick). FIX: write meta.json atomically (same tmp+fsync+rename helper as exit.json) AND treat unparseable meta.json as absent (warn + decline reuse) in both existingRun and the G1 Status path. Regression test: garbage meta.json -\u003e Start proceeds fresh (after prune/reuse logic), Status doesn't error.\n\nG5 MINOR (cmd/ahub-run/main.go:127-136): child ignoring SIGTERM/SIGHUP -\u003e ahub-run waits forever, session killed under it, token burn. FIX: after forwarding the signal, bounded wait (10s) then SIGKILL the child process group; still write exit.json+report. Test with a TERM-ignoring child script.\n\nG6 MINOR (reconcile.go:294-298): persistent Start failure (e.g. pi not on PATH) -\u003e error event appended EVERY tick, unbounded, card stuck in ready. FIX: dedup — skip appending when the task's most recent event has identical kind+payload (cheap: track last event per task in-memory in the Reconciler); AND after 5 consecutive start failures for the same (task, attempt), move the card to failed with a comment (move-then-comment) so the operator sees it. Tests for both.\n\nG7 MINOR (reconcile.go:486-496): dragging an already-finalized card back to in_progress re-runs finalize with moveCard=true -\u003e duplicate result comment per drag. FIX: fire the interrupted-finalize completion branch with moveCard=true only when rec.State == in_progress; for terminal rec.State just align per the §10 terminal rules. Test: drag in_review card to in_progress with dead session -\u003e no duplicate comment, converges.\n\nConstraints: scope exactly internal/ports (RunStatus field + doc), internal/runner, cmd/ahub-run, internal/reconcile. Do NOT touch internal/vikunja (a sibling fixer works there concurrently), internal/store, internal/config, docs/, .beads/, go.mod. Conventions testify+culpa. Build/vet/test -race ONLY ./internal/ports/... ./internal/runner/... ./cmd/ahub-run/... ./internal/reconcile/... ; never ./... . Update the reconcile fakes so an unscripted (task,attempt) Status returns the realistic three-state contract (the zero-value RunStatus masked G1 — make fakes fail loudly on unscripted queries instead). Commits: 'runner: attempt-evidence via meta.json (+atomic meta, signal escalation)' + 'reconcile: adopt actual in-flight attempt, start-failure backoff, refinalize guard'; hooks bypassed; retry on index.lock; no push.\n","status":"open","priority":0,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-07-13T01:51:43Z","created_by":"Eugene Blikh","updated_at":"2026-07-13T01:51:43Z"}
new_value {"status":"in_progress"}
comment NULL
created_at 2026-07-13T04:51:44Z
id 019f594b-dc01-75b1-9082-e8dae6ca2183
issue_id ah-nyl.13
event_type closed
actor Eugene Blikh
old_value
new_value landed a0c8858+24cb927: meta-aware attempt evidence (three-state Status w/ MetaAttempt), atomic meta + lenient corrupt handling, adopt actual in-flight attempt, SIGKILL escalation, start-failure dedup+escalation, refinalize bounce-back; fakes panic on unscripted queries; six traces pinned
comment NULL
created_at 2026-07-13T05:26:11Z
id 019f7cda-9083-7455-adcd-88d68410cbcf
issue_id ah-nyl.13
event_type label_added
actor Eugene Blikh
old_value NULL
new_value NULL
comment Added label: milestone:stage-1
created_at 2026-07-20T03:08:46Z
id 019f7cdb-fe95-7b46-b02a-bb40130db489
issue_id ah-nyl.13
event_type label_removed
actor Eugene Blikh
old_value NULL
new_value NULL
comment Removed label: milestone:stage-1
created_at 2026-07-20T03:10:20Z