main · last commit
13 days ago ·
7g0stsfu
ah-nyl.13 fix: attempt-evidence composition (phantom adopt) + meta wedge + signal escalation
Past Stand
bd reopen ah-nyl.13
| Created by | Eugene Blikh |
| Owner | bigbes@gmail.com |
| Created | 2026-07-13T01:51:43Z |
| Started | 2026-07-13T01:51:44Z |
| Updated | 2026-07-13T02:26:12Z |
| Closed | 2026-07-13T02:26:12Z |
Fix-verification review of the fix commits found a CRITICAL composition bug plus follow-ons. Findings below; line refs @ HEAD (80b565b). Scope: internal/ports (ONE additive field), internal/runner, cmd/ahub-run, internal/reconcile.
CORE RULE (fixes G1+G4): a live zellij session is evidence for attempt K ONLY when .task/meta.json's attempt == K (meta is written by Start(K)). The session task-<N> outlives attempts (keep-pane shell after normal exit; create-background default pane), so session-alive alone is NOT evidence.
G1 CRITICAL (reconcile.go:240-252 + runner.go:151-178): re-attempt after a finished run with the session alive: probe Status(task, latest+1) ignores old exit.json (attempt mismatch) and sees the alive session -> Running -> phantom-adopts a run row for attempt K+1 that was never Started -> "timeout" kill ~30min later. Also defeats the interrupted-finalize guard (reconcile.go:486: !st.Running false due to keep-pane shell) so a successful attempt can be republished as "attempt K+1 timeout". FIX in runner.Status: in the session-alive branch read meta.json; meta.attempt == queried attempt -> Running as today; meta.attempt != queried attempt (or meta absent) -> this session is NOT runtime for the queried attempt: report Completed=false, Running=false is not representable... implement as: expose the meta attempt in the status and let Running mean "session alive AND meta matches". Concretely: add field `MetaAttempt int` (0 = unknown/absent) to ports.RunStatus with a doc comment; populate it whenever meta.json is readable; Running=true ONLY when session alive && MetaAttempt == queried attempt; when session alive && MetaAttempt != attempt -> Running=false, Completed=false (a new legitimate "no evidence for this attempt" state — update the ports doc comment for Status accordingly: exactly one of Running/Completed OR neither when the live session belongs to a different attempt). Reconcile: runtimeEvidence stays (Running || Completed-non-crashed) — the neither-state naturally means "no evidence", so claim proceeds with a real Start. Verify the crash-window adopt still works (meta matches -> Running -> adopt).
G4 MAJOR (reconcile.go:478-484 + runner.go:151-158): DB-loss adoption probes attempt 1 while the live runtime is attempt K>1 -> with G1's fix alone this becomes "no evidence" -> wrongly fails the card while pi K runs unsupervised. FIX in reconcile.adoptOrFail: when the probed attempt yields no evidence but Status reports a live session with MetaAttempt M > 0 and M != probed attempt, re-probe/adopt attempt M (adopt the ACTUAL in-flight attempt: create run row at attempt M, StartedAt=now). Regression test: nil latest run + live runtime with meta{attempt:3} -> adopts attempt 3, no kill, no fail.
G3 MAJOR (runner.go:255-263, taskfiles.go:111-128): torn/corrupt meta.json permanently wedges Start (existingRun hard-errors every tick). FIX: write meta.json atomically (same tmp+fsync+rename helper as exit.json) AND treat unparseable meta.json as absent (warn + decline reuse) in both existingRun and the G1 Status path. Regression test: garbage meta.json -> Start proceeds fresh (after prune/reuse logic), Status doesn't error.
G5 MINOR (cmd/ahub-run/main.go:127-136): child ignoring SIGTERM/SIGHUP -> ahub-run waits forever, session killed under it, token burn. FIX: after forwarding the signal, bounded wait (10s) then SIGKILL the child process group; still write exit.json+report. Test with a TERM-ignoring child script.
G6 MINOR (reconcile.go:294-298): persistent Start failure (e.g. pi not on PATH) -> error event appended EVERY tick, unbounded, card stuck in ready. FIX: dedup — skip appending when the task's most recent event has identical kind+payload (cheap: track last event per task in-memory in the Reconciler); AND after 5 consecutive start failures for the same (task, attempt), move the card to failed with a comment (move-then-comment) so the operator sees it. Tests for both.
G7 MINOR (reconcile.go:486-496): dragging an already-finalized card back to in_progress re-runs finalize with moveCard=true -> duplicate result comment per drag. FIX: fire the interrupted-finalize completion branch with moveCard=true only when rec.State == in_progress; for terminal rec.State just align per the §10 terminal rules. Test: drag in_review card to in_progress with dead session -> no duplicate comment, converges.
Constraints: scope exactly internal/ports (RunStatus field + doc), internal/runner, cmd/ahub-run, internal/reconcile. Do NOT touch internal/vikunja (a sibling fixer works there concurrently), internal/store, internal/config, docs/, .beads/, go.mod. Conventions testify+culpa. Build/vet/test -race ONLY ./internal/ports/... ./internal/runner/... ./cmd/ahub-run/... ./internal/reconcile/... ; never ./... . Update the reconcile fakes so an unscripted (task,attempt) Status returns the realistic three-state contract (the zero-value RunStatus masked G1 — make fakes fail loudly on unscripted queries instead). Commits: 'runner: attempt-evidence via meta.json (+atomic meta, signal escalation)' + 'reconcile: adopt actual in-flight attempt, start-failure backoff, refinalize guard'; hooks bypassed; retry on index.lock; no push.
ah-nyl
— Stage 1 MVP: poll-driven happy path
parent-child
closed
Nothing depends on this issue.
| id | ah-nyl.13 |
| content_hash | 7cbeacf33cd82863a1322764cd5c5d84db98099b33682f72598735f7aeccfc46 |
| title | fix: attempt-evidence composition (phantom adopt) + meta wedge + signal escalation |
| description | Fix-verification review of the fix commits found a CRITICAL composition bug plus follow-ons. Findings below; line refs @ HEAD (80b565b). Scope: internal/ports (ONE additive field), internal/runner, cmd/ahub-run, internal/reconcile. CORE RULE (fixes G1+G4): a live zellij session is evidence for attempt K ONLY when .task/meta.json's attempt == K (meta is written by Start(K)). The session task-<N> outlives attempts (keep-pane shell after normal exit; create-background default pane), so session-alive alone is NOT evidence. G1 CRITICAL (reconcile.go:240-252 + runner.go:151-178): re-attempt after a finished run with the session alive: probe Status(task, latest+1) ignores old exit.json (attempt mismatch) and sees the alive session -> Running -> phantom-adopts a run row for attempt K+1 that was never Started -> "timeout" kill ~30min later. Also defeats the interrupted-finalize guard (reconcile.go:486: !st.Running false due to keep-pane shell) so a successful attempt can be republished as "attempt K+1 timeout". FIX in runner.Status: in the session-alive branch read meta.json; meta.attempt == queried attempt -> Running as today; meta.attempt != queried attempt (or meta absent) -> this session is NOT runtime for the queried attempt: report Completed=false, Running=false is not representable... implement as: expose the meta attempt in the status and let Running mean "session alive AND meta matches". Concretely: add field `MetaAttempt int` (0 = unknown/absent) to ports.RunStatus with a doc comment; populate it whenever meta.json is readable; Running=true ONLY when session alive && MetaAttempt == queried attempt; when session alive && MetaAttempt != attempt -> Running=false, Completed=false (a new legitimate "no evidence for this attempt" state — update the ports doc comment for Status accordingly: exactly one of Running/Completed OR neither when the live session belongs to a different attempt). Reconcile: runtimeEvidence stays (Running || Completed-non-crashed) — the neither-state naturally means "no evidence", so claim proceeds with a real Start. Verify the crash-window adopt still works (meta matches -> Running -> adopt). G4 MAJOR (reconcile.go:478-484 + runner.go:151-158): DB-loss adoption probes attempt 1 while the live runtime is attempt K>1 -> with G1's fix alone this becomes "no evidence" -> wrongly fails the card while pi K runs unsupervised. FIX in reconcile.adoptOrFail: when the probed attempt yields no evidence but Status reports a live session with MetaAttempt M > 0 and M != probed attempt, re-probe/adopt attempt M (adopt the ACTUAL in-flight attempt: create run row at attempt M, StartedAt=now). Regression test: nil latest run + live runtime with meta{attempt:3} -> adopts attempt 3, no kill, no fail. G3 MAJOR (runner.go:255-263, taskfiles.go:111-128): torn/corrupt meta.json permanently wedges Start (existingRun hard-errors every tick). FIX: write meta.json atomically (same tmp+fsync+rename helper as exit.json) AND treat unparseable meta.json as absent (warn + decline reuse) in both existingRun and the G1 Status path. Regression test: garbage meta.json -> Start proceeds fresh (after prune/reuse logic), Status doesn't error. G5 MINOR (cmd/ahub-run/main.go:127-136): child ignoring SIGTERM/SIGHUP -> ahub-run waits forever, session killed under it, token burn. FIX: after forwarding the signal, bounded wait (10s) then SIGKILL the child process group; still write exit.json+report. Test with a TERM-ignoring child script. G6 MINOR (reconcile.go:294-298): persistent Start failure (e.g. pi not on PATH) -> error event appended EVERY tick, unbounded, card stuck in ready. FIX: dedup — skip appending when the task's most recent event has identical kind+payload (cheap: track last event per task in-memory in the Reconciler); AND after 5 consecutive start failures for the same (task, attempt), move the card to failed with a comment (move-then-comment) so the operator sees it. Tests for both. G7 MINOR (reconcile.go:486-496): dragging an already-finalized card back to in_progress re-runs finalize with moveCard=true -> duplicate result comment per drag. FIX: fire the interrupted-finalize completion branch with moveCard=true only when rec.State == in_progress; for terminal rec.State just align per the §10 terminal rules. Test: drag in_review card to in_progress with dead session -> no duplicate comment, converges. Constraints: scope exactly internal/ports (RunStatus field + doc), internal/runner, cmd/ahub-run, internal/reconcile. Do NOT touch internal/vikunja (a sibling fixer works there concurrently), internal/store, internal/config, docs/, .beads/, go.mod. Conventions testify+culpa. Build/vet/test -race ONLY ./internal/ports/... ./internal/runner/... ./cmd/ahub-run/... ./internal/reconcile/... ; never ./... . Update the reconcile fakes so an unscripted (task,attempt) Status returns the realistic three-state contract (the zero-value RunStatus masked G1 — make fakes fail loudly on unscripted queries instead). Commits: 'runner: attempt-evidence via meta.json (+atomic meta, signal escalation)' + 'reconcile: adopt actual in-flight attempt, start-failure backoff, refinalize guard'; hooks bypassed; retry on index.lock; no push. |
| design | |
| acceptance_criteria | |
| notes | |
| status | closed |
| priority | 0 |
| issue_type | bug |
| assignee | NULL |
| estimated_minutes | NULL |
| created_at | 2026-07-13T01:51:43Z |
| created_by | Eugene Blikh |
| owner | bigbes@gmail.com |
| updated_at | 2026-07-13T02:26:12Z |
| closed_at | 2026-07-13T02:26:12Z |
| closed_by_session | |
| external_ref | NULL |
| spec_id | |
| compaction_level | 0 |
| compacted_at | NULL |
| compacted_at_commit | NULL |
| original_size | NULL |
| sender | |
| ephemeral | 0 |
| wisp_type | |
| pinned | 0 |
| is_template | 0 |
| mol_type | |
| work_type | |
| source_system | |
| metadata | �{} |
| source_repo | |
| close_reason | landed a0c8858+24cb927: meta-aware attempt evidence (three-state Status w/ MetaAttempt), atomic meta + lenient corrupt handling, adopt actual in-flight attempt, SIGKILL escalation, start-failure dedup+escalation, refinalize bounce-back; fakes panic on unscripted queries; six traces pinned |
| event_kind | |
| actor | |
| target | |
| payload | |
| await_type | |
| await_id | |
| timeout_ns | 0 |
| waiters | |
| hook_bead | |
| role_bead | |
| agent_state | |
| last_activity | NULL |
| role_type | |
| rig | |
| due_at | NULL |
| defer_until | NULL |
| no_history | 0 |
| started_at | 2026-07-13T01:51:44Z |
| is_blocked | 0 |
| id | 4372af3a-00cc-56ee-b979-e83992867124 |
| issue_id | ah-nyl.13 |
| type | parent-child |
| created_at | 2026-07-13T04:51:42Z |
| created_by | Eugene Blikh |
| metadata | �{} |
| thread_id | |
| depends_on_issue_id | ah-nyl |
| depends_on_wisp_id | NULL |
| depends_on_external | NULL |
| id | 019f592c-4a40-7e25-a8a3-2e2fb8156823 |
| issue_id | ah-nyl.13 |
| event_type | created |
| actor | Eugene Blikh |
| old_value | |
| new_value | |
| comment | NULL |
| created_at | 2026-07-13T04:51:42Z |
| id | 019f592c-5028-7626-a05a-c9bd39512fbd |
| issue_id | ah-nyl.13 |
| event_type | status_changed |
| actor | Eugene Blikh |
| old_value | {"id":"ah-nyl.13","title":"fix: attempt-evidence composition (phantom adopt) + meta wedge + signal escalation","description":"Fix-verification review of the fix commits found a CRITICAL composition bug plus follow-ons. Findings below; line refs @ HEAD (80b565b). Scope: internal/ports (ONE additive field), internal/runner, cmd/ahub-run, internal/reconcile.\n\nCORE RULE (fixes G1+G4): a live zellij session is evidence for attempt K ONLY when .task/meta.json's attempt == K (meta is written by Start(K)). The session task-\u003cN\u003e outlives attempts (keep-pane shell after normal exit; create-background default pane), so session-alive alone is NOT evidence.\n\nG1 CRITICAL (reconcile.go:240-252 + runner.go:151-178): re-attempt after a finished run with the session alive: probe Status(task, latest+1) ignores old exit.json (attempt mismatch) and sees the alive session -\u003e Running -\u003e phantom-adopts a run row for attempt K+1 that was never Started -\u003e \"timeout\" kill ~30min later. Also defeats the interrupted-finalize guard (reconcile.go:486: !st.Running false due to keep-pane shell) so a successful attempt can be republished as \"attempt K+1 timeout\". FIX in runner.Status: in the session-alive branch read meta.json; meta.attempt == queried attempt -\u003e Running as today; meta.attempt != queried attempt (or meta absent) -\u003e this session is NOT runtime for the queried attempt: report Completed=false, Running=false is not representable... implement as: expose the meta attempt in the status and let Running mean \"session alive AND meta matches\". Concretely: add field `MetaAttempt int` (0 = unknown/absent) to ports.RunStatus with a doc comment; populate it whenever meta.json is readable; Running=true ONLY when session alive \u0026\u0026 MetaAttempt == queried attempt; when session alive \u0026\u0026 MetaAttempt != attempt -\u003e Running=false, Completed=false (a new legitimate \"no evidence for this attempt\" state — update the ports doc comment for Status accordingly: exactly one of Running/Completed OR neither when the live session belongs to a different attempt). Reconcile: runtimeEvidence stays (Running || Completed-non-crashed) — the neither-state naturally means \"no evidence\", so claim proceeds with a real Start. Verify the crash-window adopt still works (meta matches -\u003e Running -\u003e adopt).\n\nG4 MAJOR (reconcile.go:478-484 + runner.go:151-158): DB-loss adoption probes attempt 1 while the live runtime is attempt K\u003e1 -\u003e with G1's fix alone this becomes \"no evidence\" -\u003e wrongly fails the card while pi K runs unsupervised. FIX in reconcile.adoptOrFail: when the probed attempt yields no evidence but Status reports a live session with MetaAttempt M \u003e 0 and M != probed attempt, re-probe/adopt attempt M (adopt the ACTUAL in-flight attempt: create run row at attempt M, StartedAt=now). Regression test: nil latest run + live runtime with meta{attempt:3} -\u003e adopts attempt 3, no kill, no fail.\n\nG3 MAJOR (runner.go:255-263, taskfiles.go:111-128): torn/corrupt meta.json permanently wedges Start (existingRun hard-errors every tick). FIX: write meta.json atomically (same tmp+fsync+rename helper as exit.json) AND treat unparseable meta.json as absent (warn + decline reuse) in both existingRun and the G1 Status path. Regression test: garbage meta.json -\u003e Start proceeds fresh (after prune/reuse logic), Status doesn't error.\n\nG5 MINOR (cmd/ahub-run/main.go:127-136): child ignoring SIGTERM/SIGHUP -\u003e ahub-run waits forever, session killed under it, token burn. FIX: after forwarding the signal, bounded wait (10s) then SIGKILL the child process group; still write exit.json+report. Test with a TERM-ignoring child script.\n\nG6 MINOR (reconcile.go:294-298): persistent Start failure (e.g. pi not on PATH) -\u003e error event appended EVERY tick, unbounded, card stuck in ready. FIX: dedup — skip appending when the task's most recent event has identical kind+payload (cheap: track last event per task in-memory in the Reconciler); AND after 5 consecutive start failures for the same (task, attempt), move the card to failed with a comment (move-then-comment) so the operator sees it. Tests for both.\n\nG7 MINOR (reconcile.go:486-496): dragging an already-finalized card back to in_progress re-runs finalize with moveCard=true -\u003e duplicate result comment per drag. FIX: fire the interrupted-finalize completion branch with moveCard=true only when rec.State == in_progress; for terminal rec.State just align per the §10 terminal rules. Test: drag in_review card to in_progress with dead session -\u003e no duplicate comment, converges.\n\nConstraints: scope exactly internal/ports (RunStatus field + doc), internal/runner, cmd/ahub-run, internal/reconcile. Do NOT touch internal/vikunja (a sibling fixer works there concurrently), internal/store, internal/config, docs/, .beads/, go.mod. Conventions testify+culpa. Build/vet/test -race ONLY ./internal/ports/... ./internal/runner/... ./cmd/ahub-run/... ./internal/reconcile/... ; never ./... . Update the reconcile fakes so an unscripted (task,attempt) Status returns the realistic three-state contract (the zero-value RunStatus masked G1 — make fakes fail loudly on unscripted queries instead). Commits: 'runner: attempt-evidence via meta.json (+atomic meta, signal escalation)' + 'reconcile: adopt actual in-flight attempt, start-failure backoff, refinalize guard'; hooks bypassed; retry on index.lock; no push.\n","status":"open","priority":0,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-07-13T01:51:43Z","created_by":"Eugene Blikh","updated_at":"2026-07-13T01:51:43Z"} |
| new_value | {"status":"in_progress"} |
| comment | NULL |
| created_at | 2026-07-13T04:51:44Z |
| id | 019f594b-dc01-75b1-9082-e8dae6ca2183 |
| issue_id | ah-nyl.13 |
| event_type | closed |
| actor | Eugene Blikh |
| old_value | |
| new_value | landed a0c8858+24cb927: meta-aware attempt evidence (three-state Status w/ MetaAttempt), atomic meta + lenient corrupt handling, adopt actual in-flight attempt, SIGKILL escalation, start-failure dedup+escalation, refinalize bounce-back; fakes panic on unscripted queries; six traces pinned |
| comment | NULL |
| created_at | 2026-07-13T05:26:11Z |
| id | 019f7cda-9083-7455-adcd-88d68410cbcf |
| issue_id | ah-nyl.13 |
| event_type | label_added |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Added label: milestone:stage-1 |
| created_at | 2026-07-20T03:08:46Z |
| id | 019f7cdb-fe95-7b46-b02a-bb40130db489 |
| issue_id | ah-nyl.13 |
| event_type | label_removed |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Removed label: milestone:stage-1 |
| created_at | 2026-07-20T03:10:20Z |
No comments.
Close reason