main · last commit
19 hours ago ·
8tmadfpi
spec-ejq.2 CI publish task fails: build secret apk-ci-s3 is missing
Past Stand
bd reopen spec-ejq.2
| Created by | Eugene Blikh |
| Owner | bigbes@gmail.com |
| Created | 2026-07-24T21:16:35Z |
| Updated | 2026-08-05T03:20:11Z |
| Closed | 2026-08-05T03:12:30Z |
Every recent builds.sr.ht run fails at the publish task while every other task passes. Confirmed on jobs #244 (e97532c), #245 (2dc6b71) and #246 (c2dd1ef): 'scss keygen version build' all SUCCESS, 'publish' FAILED. Pre-existing — #244 and #245 predate the Phase 5b work, so no code change caused it. Consequence: the apk never reaches the Garage repo bucket, so apk-mirror on phoebe has nothing new to re-index and the deployed service cannot be upgraded from CI output.
EVIDENCE. .build.yml declares 'secrets: [apk-ci-s3]', documented as a File secret installed at ~/.apk-ci.env carrying APK_CI_S3_ACCESS_KEY / APK_CI_S3_SECRET_KEY for the Garage repo bucket. The publish task's first real line is '. ~/.apk-ci.env'. But 'hut builds secret list' returns exactly four secrets and none is apk-ci-s3: agent1-deploy (SSH key), bencher-api-key, s3-cache-key-secret, s3-cache-key-id. So the file the task sources is never installed. NOT VERIFIED: the exact failure text. The raw log endpoint (https://builds.srht.bigb.es/query/log/246/publish/log) needs a Bearer token, and reading hut's credential file was correctly refused, so the diagnosis rests on the secret list plus the manifest rather than on the log line itself. Read the log to confirm before acting. TWO POSSIBILITIES, needs the owner to distinguish: 1. The secret was deleted or never created — fix is to create a File secret named apk-ci-s3 at ~/.apk-ci.env (mode 600) holding the two S3 keys. 2. builds.sr.ht resolves manifest secrets by UUID, not by name — fix is to replace the name with the secret's UUID in .build.yml. SAME BUG IN THE SIBLING: ~/data/home/sourcehut-compare/.build.yml carries a byte-identical secrets block, so compare.sr.ht's publish is broken the same way and both fix together.
A push to master produces a build whose publish task succeeds and uploads the .apk to the Garage repo bucket; apk-mirror on phoebe re-indexes it within 15 minutes.
RE-INDEX VERIFIED 2026-08-05, the half the close reason left open. Checked through repo.bigb.es, the public mirror apk-mirror builds from the Garage repo bucket (URL documented in the APKBUILD header; s3.bigb.es itself resolves to 192.168.88.18 and is only reachable from the LAN, which is why the first attempt to verify failed). spec.sr.ht-0.0.78-r0.apk 200 content-length 16059260 last-modified 03:09:39 GMT APKINDEX.tar.gz 200 last-modified 03:15:00 GMT index entry: P:spec.sr.ht V:0.0.78-r0 A:x86_64 S:16059260 The size in the index matches the object byte for byte, and 0.0.78 is 'git rev-list --count cc90b4a' — the commit build #251 ran on. The index was rebuilt 5.5 minutes after the upload, inside the 15-minute window the acceptance criteria name. Both halves of the criteria are therefore met. Correction to the close reason: the package is named spec.sr.ht-0.0.78-r0.apk, not specsrht-0.0.78-r0.apk — pkgname in APKBUILD is spec.sr.ht. The specsrht spelling 404s.
spec-ejq
— spec.sr.ht — reviewable document storage for humans and agents
parent-child
open
Nothing depends on this issue.
| id | spec-ejq.2 |
| content_hash | 0c97aa6c887b865c89b63400c0abf70ad355e41556e3ca1b0f0f3a803318b5e7 |
| title | CI publish task fails: build secret apk-ci-s3 is missing |
| description | Every recent builds.sr.ht run fails at the publish task while every other task passes. Confirmed on jobs #244 (e97532c), #245 (2dc6b71) and #246 (c2dd1ef): 'scss keygen version build' all SUCCESS, 'publish' FAILED. Pre-existing — #244 and #245 predate the Phase 5b work, so no code change caused it. Consequence: the apk never reaches the Garage repo bucket, so apk-mirror on phoebe has nothing new to re-index and the deployed service cannot be upgraded from CI output. |
| design | EVIDENCE. .build.yml declares 'secrets: [apk-ci-s3]', documented as a File secret installed at ~/.apk-ci.env carrying APK_CI_S3_ACCESS_KEY / APK_CI_S3_SECRET_KEY for the Garage repo bucket. The publish task's first real line is '. ~/.apk-ci.env'. But 'hut builds secret list' returns exactly four secrets and none is apk-ci-s3: agent1-deploy (SSH key), bencher-api-key, s3-cache-key-secret, s3-cache-key-id. So the file the task sources is never installed. NOT VERIFIED: the exact failure text. The raw log endpoint (https://builds.srht.bigb.es/query/log/246/publish/log) needs a Bearer token, and reading hut's credential file was correctly refused, so the diagnosis rests on the secret list plus the manifest rather than on the log line itself. Read the log to confirm before acting. TWO POSSIBILITIES, needs the owner to distinguish: 1. The secret was deleted or never created — fix is to create a File secret named apk-ci-s3 at ~/.apk-ci.env (mode 600) holding the two S3 keys. 2. builds.sr.ht resolves manifest secrets by UUID, not by name — fix is to replace the name with the secret's UUID in .build.yml. SAME BUG IN THE SIBLING: ~/data/home/sourcehut-compare/.build.yml carries a byte-identical secrets block, so compare.sr.ht's publish is broken the same way and both fix together. |
| acceptance_criteria | A push to master produces a build whose publish task succeeds and uploads the .apk to the Garage repo bucket; apk-mirror on phoebe re-indexes it within 15 minutes. |
| notes | RE-INDEX VERIFIED 2026-08-05, the half the close reason left open. Checked through repo.bigb.es, the public mirror apk-mirror builds from the Garage repo bucket (URL documented in the APKBUILD header; s3.bigb.es itself resolves to 192.168.88.18 and is only reachable from the LAN, which is why the first attempt to verify failed). spec.sr.ht-0.0.78-r0.apk 200 content-length 16059260 last-modified 03:09:39 GMT APKINDEX.tar.gz 200 last-modified 03:15:00 GMT index entry: P:spec.sr.ht V:0.0.78-r0 A:x86_64 S:16059260 The size in the index matches the object byte for byte, and 0.0.78 is 'git rev-list --count cc90b4a' — the commit build #251 ran on. The index was rebuilt 5.5 minutes after the upload, inside the 15-minute window the acceptance criteria name. Both halves of the criteria are therefore met. Correction to the close reason: the package is named spec.sr.ht-0.0.78-r0.apk, not specsrht-0.0.78-r0.apk — pkgname in APKBUILD is spec.sr.ht. The specsrht spelling 404s. |
| status | closed |
| priority | 2 |
| issue_type | bug |
| assignee | NULL |
| estimated_minutes | NULL |
| created_at | 2026-07-24T21:16:35Z |
| created_by | Eugene Blikh |
| owner | bigbes@gmail.com |
| updated_at | 2026-08-05T03:20:11Z |
| closed_at | 2026-08-05T03:12:30Z |
| closed_by_session | |
| external_ref | NULL |
| spec_id | |
| compaction_level | 0 |
| compacted_at | NULL |
| compacted_at_commit | NULL |
| original_size | NULL |
| sender | |
| ephemeral | 0 |
| wisp_type | |
| pinned | 0 |
| is_template | 0 |
| mol_type | |
| work_type | |
| source_system | |
| metadata | �{} |
| source_repo | |
| close_reason | Owner created the File secret apk-ci-s3 (/home/build/.apk-ci.env, mode 600) on builds.srht.bigb.es; 'hut builds secret list' now shows it. Resubmit of #250 as build #251 on the same commit cc90b4a: SUCCESS, all five tasks green including publish. VERIFIED: publish runs under 'set -e' and its body is a loop of 'rclone copyto' with an echo per file, so any failed upload would have exited non-zero and failed the task. A green publish is therefore evidence the .apk reached repo/alpine/v3.22/bigbes/x86_64/ — specsrht-0.0.78-r0.apk for this commit. NOT VERIFIED from here: the apk-mirror re-index on phoebe, the second half of the acceptance criteria. s3.bigb.es is not reachable anonymously from this workstation and phoebe host access is blocked (spec-ar4). If the package turns out not to be installable, that is apk-mirror's problem rather than this repo's and wants its own bead. The manifest needed no change: '- apk-ci-s3' resolves by name, as the diagnosis note records. The same account-level secret should also unblock sourcehut-compare, whose .build.yml carries the identical block — its next build will show it. |
| event_kind | |
| actor | |
| target | |
| payload | |
| await_type | |
| await_id | |
| timeout_ns | 0 |
| waiters | |
| hook_bead | |
| role_bead | |
| agent_state | |
| last_activity | NULL |
| role_type | |
| rig | |
| due_at | NULL |
| defer_until | NULL |
| no_history | 0 |
| started_at | NULL |
| is_blocked | 0 |
| id | d89b7b8f-bd3e-5b47-84a1-c2fb58cfb8f5 |
| issue_id | spec-ejq.2 |
| type | parent-child |
| created_at | 2026-07-25T00:16:35Z |
| created_by | Eugene Blikh |
| metadata | �{} |
| thread_id | |
| depends_on_issue_id | spec-ejq |
| depends_on_wisp_id | NULL |
| depends_on_external | NULL |
| id | 019f95fc-b75e-7b00-979a-510d501bbb9f |
| issue_id | spec-ejq.2 |
| event_type | created |
| actor | Eugene Blikh |
| old_value | |
| new_value | |
| comment | NULL |
| created_at | 2026-07-25T00:16:35Z |
| id | 019fcfe1-a1db-7d94-a36f-7dbeac40c1bc |
| issue_id | spec-ejq.2 |
| event_type | updated |
| actor | Eugene Blikh |
| old_value | {"id":"spec-ejq.2","title":"CI publish task fails: build secret apk-ci-s3 is missing","description":"Every recent builds.sr.ht run fails at the publish task while every other task passes. Confirmed on jobs #244 (e97532c), #245 (2dc6b71) and #246 (c2dd1ef): 'scss keygen version build' all SUCCESS, 'publish' FAILED. Pre-existing — #244 and #245 predate the Phase 5b work, so no code change caused it. Consequence: the apk never reaches the Garage repo bucket, so apk-mirror on phoebe has nothing new to re-index and the deployed service cannot be upgraded from CI output.","design":"EVIDENCE. .build.yml declares 'secrets: [apk-ci-s3]', documented as a File secret installed at ~/.apk-ci.env carrying APK_CI_S3_ACCESS_KEY / APK_CI_S3_SECRET_KEY for the Garage repo bucket. The publish task's first real line is '. ~/.apk-ci.env'. But 'hut builds secret list' returns exactly four secrets and none is apk-ci-s3: agent1-deploy (SSH key), bencher-api-key, s3-cache-key-secret, s3-cache-key-id. So the file the task sources is never installed.\n\nNOT VERIFIED: the exact failure text. The raw log endpoint (https://builds.srht.bigb.es/query/log/246/publish/log) needs a Bearer token, and reading hut's credential file was correctly refused, so the diagnosis rests on the secret list plus the manifest rather than on the log line itself. Read the log to confirm before acting.\n\nTWO POSSIBILITIES, needs the owner to distinguish:\n1. The secret was deleted or never created — fix is to create a File secret named apk-ci-s3 at ~/.apk-ci.env (mode 600) holding the two S3 keys.\n2. builds.sr.ht resolves manifest secrets by UUID, not by name — fix is to replace the name with the secret's UUID in .build.yml.\n\nSAME BUG IN THE SIBLING: ~/data/home/sourcehut-compare/.build.yml carries a byte-identical secrets block, so compare.sr.ht's publish is broken the same way and both fix together.","acceptance_criteria":"A push to master produces a build whose publish task succeeds and uploads the .apk to the Garage repo bucket; apk-mirror on phoebe re-indexes it within 15 minutes.","status":"open","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-07-24T21:16:35Z","created_by":"Eugene Blikh","updated_at":"2026-07-24T21:16:35Z"} |
| new_value | {"notes":"DIAGNOSIS RESOLVED 2026-08-05, still blocked on the owner.\n\nPossibility 2 (builds.sr.ht resolves manifest secrets by UUID only) is RULED OUT. buildsrht/manifest.py parses each secrets: entry as a UUID first and, on failure, treats it as a 3-512 character name and looks it up by name. So '- apk-ci-s3' in the manifest is valid as written; nothing in .build.yml needs changing.\n\nPossibility 1 confirmed: 'hut builds secret list' still returns exactly four secrets (agent1-deploy, bencher-api-key, s3-cache-key-secret, s3-cache-key-id) and none is apk-ci-s3. The secret was never created or was deleted.\n\nFailure mechanism, derived rather than read from the log: the task preamble is 'set -xe' and publish's first real line is '. ~/.apk-ci.env'. A missing file makes '.' exit non-zero and set -e kills the task. That reproduces the observed signature exactly - scss/keygen/version/build green, publish red - and matches build #250 (cc90b4a) as well as #247 (61515a5), so the port to the line-numbered diff did not change it.\n\nFIX, both steps need the owner:\n1. Garage, on the host that has the CLI: 'garage key create apk-ci' then 'garage bucket allow --read --write repo --key apk-ci'. No --owner: publish only uploads and never deletes. Check first whether the key behind the existing s3-cache-key-* secrets already has write on the repo bucket.\n2. Register a File secret on builds.srht.bigb.es (web form or GraphQL - 'hut builds secret' is list/share only, it cannot create): name apk-ci-s3, path /home/build/.apk-ci.env, mode 600, contents two lines APK_CI_S3_ACCESS_KEY=... and APK_CI_S3_SECRET_KEY=... . Absolute path rather than ~/.apk-ci.env because the worker writes the literal string it was given; /home/build is the build user's home, so the manifest's '. ~/.apk-ci.env' still resolves. Mode 600 rather than the 644 the existing s3-cache-key-* secrets use: this is a key, not an identifier.\n\nSecrets are account-level, so the same one also fixes sourcehut-compare, whose manifest carries the identical block."} |
| comment | NULL |
| created_at | 2026-08-05T06:04:58Z |
| id | 019fcfe8-8394-769f-9863-afe001048233 |
| issue_id | spec-ejq.2 |
| event_type | closed |
| actor | Eugene Blikh |
| old_value | |
| new_value | Owner created the File secret apk-ci-s3 (/home/build/.apk-ci.env, mode 600) on builds.srht.bigb.es; 'hut builds secret list' now shows it. Resubmit of #250 as build #251 on the same commit cc90b4a: SUCCESS, all five tasks green including publish. VERIFIED: publish runs under 'set -e' and its body is a loop of 'rclone copyto' with an echo per file, so any failed upload would have exited non-zero and failed the task. A green publish is therefore evidence the .apk reached repo/alpine/v3.22/bigbes/x86_64/ — specsrht-0.0.78-r0.apk for this commit. NOT VERIFIED from here: the apk-mirror re-index on phoebe, the second half of the acceptance criteria. s3.bigb.es is not reachable anonymously from this workstation and phoebe host access is blocked (spec-ar4). If the package turns out not to be installable, that is apk-mirror's problem rather than this repo's and wants its own bead. The manifest needed no change: '- apk-ci-s3' resolves by name, as the diagnosis note records. The same account-level secret should also unblock sourcehut-compare, whose .build.yml carries the identical block — its next build will show it. |
| comment | NULL |
| created_at | 2026-08-05T06:12:29Z |
| id | 019fcfef-8f21-7196-a3ab-faabe432786d |
| issue_id | spec-ejq.2 |
| event_type | updated |
| actor | Eugene Blikh |
| old_value | {"id":"spec-ejq.2","title":"CI publish task fails: build secret apk-ci-s3 is missing","description":"Every recent builds.sr.ht run fails at the publish task while every other task passes. Confirmed on jobs #244 (e97532c), #245 (2dc6b71) and #246 (c2dd1ef): 'scss keygen version build' all SUCCESS, 'publish' FAILED. Pre-existing — #244 and #245 predate the Phase 5b work, so no code change caused it. Consequence: the apk never reaches the Garage repo bucket, so apk-mirror on phoebe has nothing new to re-index and the deployed service cannot be upgraded from CI output.","design":"EVIDENCE. .build.yml declares 'secrets: [apk-ci-s3]', documented as a File secret installed at ~/.apk-ci.env carrying APK_CI_S3_ACCESS_KEY / APK_CI_S3_SECRET_KEY for the Garage repo bucket. The publish task's first real line is '. ~/.apk-ci.env'. But 'hut builds secret list' returns exactly four secrets and none is apk-ci-s3: agent1-deploy (SSH key), bencher-api-key, s3-cache-key-secret, s3-cache-key-id. So the file the task sources is never installed.\n\nNOT VERIFIED: the exact failure text. The raw log endpoint (https://builds.srht.bigb.es/query/log/246/publish/log) needs a Bearer token, and reading hut's credential file was correctly refused, so the diagnosis rests on the secret list plus the manifest rather than on the log line itself. Read the log to confirm before acting.\n\nTWO POSSIBILITIES, needs the owner to distinguish:\n1. The secret was deleted or never created — fix is to create a File secret named apk-ci-s3 at ~/.apk-ci.env (mode 600) holding the two S3 keys.\n2. builds.sr.ht resolves manifest secrets by UUID, not by name — fix is to replace the name with the secret's UUID in .build.yml.\n\nSAME BUG IN THE SIBLING: ~/data/home/sourcehut-compare/.build.yml carries a byte-identical secrets block, so compare.sr.ht's publish is broken the same way and both fix together.","acceptance_criteria":"A push to master produces a build whose publish task succeeds and uploads the .apk to the Garage repo bucket; apk-mirror on phoebe re-indexes it within 15 minutes.","notes":"DIAGNOSIS RESOLVED 2026-08-05, still blocked on the owner.\n\nPossibility 2 (builds.sr.ht resolves manifest secrets by UUID only) is RULED OUT. buildsrht/manifest.py parses each secrets: entry as a UUID first and, on failure, treats it as a 3-512 character name and looks it up by name. So '- apk-ci-s3' in the manifest is valid as written; nothing in .build.yml needs changing.\n\nPossibility 1 confirmed: 'hut builds secret list' still returns exactly four secrets (agent1-deploy, bencher-api-key, s3-cache-key-secret, s3-cache-key-id) and none is apk-ci-s3. The secret was never created or was deleted.\n\nFailure mechanism, derived rather than read from the log: the task preamble is 'set -xe' and publish's first real line is '. ~/.apk-ci.env'. A missing file makes '.' exit non-zero and set -e kills the task. That reproduces the observed signature exactly - scss/keygen/version/build green, publish red - and matches build #250 (cc90b4a) as well as #247 (61515a5), so the port to the line-numbered diff did not change it.\n\nFIX, both steps need the owner:\n1. Garage, on the host that has the CLI: 'garage key create apk-ci' then 'garage bucket allow --read --write repo --key apk-ci'. No --owner: publish only uploads and never deletes. Check first whether the key behind the existing s3-cache-key-* secrets already has write on the repo bucket.\n2. Register a File secret on builds.srht.bigb.es (web form or GraphQL - 'hut builds secret' is list/share only, it cannot create): name apk-ci-s3, path /home/build/.apk-ci.env, mode 600, contents two lines APK_CI_S3_ACCESS_KEY=... and APK_CI_S3_SECRET_KEY=... . Absolute path rather than ~/.apk-ci.env because the worker writes the literal string it was given; /home/build is the build user's home, so the manifest's '. ~/.apk-ci.env' still resolves. Mode 600 rather than the 644 the existing s3-cache-key-* secrets use: this is a key, not an identifier.\n\nSecrets are account-level, so the same one also fixes sourcehut-compare, whose manifest carries the identical block.","status":"closed","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-07-24T21:16:35Z","created_by":"Eugene Blikh","updated_at":"2026-08-05T03:12:30Z","closed_at":"2026-08-05T03:12:30Z","close_reason":"Owner created the File secret apk-ci-s3 (/home/build/.apk-ci.env, mode 600) on builds.srht.bigb.es; 'hut builds secret list' now shows it. Resubmit of #250 as build #251 on the same commit cc90b4a: SUCCESS, all five tasks green including publish.\n\nVERIFIED: publish runs under 'set -e' and its body is a loop of 'rclone copyto' with an echo per file, so any failed upload would have exited non-zero and failed the task. A green publish is therefore evidence the .apk reached repo/alpine/v3.22/bigbes/x86_64/ — specsrht-0.0.78-r0.apk for this commit.\n\nNOT VERIFIED from here: the apk-mirror re-index on phoebe, the second half of the acceptance criteria. s3.bigb.es is not reachable anonymously from this workstation and phoebe host access is blocked (spec-ar4). If the package turns out not to be installable, that is apk-mirror's problem rather than this repo's and wants its own bead.\n\nThe manifest needed no change: '- apk-ci-s3' resolves by name, as the diagnosis note records. The same account-level secret should also unblock sourcehut-compare, whose .build.yml carries the identical block — its next build will show it."} |
| new_value | {"notes":"RE-INDEX VERIFIED 2026-08-05, the half the close reason left open.\n\nChecked through repo.bigb.es, the public mirror apk-mirror builds from the Garage repo bucket (URL documented in the APKBUILD header; s3.bigb.es itself resolves to 192.168.88.18 and is only reachable from the LAN, which is why the first attempt to verify failed).\n\n spec.sr.ht-0.0.78-r0.apk 200 content-length 16059260 last-modified 03:09:39 GMT\n APKINDEX.tar.gz 200 last-modified 03:15:00 GMT\n index entry: P:spec.sr.ht V:0.0.78-r0 A:x86_64 S:16059260\n\nThe size in the index matches the object byte for byte, and 0.0.78 is 'git rev-list --count cc90b4a' — the commit build #251 ran on. The index was rebuilt 5.5 minutes after the upload, inside the 15-minute window the acceptance criteria name. Both halves of the criteria are therefore met.\n\nCorrection to the close reason: the package is named spec.sr.ht-0.0.78-r0.apk, not specsrht-0.0.78-r0.apk — pkgname in APKBUILD is spec.sr.ht. The specsrht spelling 404s."} |
| comment | NULL |
| created_at | 2026-08-05T06:20:11Z |
No comments.
Close reason