rgvvf3evjll11gdkvkibfr4vovt6ag25 · 221 rows
| id | issue_id | event_type | actor | old_value | new_value | comment | created_at |
|---|---|---|---|---|---|---|---|
| 000fef98-f0eb-4c85-8118-79fdda5bff24 | tarantool-etcd-4y9 | created | Eugene Blikh | NULL | 2026-05-19T17:54:52Z | ||
| 003ca104-1e2d-4573-ac46-dda795e9a558 | tarantool-etcd-2al | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:26Z |
| 00d9d027-a5b6-4b23-9a55-7ff6dcfc9682 | tarantool-etcd-iij | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:22Z |
| 01265226-b3bd-4381-866a-6dc7a85b54a1 | tarantool-etcd-8zm | created | Eugene Blikh | NULL | 2026-05-19T17:54:13Z | ||
| 01bbab41-c31a-42f2-9802-dca6e420ec4d | tarantool-etcd-bq8 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:32Z |
| 02b3e265-33a8-4e1b-bf2b-f8ee8d47ba31 | tarantool-etcd-3sp | claimed | Eugene Blikh | {"id":"tarantool-etcd-3sp","title":"LeaseKeepAlive fsyncs WAL on every renew","description":"On btrfs SSD bench (2026-05-19 disk run), LeaseKeepAlive/n_X drops from 4 234 ops/s (tmpfs) to 207 ops/s on tarantool while etcd holds at ~7 400 ops/s in both configs.\n\np50 goes from 224 µs (tmpfs) to 4 548 µs (disk) — exactly one btrfs fsync. Root cause at app/etcd/lease.lua:191:\n\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n\nThe leases space is is_sync=true so every keep-alive does a full WAL append + fsync. Etcd's KeepAlive is an in-memory TTL bump with no disk write.\n\nFix sketch: split expiry into an in-memory map (refreshed on every KeepAlive) and persist only on grant/revoke. The expiry fiber should read from the in-memory map. Trade-off: leases may extend further into the future than the on-disk record after a crash, which is fine — clients have to re-keep-alive after reconnect anyway.\n\nSee docs/BENCH.md Linux x86_64 disk section (Open follow-ups).","design":"GOAL: lease keep-alive must not touch the WAL. Mirror etcd's split — durable\nstate = {lease record, granted TTL, attached keys}; ephemeral leader-local\nstate = the expiry deadline (refreshed on every keep-alive, never persisted).\n\n=== STATE MODEL ===\nAdd a module-level in-memory map in app/etcd/lease.lua:\n local deadlines = {} -- [lease_id] = monotonic deadline (clock.monotonic()+ttl)\nThis is leader-local. Followers never keep-alive (the gRPC write gate routes\nLeaseKeepAlive to the leader), and only the leader runs the expiry fiber and\ncan revoke. So the countdown is structurally a leader concern.\n\nThe `leases` space (schema.lua:97-118) stays is_sync=true and keeps its\n{id, ttl, expiry_time, granted_ttl} format. The `expiry_time` FIELD and its\nTREE index (schema.lua:113) become vestigial — written once at grant, never\nthe source of liveness again. Leave them to avoid a space migration; an\noptional follow-up can drop the field+index. Nothing reads expiry_time after\nthis change.\n\n=== HANDLER CHANGES (app/etcd/lease.lua) ===\n1. grant() ~L70: after `box.space.leases:insert(...)`, seed\n `deadlines[id] = now + ttl`. Insert still persists (durable, required).\n2. keepalive() ~L191: REPLACE\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n with\n deadlines[id] = clock.monotonic() + lease.ttl\n Keep the `box.space.leases:get(id)` existence check above it (cheap read,\n no fsync) — it supplies lease.ttl and preserves the NOT_FOUND-raises\n contract the gRPC stream handler depends on (see L169-171).\n3. revoke() ~L150: after `box.space.leases:delete(id)`, clear\n `deadlines[id] = nil`.\n4. time_to_live() ~L236: REPLACE `lease.expiry_time` read with `deadlines[id]`.\n If deadlines[id] is nil (lease loaded but not yet rebuilt), fall back to\n `clock.monotonic() + lease.granted_ttl`. remaining = max(0, floor(dl - now)).\n5. expiry_loop() L287-338: REPLACE the `box.space.leases.index.expiry_time`\n scan (L302-309) with a scan of `deadlines`: collect ids where\n `deadline \u003c= now`. Plain full-table scan is fine (numeric compares in Lua,\n runs every 500ms); a min-heap (etcd's leaseExpiredNotifier shape) is the\n optional optimization if lease counts get large. revoke() already nils the\n map entry on success.\n\n=== REBUILD ON PROMOTE ===\nAdd lease.M.rebuild_deadlines():\n clears `deadlines`, then for each tuple in box.space.leases:pairs() sets\n deadlines[tuple.id] = clock.monotonic() + tuple.ttl\nCall it in the RW branch of the box.watch('box.status', ...) callback in\napp/roles/etcd.lua (L341-353), immediately before lease.start_expiry().\nThis resets every inherited lease's deadline to a full-TTL grace period on the\nnew leader — exactly etcd's failover behavior.\n\n=== LATENT BUG THIS ALSO FIXES ===\nexpiry_time is currently computed as clock.monotonic()+ttl and REPLICATED.\nclock.monotonic() is process-local (relative to boot), so the persisted value\nis meaningless on any other node. Today a promoted replica's fiber scans\ninherited expiry_time values from a different monotonic timeline — could\nexpire leases instantly or never. Rebuild-on-promote with the new leader's own\nclock removes this entirely.","acceptance_criteria":"- keepalive issues ZERO WAL writes: box.info.lsn delta == 0 across N successive\n M.keepalive() calls on the same lease.\n- BenchmarkLeaseKeepAlive/n_* on btrfs SSD (work.lab.local, real-disk run)\n recovers from 207 ops/s to within ~2x of etcd (~4000+ ops/s, near the tmpfs\n baseline). Re-run: TMPDIR=$HOME/bench-data go test -run=^$ \\\n -bench=^BenchmarkLeaseKeepAlive$ -benchtime=5s ./bench/\n- Lease still expires correctly: grant short TTL, no keep-alive, attached keys\n are deleted by the fiber after TTL elapses; keep-alive before TTL prevents it.\n- TimeToLive reflects the latest keep-alive (remaining resets to ~ttl after a\n renew).\n- Promote rebuilds deadlines: a freshly-promoted leader expires inherited\n leases using its own monotonic clock (no instant-expiry, no never-expiry).\n- All existing lease conformance cells (tarantool, tarantool_json,\n tarantool_rs3) and Lua lease tests pass.","notes":"TESTS (add Lua-side tests pinning the fix per the project's regression rule):\n- no-WAL: assert box.info.lsn unchanged across keepalive calls.\n- liveness: keepalive refreshes deadline (TimeToLive resets, lease survives).\n- promote rebuild: use the box.cfg{read_only=true}...{read_only=false}+\n box.ctl.promote() in-process follower-simulation pattern (see role_test.lua\n group role/write_gates_in_process) to verify rebuild_deadlines repopulates.\n- cross-node: optionally exercise via the replicaset harness that a promoted\n replica expires an inherited lease.\nConformance already covers cross-wire correctness; this is about the Lua unit\npins + the bench recovery number.\n\nDOCS: update docs/BENCH.md Linux x86_64 disk section once re-benched; remove\nthe LeaseKeepAlive \"bug\" annotation from the headline table.\n\nSCOPE NOTE: leave the expiry_time field + index in place (no migration). Only\nthe keepalive write path and the fiber's liveness source change.","status":"open","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-05-19T17:33:47Z","created_by":"Eugene Blikh","updated_at":"2026-05-20T04:39:23Z"} | {"assignee":"Eugene Blikh","status":"in_progress"} | NULL | 2026-05-20T15:46:48Z |
| 033a914b-1f3e-4bdb-9553-a81981f72943 | tarantool-etcd-00h | created | Eugene Blikh | NULL | 2026-05-20T09:55:25Z | ||
| 037ee7a8-6029-4854-bf3b-af0456dd52e7 | tarantool-etcd-3sp | closed | Eugene Blikh | keep-alive now bumps a leader-local in-memory deadline map (zero WAL on renew); leases space written only on grant/revoke; rebuild_deadlines on promote. Lua regression tests + TestLease conformance pass. Disk re-bench on btrfs SSD still pending to replace pre-fix BENCH.md numbers. | NULL | 2026-05-20T15:53:33Z | |
| 04008201-ca9f-4fda-8cb6-4ee3ef8888f1 | tarantool-etcd-bol | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:25Z |
| 041bd93e-2d34-42ae-893f-1c75f0090ae5 | tarantool-etcd-axs | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:40Z |
| 054042b2-efb1-4780-906f-ea6461c2bbe0 | tarantool-etcd-5zx | created | Eugene Blikh | NULL | 2026-05-19T17:54:36Z | ||
| 0664fe02-5b7b-4bfe-a708-5db2886620ef | tarantool-etcd-mo5 | created | Eugene Blikh | NULL | 2026-05-19T17:54:26Z | ||
| 07d96215-4a88-46f5-b1c9-b30556edaec5 | tarantool-etcd-70j | created | Eugene Blikh | NULL | 2026-05-19T17:54:28Z | ||
| 0a9851f8-9f2e-4b02-9d0b-c718b271c28f | tarantool-etcd-1qa | created | Eugene Blikh | NULL | 2026-05-20T14:59:03Z | ||
| 0b4eba01-b820-4fd1-b63f-577657999fff | tarantool-etcd-ovt | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:21Z |
| 0bc59661-158a-48bf-b22b-78f0adeefeb1 | tarantool-etcd-tt6 | created | Eugene Blikh | NULL | 2026-05-20T09:17:56Z | ||
| 0ca717eb-443f-4591-a0ea-ad10ca59e7af | tarantool-etcd-4y9 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:09Z |
| 0cb660ac-8609-49d6-979e-b78259436280 | tarantool-etcd-3c2 | created | Eugene Blikh | NULL | 2026-05-19T17:54:23Z | ||
| 0d70b3d7-2b63-4cb3-b36a-076be486997b | tarantool-etcd-ymm | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:13Z |
| 0e0fb1ab-9762-4a05-8ccb-4fc6668a13bd | tarantool-etcd-boq | created | Eugene Blikh | NULL | 2026-05-20T09:45:15Z | ||
| 10f54019-0176-4300-93e3-1266dd4109c5 | tarantool-etcd-3dk | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:34Z |
| 1139004b-5cec-4bd9-b1d9-5ab3bdb9b441 | tarantool-etcd-nwl | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:25Z |
| 11d3eb85-2285-432e-86c0-db17b5ce6c59 | tarantool-etcd-tt6 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:41Z |
| 12575c79-b483-44bc-b2d2-5abd1fc4dca3 | tarantool-etcd-zkg | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:37Z |
| 12ea9ddb-ef88-4b53-877b-f86778387cb6 | tarantool-etcd-3sp | updated | Eugene Blikh | {"id":"tarantool-etcd-3sp","title":"LeaseKeepAlive fsyncs WAL on every renew","description":"On btrfs SSD bench (2026-05-19 disk run), LeaseKeepAlive/n_X drops from 4 234 ops/s (tmpfs) to 207 ops/s on tarantool while etcd holds at ~7 400 ops/s in both configs.\n\np50 goes from 224 µs (tmpfs) to 4 548 µs (disk) — exactly one btrfs fsync. Root cause at app/etcd/lease.lua:191:\n\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n\nThe leases space is is_sync=true so every keep-alive does a full WAL append + fsync. Etcd's KeepAlive is an in-memory TTL bump with no disk write.\n\nFix sketch: split expiry into an in-memory map (refreshed on every KeepAlive) and persist only on grant/revoke. The expiry fiber should read from the in-memory map. Trade-off: leases may extend further into the future than the on-disk record after a crash, which is fine — clients have to re-keep-alive after reconnect anyway.\n\nSee docs/BENCH.md Linux x86_64 disk section (Open follow-ups).","design":"GOAL: lease keep-alive must not touch the WAL. Mirror etcd's split — durable\nstate = {lease record, granted TTL, attached keys}; ephemeral leader-local\nstate = the expiry deadline (refreshed on every keep-alive, never persisted).\n\n=== STATE MODEL ===\nAdd a module-level in-memory map in app/etcd/lease.lua:\n local deadlines = {} -- [lease_id] = monotonic deadline (clock.monotonic()+ttl)\nThis is leader-local. Followers never keep-alive (the gRPC write gate routes\nLeaseKeepAlive to the leader), and only the leader runs the expiry fiber and\ncan revoke. So the countdown is structurally a leader concern.\n\nThe `leases` space (schema.lua:97-118) stays is_sync=true and keeps its\n{id, ttl, expiry_time, granted_ttl} format. The `expiry_time` FIELD and its\nTREE index (schema.lua:113) become vestigial — written once at grant, never\nthe source of liveness again. Leave them to avoid a space migration; an\noptional follow-up can drop the field+index. Nothing reads expiry_time after\nthis change.\n\n=== HANDLER CHANGES (app/etcd/lease.lua) ===\n1. grant() ~L70: after `box.space.leases:insert(...)`, seed\n `deadlines[id] = now + ttl`. Insert still persists (durable, required).\n2. keepalive() ~L191: REPLACE\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n with\n deadlines[id] = clock.monotonic() + lease.ttl\n Keep the `box.space.leases:get(id)` existence check above it (cheap read,\n no fsync) — it supplies lease.ttl and preserves the NOT_FOUND-raises\n contract the gRPC stream handler depends on (see L169-171).\n3. revoke() ~L150: after `box.space.leases:delete(id)`, clear\n `deadlines[id] = nil`.\n4. time_to_live() ~L236: REPLACE `lease.expiry_time` read with `deadlines[id]`.\n If deadlines[id] is nil (lease loaded but not yet rebuilt), fall back to\n `clock.monotonic() + lease.granted_ttl`. remaining = max(0, floor(dl - now)).\n5. expiry_loop() L287-338: REPLACE the `box.space.leases.index.expiry_time`\n scan (L302-309) with a scan of `deadlines`: collect ids where\n `deadline \u003c= now`. Plain full-table scan is fine (numeric compares in Lua,\n runs every 500ms); a min-heap (etcd's leaseExpiredNotifier shape) is the\n optional optimization if lease counts get large. revoke() already nils the\n map entry on success.\n\n=== REBUILD ON PROMOTE ===\nAdd lease.M.rebuild_deadlines():\n clears `deadlines`, then for each tuple in box.space.leases:pairs() sets\n deadlines[tuple.id] = clock.monotonic() + tuple.ttl\nCall it in the RW branch of the box.watch('box.status', ...) callback in\napp/roles/etcd.lua (L341-353), immediately before lease.start_expiry().\nThis resets every inherited lease's deadline to a full-TTL grace period on the\nnew leader — exactly etcd's failover behavior.\n\n=== LATENT BUG THIS ALSO FIXES ===\nexpiry_time is currently computed as clock.monotonic()+ttl and REPLICATED.\nclock.monotonic() is process-local (relative to boot), so the persisted value\nis meaningless on any other node. Today a promoted replica's fiber scans\ninherited expiry_time values from a different monotonic timeline — could\nexpire leases instantly or never. Rebuild-on-promote with the new leader's own\nclock removes this entirely.","status":"open","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-05-19T17:33:47Z","created_by":"Eugene Blikh","updated_at":"2026-05-20T04:39:03Z"} | {"acceptance_criteria":"- keepalive issues ZERO WAL writes: box.info.lsn delta == 0 across N successive\n M.keepalive() calls on the same lease.\n- BenchmarkLeaseKeepAlive/n_* on btrfs SSD (work.lab.local, real-disk run)\n recovers from 207 ops/s to within ~2x of etcd (~4000+ ops/s, near the tmpfs\n baseline). Re-run: TMPDIR=$HOME/bench-data go test -run=^$ \\\n -bench=^BenchmarkLeaseKeepAlive$ -benchtime=5s ./bench/\n- Lease still expires correctly: grant short TTL, no keep-alive, attached keys\n are deleted by the fiber after TTL elapses; keep-alive before TTL prevents it.\n- TimeToLive reflects the latest keep-alive (remaining resets to ~ttl after a\n renew).\n- Promote rebuilds deadlines: a freshly-promoted leader expires inherited\n leases using its own monotonic clock (no instant-expiry, no never-expiry).\n- All existing lease conformance cells (tarantool, tarantool_json,\n tarantool_rs3) and Lua lease tests pass.","notes":"TESTS (add Lua-side tests pinning the fix per the project's regression rule):\n- no-WAL: assert box.info.lsn unchanged across keepalive calls.\n- liveness: keepalive refreshes deadline (TimeToLive resets, lease survives).\n- promote rebuild: use the box.cfg{read_only=true}...{read_only=false}+\n box.ctl.promote() in-process follower-simulation pattern (see role_test.lua\n group role/write_gates_in_process) to verify rebuild_deadlines repopulates.\n- cross-node: optionally exercise via the replicaset harness that a promoted\n replica expires an inherited lease.\nConformance already covers cross-wire correctness; this is about the Lua unit\npins + the bench recovery number.\n\nDOCS: update docs/BENCH.md Linux x86_64 disk section once re-benched; remove\nthe LeaseKeepAlive \"bug\" annotation from the headline table.\n\nSCOPE NOTE: leave the expiry_time field + index in place (no migration). Only\nthe keepalive write path and the fiber's liveness source change."} | NULL | 2026-05-20T07:39:22Z |
| 13b0fa86-dc00-4254-8b3a-ac9cac94eb76 | tarantool-etcd-w6z | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:12Z |
| 15b1c0c8-8f99-4d3f-a5d7-63353458b43a | tarantool-etcd-zws | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:42Z |
| 1632bb64-4a7b-4e1c-b45e-a87309d9a8db | tarantool-etcd-uee | created | Eugene Blikh | NULL | 2026-05-20T09:17:54Z | ||
| 17215ff6-20bb-436c-9606-d550d82020a1 | tarantool-etcd-ak9 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:29Z |
| 18fe139b-df94-451a-8e87-a2ca72e657e0 | tarantool-etcd-ymm | created | Eugene Blikh | NULL | 2026-05-19T17:54:41Z | ||
| 1933ad02-e1f6-4f3f-be72-bbf884ed28d8 | tarantool-etcd-m0w | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:22Z |
| 1bb60270-a6bd-466f-b41e-3c248f446b21 | tarantool-etcd-wqn | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:17Z |
| 1ce593c2-ac19-43ae-b8a2-3334751c95dd | tarantool-etcd-nwl | created | Eugene Blikh | NULL | 2026-05-19T17:54:11Z | ||
| 1d7bbef0-a625-4b7e-9459-5d59c8029828 | tarantool-etcd-w8q | created | Eugene Blikh | NULL | 2026-05-19T17:53:51Z | ||
| 1d8a485e-0bb1-4902-8690-ecd2e4d36a43 | tarantool-etcd-929 | created | Eugene Blikh | NULL | 2026-05-19T17:54:30Z | ||
| 1e73c28b-b088-4d06-bbaa-f1311828aae2 | tarantool-etcd-b08 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m1-conformance | 2026-05-21T14:37:02Z |
| 1fa4e29e-2f25-4419-9818-77e0f4af47b2 | tarantool-etcd-9z2 | created | Eugene Blikh | NULL | 2026-05-19T16:34:06Z | ||
| 24b879c4-8ab9-434b-98fa-8347bd20bdfc | tarantool-etcd-0uw | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:34Z |
| 260ce364-1139-4df6-bb2c-ac7a3c50117f | tarantool-etcd-dkr | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:36Z |
| 2921ede2-b70c-44f4-8e52-54255702370a | tarantool-etcd-dmt | created | Eugene Blikh | NULL | 2026-05-19T17:54:53Z | ||
| 2d2828f8-6cf0-4859-9db8-430ae1f7a9d7 | tarantool-etcd-g1a | created | Eugene Blikh | NULL | 2026-05-19T17:54:25Z | ||
| 2d6bf251-1a2a-458e-92dd-4746e289448a | tarantool-etcd-70j | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:18Z |
| 2e9cfbea-e2f5-4e74-8f33-41ce4f2a9370 | tarantool-etcd-iz4 | created | Eugene Blikh | NULL | 2026-05-19T17:54:15Z | ||
| 30ab4f56-2ac0-447d-abdf-b2e0975adb7a | tarantool-etcd-zc7 | created | Eugene Blikh | NULL | 2026-05-20T17:33:30Z | ||
| 323218d7-991c-4baa-9e61-32ec89198c10 | tarantool-etcd-7kb | created | Eugene Blikh | NULL | 2026-05-19T17:54:18Z | ||
| 348e1509-080d-42db-8a04-d6ce767366f7 | tarantool-etcd-tf0 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:14Z |
| 353c3076-8b72-42ef-8fb5-ef1f700df311 | tarantool-etcd-ovt | created | Eugene Blikh | NULL | 2026-05-19T17:54:21Z | ||
| 35af2e9c-c033-4f03-8b56-efe92ec8b347 | tarantool-etcd-0uw | created | Eugene Blikh | NULL | 2026-05-19T17:53:54Z | ||
| 3690d19b-6bf7-4ee3-b57b-858ee5a8f937 | tarantool-etcd-3sp | created | Eugene Blikh | NULL | 2026-05-19T20:33:47Z | ||
| 36a3b9e9-2035-470e-ab66-3380d4d56902 | tarantool-etcd-0gv | created | Eugene Blikh | NULL | 2026-05-19T17:54:09Z |