gnlb1p1g1jqt8pmvprvb83bvmpl66qld · 281 rows
| id | issue_id | event_type | actor | old_value | new_value | comment | created_at |
|---|---|---|---|---|---|---|---|
| e2ea8dd7-f559-5778-8039-5ac01d6c9c02 | tarantool-etcd-3sp | claimed | Eugene Blikh | {"id":"tarantool-etcd-3sp","title":"LeaseKeepAlive fsyncs WAL on every renew","description":"On btrfs SSD bench (2026-05-19 disk run), LeaseKeepAlive/n_X drops from 4 234 ops/s (tmpfs) to 207 ops/s on tarantool while etcd holds at ~7 400 ops/s in both configs.\n\np50 goes from 224 µs (tmpfs) to 4 548 µs (disk) — exactly one btrfs fsync. Root cause at app/etcd/lease.lua:191:\n\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n\nThe leases space is is_sync=true so every keep-alive does a full WAL append + fsync. Etcd's KeepAlive is an in-memory TTL bump with no disk write.\n\nFix sketch: split expiry into an in-memory map (refreshed on every KeepAlive) and persist only on grant/revoke. The expiry fiber should read from the in-memory map. Trade-off: leases may extend further into the future than the on-disk record after a crash, which is fine — clients have to re-keep-alive after reconnect anyway.\n\nSee docs/BENCH.md Linux x86_64 disk section (Open follow-ups).","design":"GOAL: lease keep-alive must not touch the WAL. Mirror etcd's split — durable\nstate = {lease record, granted TTL, attached keys}; ephemeral leader-local\nstate = the expiry deadline (refreshed on every keep-alive, never persisted).\n\n=== STATE MODEL ===\nAdd a module-level in-memory map in app/etcd/lease.lua:\n local deadlines = {} -- [lease_id] = monotonic deadline (clock.monotonic()+ttl)\nThis is leader-local. Followers never keep-alive (the gRPC write gate routes\nLeaseKeepAlive to the leader), and only the leader runs the expiry fiber and\ncan revoke. So the countdown is structurally a leader concern.\n\nThe `leases` space (schema.lua:97-118) stays is_sync=true and keeps its\n{id, ttl, expiry_time, granted_ttl} format. The `expiry_time` FIELD and its\nTREE index (schema.lua:113) become vestigial — written once at grant, never\nthe source of liveness again. Leave them to avoid a space migration; an\noptional follow-up can drop the field+index. Nothing reads expiry_time after\nthis change.\n\n=== HANDLER CHANGES (app/etcd/lease.lua) ===\n1. grant() ~L70: after `box.space.leases:insert(...)`, seed\n `deadlines[id] = now + ttl`. Insert still persists (durable, required).\n2. keepalive() ~L191: REPLACE\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n with\n deadlines[id] = clock.monotonic() + lease.ttl\n Keep the `box.space.leases:get(id)` existence check above it (cheap read,\n no fsync) — it supplies lease.ttl and preserves the NOT_FOUND-raises\n contract the gRPC stream handler depends on (see L169-171).\n3. revoke() ~L150: after `box.space.leases:delete(id)`, clear\n `deadlines[id] = nil`.\n4. time_to_live() ~L236: REPLACE `lease.expiry_time` read with `deadlines[id]`.\n If deadlines[id] is nil (lease loaded but not yet rebuilt), fall back to\n `clock.monotonic() + lease.granted_ttl`. remaining = max(0, floor(dl - now)).\n5. expiry_loop() L287-338: REPLACE the `box.space.leases.index.expiry_time`\n scan (L302-309) with a scan of `deadlines`: collect ids where\n `deadline \u003c= now`. Plain full-table scan is fine (numeric compares in Lua,\n runs every 500ms); a min-heap (etcd's leaseExpiredNotifier shape) is the\n optional optimization if lease counts get large. revoke() already nils the\n map entry on success.\n\n=== REBUILD ON PROMOTE ===\nAdd lease.M.rebuild_deadlines():\n clears `deadlines`, then for each tuple in box.space.leases:pairs() sets\n deadlines[tuple.id] = clock.monotonic() + tuple.ttl\nCall it in the RW branch of the box.watch('box.status', ...) callback in\napp/roles/etcd.lua (L341-353), immediately before lease.start_expiry().\nThis resets every inherited lease's deadline to a full-TTL grace period on the\nnew leader — exactly etcd's failover behavior.\n\n=== LATENT BUG THIS ALSO FIXES ===\nexpiry_time is currently computed as clock.monotonic()+ttl and REPLICATED.\nclock.monotonic() is process-local (relative to boot), so the persisted value\nis meaningless on any other node. Today a promoted replica's fiber scans\ninherited expiry_time values from a different monotonic timeline — could\nexpire leases instantly or never. Rebuild-on-promote with the new leader's own\nclock removes this entirely.","acceptance_criteria":"- keepalive issues ZERO WAL writes: box.info.lsn delta == 0 across N successive\n M.keepalive() calls on the same lease.\n- BenchmarkLeaseKeepAlive/n_* on btrfs SSD (work.lab.local, real-disk run)\n recovers from 207 ops/s to within ~2x of etcd (~4000+ ops/s, near the tmpfs\n baseline). Re-run: TMPDIR=$HOME/bench-data go test -run=^$ \\\n -bench=^BenchmarkLeaseKeepAlive$ -benchtime=5s ./bench/\n- Lease still expires correctly: grant short TTL, no keep-alive, attached keys\n are deleted by the fiber after TTL elapses; keep-alive before TTL prevents it.\n- TimeToLive reflects the latest keep-alive (remaining resets to ~ttl after a\n renew).\n- Promote rebuilds deadlines: a freshly-promoted leader expires inherited\n leases using its own monotonic clock (no instant-expiry, no never-expiry).\n- All existing lease conformance cells (tarantool, tarantool_json,\n tarantool_rs3) and Lua lease tests pass.","notes":"TESTS (add Lua-side tests pinning the fix per the project's regression rule):\n- no-WAL: assert box.info.lsn unchanged across keepalive calls.\n- liveness: keepalive refreshes deadline (TimeToLive resets, lease survives).\n- promote rebuild: use the box.cfg{read_only=true}...{read_only=false}+\n box.ctl.promote() in-process follower-simulation pattern (see role_test.lua\n group role/write_gates_in_process) to verify rebuild_deadlines repopulates.\n- cross-node: optionally exercise via the replicaset harness that a promoted\n replica expires an inherited lease.\nConformance already covers cross-wire correctness; this is about the Lua unit\npins + the bench recovery number.\n\nDOCS: update docs/BENCH.md Linux x86_64 disk section once re-benched; remove\nthe LeaseKeepAlive \"bug\" annotation from the headline table.\n\nSCOPE NOTE: leave the expiry_time field + index in place (no migration). Only\nthe keepalive write path and the fiber's liveness source change.","status":"open","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-05-19T17:33:47Z","created_by":"Eugene Blikh","updated_at":"2026-05-20T04:39:23Z"} | {"assignee":"Eugene Blikh","status":"in_progress"} | NULL | 2026-05-20T15:46:48Z |
| e315dfd4-c556-5638-b364-ccc96f555f16 | tarantool-etcd-q5f | created | Eugene Blikh | NULL | 2026-05-20T09:45:16Z | ||
| e372b6b6-bab8-579f-885b-293bd18626bd | tarantool-etcd-sqv | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:11Z |
| e37aab6e-321e-57d3-97e4-8e18c945c1ed | tarantool-etcd-lko | created | Eugene Blikh | NULL | 2026-05-19T16:33:55Z | ||
| e3c93650-9f69-5bcd-8a50-a33ca65f94d2 | tarantool-etcd-boq | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:39Z |
| e3ed2e0c-31c5-5daf-b59c-634dfe7dd393 | tarantool-etcd-5zx | created | Eugene Blikh | NULL | 2026-05-19T17:54:36Z | ||
| e87c8639-bf6c-59e6-a0db-efc65dc782d1 | tarantool-etcd-bg4 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:24Z |
| e8e98e27-de06-534a-9e96-9a4910f0be8a | tarantool-etcd-q5f | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:38Z |
| ea70ee7c-2c41-5c31-b6be-90692e614fea | tarantool-etcd-rzs | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:20Z |
| ead456d0-04f2-59a4-b2e7-b47f8b774fba | tarantool-etcd-j1c | created | Eugene Blikh | NULL | 2026-05-19T16:34:30Z | ||
| ebb46a77-efa9-5b7f-85aa-646fcfe81283 | tarantool-etcd-234 | created | Eugene Blikh | NULL | 2026-05-19T16:34:48Z | ||
| ee32ba1e-b4d6-540f-8ebf-71f286b42572 | tarantool-etcd-4nr | created | Eugene Blikh | NULL | 2026-05-19T17:54:34Z | ||
| ef03b433-1420-5d38-9680-be91c5e23b69 | tarantool-etcd-cz5 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:13Z |
| f0328c15-10f2-5434-8a8c-7a424450f1b9 | tarantool-etcd-70j | created | Eugene Blikh | NULL | 2026-05-19T17:54:28Z | ||
| f09c9c57-f1a2-5a14-8e2a-f80de1b83eb5 | tarantool-etcd-ovo | created | Eugene Blikh | NULL | 2026-05-20T09:45:16Z | ||
| f1c73b0e-924d-598f-9e31-d3ee02a668fe | tarantool-etcd-dmt | created | Eugene Blikh | NULL | 2026-05-19T17:54:53Z | ||
| f1e6ab12-ea35-5761-98ca-82912cc66d6a | tarantool-etcd-iz4 | created | Eugene Blikh | NULL | 2026-05-19T17:54:15Z | ||
| f2a55eeb-dee3-5fa0-a797-9d61093a17d1 | tarantool-etcd-iij | created | Eugene Blikh | NULL | 2026-05-19T17:54:17Z | ||
| f36e61cc-3c12-5d06-9e31-677f317c7f48 | tarantool-etcd-4yb | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:39Z |
| f406968b-245f-5f2c-a1b0-3f685d35ca2e | tarantool-etcd-ymm | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:13Z |
| f4c50ff4-bdf0-59b0-9c26-a47a86bae33c | tarantool-etcd-50g | created | Eugene Blikh | NULL | 2026-05-19T17:54:19Z | ||
| f4d21671-525a-5220-a05c-c1995711cf53 | tarantool-etcd-sdt | created | Eugene Blikh | NULL | 2026-05-19T17:54:04Z | ||
| f51e719b-a9b1-5618-a0d2-a78364520e2d | tarantool-etcd-buq | created | Eugene Blikh | NULL | 2026-05-19T16:34:52Z | ||
| f6c16d22-40b7-5e74-a023-795af0d3cec4 | tarantool-etcd-070 | created | Eugene Blikh | NULL | 2026-05-19T17:53:57Z | ||
| f818cea8-4f11-58a8-8d4d-7d8dd9f31f73 | tarantool-etcd-6js | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:38Z |
| f844987c-3d5a-58a7-a9a6-60656da06368 | tarantool-etcd-929 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:17Z |
| fa8820da-f652-52d9-b51f-d1bc3b91188c | tarantool-etcd-6js | created | Eugene Blikh | NULL | 2026-05-20T09:45:15Z | ||
| fb52a936-d403-5d89-badf-57d3eb672312 | tarantool-etcd-4nr | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:16Z |
| fc0e86f3-9055-5ad3-b09a-e3b39ee88916 | tarantool-etcd-w5r | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:12Z |
| fd59ef07-aa39-54e5-be57-e6a91ec68917 | tarantool-etcd-8zm | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:24Z |
| ff28f6d2-ac5d-531f-8b0d-55482ef6b024 | tarantool-etcd-hco | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:10Z |