ej4o835l773jfncr0fda1u0j79fhtu4s · 251 rows
| id | issue_id | event_type | actor | old_value | new_value | comment | created_at |
|---|---|---|---|---|---|---|---|
| 211f17a2-763e-5f08-aead-3a2196ee8788 | tarantool-etcd-4y9 | created | Eugene Blikh | NULL | 2026-05-19T17:54:52Z | ||
| 21fc93e6-3bcd-5f7b-acec-745bcb0f2d5b | tarantool-etcd-60x | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:31Z |
| 224c2a43-8db1-5966-b1e2-17e6c11f33f9 | tarantool-etcd-9z2 | created | Eugene Blikh | NULL | 2026-05-19T16:34:06Z | ||
| 24d3ebd3-a7ca-50b2-ae1f-34e8da8c0a5c | tarantool-etcd-584 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:34Z |
| 273bc102-02cd-51b4-bc8b-ba8ef983a914 | tarantool-etcd-gz4 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:35Z |
| 27b58acb-29e5-5525-9be1-9c1c3f97a393 | tarantool-etcd-unw | created | Eugene Blikh | NULL | 2026-05-20T09:55:22Z | ||
| 2a557ed5-9b04-528e-b7c0-c84d21b905cb | tarantool-etcd-4y9 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:09Z |
| 2a560375-ce81-57d9-a89d-55f8c0ad3def | tarantool-etcd-3dk | created | Eugene Blikh | NULL | 2026-05-19T17:53:55Z | ||
| 2ab8ffc9-93c7-596a-8b09-0fa771897bd0 | tarantool-etcd-a4v | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:30Z |
| 2af6d317-371a-57d1-a6eb-738530e6ba88 | tarantool-etcd-00h | created | Eugene Blikh | NULL | 2026-05-20T09:55:25Z | ||
| 2d2921da-ee80-5cc0-a09c-d4f77d9f9c99 | tarantool-etcd-bq8 | created | Eugene Blikh | NULL | 2026-05-19T17:53:59Z | ||
| 2e423a8e-65ef-54e2-bd3c-328e5aa11ea5 | tarantool-etcd-zc7 | created | Eugene Blikh | NULL | 2026-05-20T17:33:30Z | ||
| 2e4bb86d-cb02-5103-b235-512b574a636c | tarantool-etcd-3sp | updated | Eugene Blikh | {"id":"tarantool-etcd-3sp","title":"LeaseKeepAlive fsyncs WAL on every renew","description":"On btrfs SSD bench (2026-05-19 disk run), LeaseKeepAlive/n_X drops from 4 234 ops/s (tmpfs) to 207 ops/s on tarantool while etcd holds at ~7 400 ops/s in both configs.\n\np50 goes from 224 µs (tmpfs) to 4 548 µs (disk) — exactly one btrfs fsync. Root cause at app/etcd/lease.lua:191:\n\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n\nThe leases space is is_sync=true so every keep-alive does a full WAL append + fsync. Etcd's KeepAlive is an in-memory TTL bump with no disk write.\n\nFix sketch: split expiry into an in-memory map (refreshed on every KeepAlive) and persist only on grant/revoke. The expiry fiber should read from the in-memory map. Trade-off: leases may extend further into the future than the on-disk record after a crash, which is fine — clients have to re-keep-alive after reconnect anyway.\n\nSee docs/BENCH.md Linux x86_64 disk section (Open follow-ups).","design":"GOAL: lease keep-alive must not touch the WAL. Mirror etcd's split — durable\nstate = {lease record, granted TTL, attached keys}; ephemeral leader-local\nstate = the expiry deadline (refreshed on every keep-alive, never persisted).\n\n=== STATE MODEL ===\nAdd a module-level in-memory map in app/etcd/lease.lua:\n local deadlines = {} -- [lease_id] = monotonic deadline (clock.monotonic()+ttl)\nThis is leader-local. Followers never keep-alive (the gRPC write gate routes\nLeaseKeepAlive to the leader), and only the leader runs the expiry fiber and\ncan revoke. So the countdown is structurally a leader concern.\n\nThe `leases` space (schema.lua:97-118) stays is_sync=true and keeps its\n{id, ttl, expiry_time, granted_ttl} format. The `expiry_time` FIELD and its\nTREE index (schema.lua:113) become vestigial — written once at grant, never\nthe source of liveness again. Leave them to avoid a space migration; an\noptional follow-up can drop the field+index. Nothing reads expiry_time after\nthis change.\n\n=== HANDLER CHANGES (app/etcd/lease.lua) ===\n1. grant() ~L70: after `box.space.leases:insert(...)`, seed\n `deadlines[id] = now + ttl`. Insert still persists (durable, required).\n2. keepalive() ~L191: REPLACE\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n with\n deadlines[id] = clock.monotonic() + lease.ttl\n Keep the `box.space.leases:get(id)` existence check above it (cheap read,\n no fsync) — it supplies lease.ttl and preserves the NOT_FOUND-raises\n contract the gRPC stream handler depends on (see L169-171).\n3. revoke() ~L150: after `box.space.leases:delete(id)`, clear\n `deadlines[id] = nil`.\n4. time_to_live() ~L236: REPLACE `lease.expiry_time` read with `deadlines[id]`.\n If deadlines[id] is nil (lease loaded but not yet rebuilt), fall back to\n `clock.monotonic() + lease.granted_ttl`. remaining = max(0, floor(dl - now)).\n5. expiry_loop() L287-338: REPLACE the `box.space.leases.index.expiry_time`\n scan (L302-309) with a scan of `deadlines`: collect ids where\n `deadline \u003c= now`. Plain full-table scan is fine (numeric compares in Lua,\n runs every 500ms); a min-heap (etcd's leaseExpiredNotifier shape) is the\n optional optimization if lease counts get large. revoke() already nils the\n map entry on success.\n\n=== REBUILD ON PROMOTE ===\nAdd lease.M.rebuild_deadlines():\n clears `deadlines`, then for each tuple in box.space.leases:pairs() sets\n deadlines[tuple.id] = clock.monotonic() + tuple.ttl\nCall it in the RW branch of the box.watch('box.status', ...) callback in\napp/roles/etcd.lua (L341-353), immediately before lease.start_expiry().\nThis resets every inherited lease's deadline to a full-TTL grace period on the\nnew leader — exactly etcd's failover behavior.\n\n=== LATENT BUG THIS ALSO FIXES ===\nexpiry_time is currently computed as clock.monotonic()+ttl and REPLICATED.\nclock.monotonic() is process-local (relative to boot), so the persisted value\nis meaningless on any other node. Today a promoted replica's fiber scans\ninherited expiry_time values from a different monotonic timeline — could\nexpire leases instantly or never. Rebuild-on-promote with the new leader's own\nclock removes this entirely.","status":"open","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-05-19T17:33:47Z","created_by":"Eugene Blikh","updated_at":"2026-05-20T04:39:03Z"} | {"acceptance_criteria":"- keepalive issues ZERO WAL writes: box.info.lsn delta == 0 across N successive\n M.keepalive() calls on the same lease.\n- BenchmarkLeaseKeepAlive/n_* on btrfs SSD (work.lab.local, real-disk run)\n recovers from 207 ops/s to within ~2x of etcd (~4000+ ops/s, near the tmpfs\n baseline). Re-run: TMPDIR=$HOME/bench-data go test -run=^$ \\\n -bench=^BenchmarkLeaseKeepAlive$ -benchtime=5s ./bench/\n- Lease still expires correctly: grant short TTL, no keep-alive, attached keys\n are deleted by the fiber after TTL elapses; keep-alive before TTL prevents it.\n- TimeToLive reflects the latest keep-alive (remaining resets to ~ttl after a\n renew).\n- Promote rebuilds deadlines: a freshly-promoted leader expires inherited\n leases using its own monotonic clock (no instant-expiry, no never-expiry).\n- All existing lease conformance cells (tarantool, tarantool_json,\n tarantool_rs3) and Lua lease tests pass.","notes":"TESTS (add Lua-side tests pinning the fix per the project's regression rule):\n- no-WAL: assert box.info.lsn unchanged across keepalive calls.\n- liveness: keepalive refreshes deadline (TimeToLive resets, lease survives).\n- promote rebuild: use the box.cfg{read_only=true}...{read_only=false}+\n box.ctl.promote() in-process follower-simulation pattern (see role_test.lua\n group role/write_gates_in_process) to verify rebuild_deadlines repopulates.\n- cross-node: optionally exercise via the replicaset harness that a promoted\n replica expires an inherited lease.\nConformance already covers cross-wire correctness; this is about the Lua unit\npins + the bench recovery number.\n\nDOCS: update docs/BENCH.md Linux x86_64 disk section once re-benched; remove\nthe LeaseKeepAlive \"bug\" annotation from the headline table.\n\nSCOPE NOTE: leave the expiry_time field + index in place (no migration). Only\nthe keepalive write path and the fiber's liveness source change."} | NULL | 2026-05-20T07:39:22Z |
| 2f276f77-e202-54ef-b062-c93aca3cecad | tarantool-etcd-1qa | created | Eugene Blikh | NULL | 2026-05-20T14:59:03Z | ||
| 2fe0aa28-0e64-5022-9f40-fef3762e3705 | tarantool-etcd-cpp | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m3-performance | 2026-05-21T14:37:07Z |
| 304f1a99-c12b-53c8-803f-894d5ac24900 | tarantool-etcd-929 | created | Eugene Blikh | NULL | 2026-05-19T17:54:30Z | ||
| 3114e9cd-ad31-5a26-95ec-70009b03eeb7 | tarantool-etcd-9z2 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m1-conformance | 2026-05-21T14:37:00Z |
| 3150db32-8d23-59b8-a3a2-5a1acf8acc54 | tarantool-etcd-mo5 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:19Z |
| 315e7be9-6d93-5ed9-bee8-f5ac6acb6334 | tarantool-etcd-ovo | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:38Z |
| 335be1e1-e2fa-55d9-a88e-79e9d69f207c | tarantool-etcd-rzr | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m3-performance | 2026-05-21T14:37:07Z |
| 33f49e68-307a-5917-a329-d9bb120ee9e2 | tarantool-etcd-234 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m3-performance | 2026-05-21T14:37:08Z |
| 34cff267-d3fc-5b04-bf47-4e2a07d58c9f | tarantool-etcd-2al | created | Eugene Blikh | NULL | 2026-05-19T17:54:10Z | ||
| 35aa9bee-3b46-59df-b047-ef80eef443b7 | tarantool-etcd-1tw | created | Eugene Blikh | NULL | 2026-05-20T09:55:24Z | ||
| 36264f30-a142-5446-9a0a-fc667a3601fc | tarantool-etcd-wgx | created | Eugene Blikh | NULL | 2026-05-19T17:54:05Z | ||
| 37ceff1e-fe86-5447-aab6-15210d342963 | tarantool-etcd-p3s | created | Eugene Blikh | NULL | 2026-05-19T17:54:48Z | ||
| 3cc6dd53-fbe3-5180-a694-3fcbc8684f40 | tarantool-etcd-hx9 | created | Eugene Blikh | NULL | 2026-05-19T17:54:20Z | ||
| 3cd7fec2-670c-53d6-b11f-c9dbeb2faa3a | tarantool-etcd-2al | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:26Z |
| 3d79b61c-2c21-53f2-8af2-dcb7bb5b41bd | tarantool-etcd-sdt | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:29Z |
| 3f4aac09-c0ef-5377-aa03-5b782054e6fb | tarantool-etcd-w8q | created | Eugene Blikh | NULL | 2026-05-19T17:53:51Z | ||
| 42330dca-3276-54ef-a503-e0aec9131e67 | tarantool-etcd-bdf | created | Eugene Blikh | NULL | 2026-05-20T09:55:23Z | ||
| 45d5d964-749f-56d7-8ba9-d88173c9a77d | tarantool-etcd-ag2 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:28Z |
| 46b35be7-0e7f-5aca-8d20-68f0ff1f5443 | tarantool-etcd-ovt | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:21Z |
| 4761d06f-59a7-587a-b2d6-8cdbb4851b59 | tarantool-etcd-ovt | created | Eugene Blikh | NULL | 2026-05-19T17:54:21Z | ||
| 479cc8dd-39c7-5ba4-a6fa-b17f0bdcca8d | tarantool-etcd-9ar | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:11Z |
| 4a87379f-a73c-5791-afac-b8308b910495 | tarantool-etcd-75u | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:41Z |
| 4ba9cdce-cb73-5307-a9b8-9a06c65164a8 | tarantool-etcd-ak9 | created | Eugene Blikh | NULL | 2026-05-19T17:54:03Z | ||
| 4d6ef7c8-62fe-5bd6-b1be-e6df257e14df | tarantool-etcd-zws | created | Eugene Blikh | NULL | 2026-05-20T09:17:55Z | ||
| 4db8d2e2-61a9-577a-b51d-119d81373640 | tarantool-etcd-6zn | created | Eugene Blikh | NULL | 2026-05-20T09:45:17Z | ||
| 506ed28f-9180-50a4-ab10-01aa27b336d1 | tarantool-etcd-75u | created | Eugene Blikh | NULL | 2026-05-20T09:17:55Z | ||
| 507a4a75-e35d-55e0-b260-f16ea0358519 | tarantool-etcd-6lj | created | Eugene Blikh | NULL | 2026-05-19T17:53:56Z | ||
| 51b856a5-a491-563f-914d-e70e230bb677 | tarantool-etcd-b08 | created | Eugene Blikh | NULL | 2026-05-19T16:34:09Z | ||
| 531a6daa-fde6-50fe-b5b4-b23e6d7e6100 | tarantool-etcd-95d | created | Eugene Blikh | NULL | 2026-05-20T09:55:37Z | ||
| 53c332fc-a538-51d7-b1a9-06ed5c7832dc | tarantool-etcd-2pa | created | Eugene Blikh | NULL | 2026-05-19T17:54:12Z | ||
| 562727d7-f99a-5fb7-bdb2-463b81081d68 | tarantool-etcd-0us | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:16Z |
| 565f9e78-a5c0-5cd7-b626-25c094ae1ccd | tarantool-etcd-bq8 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:32Z |
| 57e9f746-bc4b-567a-9dfa-44135e1de10f | tarantool-etcd-828 | created | Eugene Blikh | NULL | 2026-05-19T17:54:27Z | ||
| 57fcddfd-d5cf-5a74-96f4-4b6cdb2fe6ff | tarantool-etcd-1dk | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m2-production | 2026-05-21T14:37:04Z |
| 598319eb-265f-5fb3-b190-c440192bf0f6 | tarantool-etcd-ns0 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:15Z |
| 5a299bdd-bac1-5ec3-8339-a59146f35cd0 | tarantool-etcd-bol | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:25Z |
| 5b853ccb-dea0-59b3-a793-3e0ef66d3f55 | tarantool-etcd-jg3 | created | Eugene Blikh | NULL | 2026-05-20T09:17:53Z |