goivt866tor942349eqg5v810ikd0m1p · 263 rows
| id | issue_id | event_type | actor | old_value | new_value | comment | created_at |
|---|---|---|---|---|---|---|---|
| 83edfeeb-7a05-588a-a6c0-a8ef5541250f | tarantool-etcd-8ss | updated | Eugene Blikh | {"id":"tarantool-etcd-8ss","title":"Example 2: 3-node EE replicaset self-hosting its own etcd config (file → etcd migration)","description":"Add examples/tarantool-ee-self-hosted-etcd: boot a 3-instance EE replicaset (election failover) from a local cluster-config.yaml where every node runs app.roles.etcd (replicated KV store, read_pref=any). Seed the cluster's OWN etcd store with that config, then rolling-restart each instance to bootstrap from TT_CONFIG_ETCD_* (no --config). Document the cold-boot circularity: self-hosted etcd-config is HA under rolling restart but needs the local file retained as a cold-boot seed. Justfile + README + cluster-config.yaml.","notes":"v2: hardened migrate after EE run hit 'etcdserver: no leader' on the 2nd node (election churn from restarting a node that became leader). Fixes: (1) recompute leader each step, always migrate a follower, leader last; (2) _wait-healthy gate (all live nodes agree on one leader) between steps; (3) endpoints peers-first/self-last; (4) append logs not truncate. Rolling-restart pacing + leader-pick logic verified on Community; EE etcd-source fetch still needs an EE re-run to confirm.","status":"in_progress","priority":2,"issue_type":"feature","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-20T11:16:17Z","created_by":"Eugene Blikh","updated_at":"2026-05-20T11:30:21Z","started_at":"2026-05-20T11:16:28Z"} | {"notes":"VERIFIED end-to-end on EE 3.7.0 (up-file/seed/migrate/probe/reload all pass; migrate idempotent). Root cause of the user's failure: EE config.etcd source pins to Endpoints[0] without failover, so endpoints must be LEADER-FIRST (not peers-first). Also fixed: leader-migrated-last via per-step recompute, _wait-healthy gate, and a pipefail bug (grep -vx on last node exited 1 after work was done). Example complete."} | NULL | 2026-05-20T14:38:01Z |
| 847644fe-7529-5ad5-bcb6-92f0a6fd85f8 | tarantool-etcd-0gv | created | Eugene Blikh | NULL | 2026-05-19T17:54:09Z | ||
| 853285f2-6823-5330-9d41-95dd7170c742 | tarantool-etcd-hlx | created | Eugene Blikh | NULL | 2026-05-20T09:17:54Z | ||
| 86b724e1-e608-5db6-8a69-feee0bdf517f | tarantool-etcd-dkr | created | Eugene Blikh | NULL | 2026-05-20T09:55:25Z | ||
| 889a2887-a179-54bc-94e9-4de55b4f5878 | tarantool-etcd-97v | created | Eugene Blikh | NULL | 2026-05-19T17:54:39Z | ||
| 89b6b91c-c9fc-5c2d-9e08-cc773ef75cab | tarantool-etcd-hlx | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:42Z |
| 89cebc94-e850-5726-9f4f-65137f5cdf7b | tarantool-etcd-9cd | created | Eugene Blikh | NULL | 2026-05-19T17:53:58Z | ||
| 8a497a80-f880-5439-b9d3-49c8842bfdc5 | tarantool-etcd-nwl | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:25Z |
| 8a6c8a11-52de-5f00-8098-4654ce3b08c9 | tarantool-etcd-bg4 | created | Eugene Blikh | NULL | 2026-05-19T17:54:14Z | ||
| 8adefbb6-838b-5fb5-9704-dbe7ad4fa000 | tarantool-etcd-16d | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m2-production | 2026-05-21T14:37:06Z |
| 8baa8026-2fcb-526d-9ab3-7b726c3cc4aa | tarantool-etcd-6lj | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:33Z |
| 8c7cf48d-8e9b-503d-b15e-6d58014b6ff3 | tarantool-etcd-wqn | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:17Z |
| 8e61e59e-dfbe-5aad-a6cc-52102b681d64 | tarantool-etcd-8ss | updated | Eugene Blikh | {"id":"tarantool-etcd-8ss","title":"Example 2: 3-node EE replicaset self-hosting its own etcd config (file → etcd migration)","description":"Add examples/tarantool-ee-self-hosted-etcd: boot a 3-instance EE replicaset (election failover) from a local cluster-config.yaml where every node runs app.roles.etcd (replicated KV store, read_pref=any). Seed the cluster's OWN etcd store with that config, then rolling-restart each instance to bootstrap from TT_CONFIG_ETCD_* (no --config). Document the cold-boot circularity: self-hosted etcd-config is HA under rolling restart but needs the local file retained as a cold-boot seed. Justfile + README + cluster-config.yaml.","notes":"Files written + phases 1-2 (up-file/seed/probe/config-show) verified on Community Tarantool 3.8. Ports 3501-3503/2382-2482-2582 (distinct from example-1's 3401/2379). migrate+reload need an EE binary (config.etcd source) so were not run here; mechanics mirror the working tarantool-ee-config example. EE guard on migrate verified to fire.","status":"in_progress","priority":2,"issue_type":"feature","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-20T11:16:17Z","created_by":"Eugene Blikh","updated_at":"2026-05-20T11:22:45Z","started_at":"2026-05-20T11:16:28Z"} | {"notes":"v2: hardened migrate after EE run hit 'etcdserver: no leader' on the 2nd node (election churn from restarting a node that became leader). Fixes: (1) recompute leader each step, always migrate a follower, leader last; (2) _wait-healthy gate (all live nodes agree on one leader) between steps; (3) endpoints peers-first/self-last; (4) append logs not truncate. Rolling-restart pacing + leader-pick logic verified on Community; EE etcd-source fetch still needs an EE re-run to confirm."} | NULL | 2026-05-20T14:30:21Z |
| 90cfe1cf-1797-5900-a648-fe00f7daf2e9 | tarantool-etcd-584 | created | Eugene Blikh | NULL | 2026-05-19T17:53:53Z | ||
| 94d2a36e-9234-56bc-94d4-d270d98a5406 | tarantool-etcd-ag2 | created | Eugene Blikh | NULL | 2026-05-19T17:54:06Z | ||
| 952d547e-ebbe-59cf-a86b-e9e86f2d6247 | tarantool-etcd-cpp | created | Eugene Blikh | NULL | 2026-05-20T09:55:23Z | ||
| 98857065-d100-532e-ab3d-27bb94b54874 | tarantool-etcd-rzh | created | Eugene Blikh | NULL | 2026-05-19T17:54:32Z | ||
| 9a29c48a-d378-5092-b327-0c780e240291 | tarantool-etcd-m0w | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:22Z |
| 9a5fc12c-bf33-5aab-bdc1-d2f6e5eb97a6 | tarantool-etcd-mo5 | created | Eugene Blikh | NULL | 2026-05-19T17:54:26Z | ||
| 9a644bac-cc82-5a2a-9e31-87b48d0cf6b1 | tarantool-etcd-8zm | created | Eugene Blikh | NULL | 2026-05-19T17:54:13Z | ||
| 9b54496f-698d-5441-9792-4a43fe667ec0 | tarantool-etcd-9ar | created | Eugene Blikh | NULL | 2026-05-19T17:54:46Z | ||
| 9c78d1b3-2262-522e-a6d3-f20350d2d32f | tarantool-etcd-3sp | created | Eugene Blikh | NULL | 2026-05-19T20:33:47Z | ||
| 9cb6edf1-455e-520a-840c-ff16a8096025 | tarantool-etcd-nbs | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m2-production | 2026-05-21T14:37:03Z |
| 9e0b2dec-e482-5f79-9afc-16f48f226f38 | tarantool-etcd-ns0 | created | Eugene Blikh | NULL | 2026-05-19T17:54:34Z | ||
| 9e6cc6c5-a58c-5f74-98e8-c7d01f49122b | tarantool-etcd-b08 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m1-conformance | 2026-05-21T14:37:02Z |
| 9e747415-a28e-52e3-96e7-f2583fb0298f | tarantool-etcd-h8r | created | Eugene Blikh | NULL | 2026-05-19T17:54:50Z | ||
| 9f8323da-55e4-588c-ac88-f376bb20465c | tarantool-etcd-zws | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m5-clustering | 2026-05-21T14:37:42Z |
| a1fe4827-336f-5b8e-8e43-d5a7897ef0b2 | tarantool-etcd-1tw | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m2-production | 2026-05-21T14:37:04Z |
| a26ca896-81a3-543f-ba50-c18cf5ab6fe7 | tarantool-etcd-sqv | created | Eugene Blikh | NULL | 2026-05-19T17:54:45Z | ||
| a4ae3524-e546-5518-beee-94665d4cf440 | tarantool-etcd-tf0 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:14Z |
| a4cfdcdc-aa9e-570c-946e-f8316818fbb4 | tarantool-etcd-zc7 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m2-production | 2026-05-21T14:37:05Z |
| a63820a2-175a-507d-a475-0097ed02d089 | tarantool-etcd-h8r | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:09Z |
| a6df7449-e43a-55bf-8556-5e76bb05c35d | tarantool-etcd-070 | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:32Z |
| a6f20dc1-2e9d-5529-ad9e-05c455839104 | tarantool-etcd-cz5 | created | Eugene Blikh | NULL | 2026-05-19T17:54:41Z | ||
| a96bdeea-9d8f-5d5c-a2fb-80c9dba72457 | tarantool-etcd-w5r | created | Eugene Blikh | NULL | 2026-05-19T17:54:43Z | ||
| adf1fce8-4640-5c8d-b90c-4300167ebd09 | tarantool-etcd-70j | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:18Z |
| ae83a00a-6439-5966-9658-40e00aa4292c | tarantool-etcd-ozm | created | Eugene Blikh | NULL | 2026-05-19T17:54:08Z | ||
| af3a800c-52ef-5ba7-b274-dccbc4d6fc31 | tarantool-etcd-uee | created | Eugene Blikh | NULL | 2026-05-20T09:17:54Z | ||
| b48a75f5-2a8c-5d7f-8385-897dfe00e67a | tarantool-etcd-m0w | created | Eugene Blikh | NULL | 2026-05-19T17:54:17Z | ||
| b4c20140-96dd-5fde-a7aa-9197af66b5a1 | tarantool-etcd-95d | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m3-performance | 2026-05-21T14:37:06Z |
| b5988b8e-2fc1-505a-a8d6-e20273e68e9f | tarantool-etcd-0gv | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:26Z |
| b626f9f2-b874-5719-bb14-2ce0d8ae9e87 | tarantool-etcd-3sp | updated | Eugene Blikh | {"id":"tarantool-etcd-3sp","title":"LeaseKeepAlive fsyncs WAL on every renew","description":"On btrfs SSD bench (2026-05-19 disk run), LeaseKeepAlive/n_X drops from 4 234 ops/s (tmpfs) to 207 ops/s on tarantool while etcd holds at ~7 400 ops/s in both configs.\n\np50 goes from 224 µs (tmpfs) to 4 548 µs (disk) — exactly one btrfs fsync. Root cause at app/etcd/lease.lua:191:\n\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n\nThe leases space is is_sync=true so every keep-alive does a full WAL append + fsync. Etcd's KeepAlive is an in-memory TTL bump with no disk write.\n\nFix sketch: split expiry into an in-memory map (refreshed on every KeepAlive) and persist only on grant/revoke. The expiry fiber should read from the in-memory map. Trade-off: leases may extend further into the future than the on-disk record after a crash, which is fine — clients have to re-keep-alive after reconnect anyway.\n\nSee docs/BENCH.md Linux x86_64 disk section (Open follow-ups).","status":"open","priority":2,"issue_type":"bug","owner":"bigbes@gmail.com","created_at":"2026-05-19T17:33:47Z","created_by":"Eugene Blikh","updated_at":"2026-05-19T17:33:47Z"} | {"design":"GOAL: lease keep-alive must not touch the WAL. Mirror etcd's split — durable\nstate = {lease record, granted TTL, attached keys}; ephemeral leader-local\nstate = the expiry deadline (refreshed on every keep-alive, never persisted).\n\n=== STATE MODEL ===\nAdd a module-level in-memory map in app/etcd/lease.lua:\n local deadlines = {} -- [lease_id] = monotonic deadline (clock.monotonic()+ttl)\nThis is leader-local. Followers never keep-alive (the gRPC write gate routes\nLeaseKeepAlive to the leader), and only the leader runs the expiry fiber and\ncan revoke. So the countdown is structurally a leader concern.\n\nThe `leases` space (schema.lua:97-118) stays is_sync=true and keeps its\n{id, ttl, expiry_time, granted_ttl} format. The `expiry_time` FIELD and its\nTREE index (schema.lua:113) become vestigial — written once at grant, never\nthe source of liveness again. Leave them to avoid a space migration; an\noptional follow-up can drop the field+index. Nothing reads expiry_time after\nthis change.\n\n=== HANDLER CHANGES (app/etcd/lease.lua) ===\n1. grant() ~L70: after `box.space.leases:insert(...)`, seed\n `deadlines[id] = now + ttl`. Insert still persists (durable, required).\n2. keepalive() ~L191: REPLACE\n box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})\n with\n deadlines[id] = clock.monotonic() + lease.ttl\n Keep the `box.space.leases:get(id)` existence check above it (cheap read,\n no fsync) — it supplies lease.ttl and preserves the NOT_FOUND-raises\n contract the gRPC stream handler depends on (see L169-171).\n3. revoke() ~L150: after `box.space.leases:delete(id)`, clear\n `deadlines[id] = nil`.\n4. time_to_live() ~L236: REPLACE `lease.expiry_time` read with `deadlines[id]`.\n If deadlines[id] is nil (lease loaded but not yet rebuilt), fall back to\n `clock.monotonic() + lease.granted_ttl`. remaining = max(0, floor(dl - now)).\n5. expiry_loop() L287-338: REPLACE the `box.space.leases.index.expiry_time`\n scan (L302-309) with a scan of `deadlines`: collect ids where\n `deadline \u003c= now`. Plain full-table scan is fine (numeric compares in Lua,\n runs every 500ms); a min-heap (etcd's leaseExpiredNotifier shape) is the\n optional optimization if lease counts get large. revoke() already nils the\n map entry on success.\n\n=== REBUILD ON PROMOTE ===\nAdd lease.M.rebuild_deadlines():\n clears `deadlines`, then for each tuple in box.space.leases:pairs() sets\n deadlines[tuple.id] = clock.monotonic() + tuple.ttl\nCall it in the RW branch of the box.watch('box.status', ...) callback in\napp/roles/etcd.lua (L341-353), immediately before lease.start_expiry().\nThis resets every inherited lease's deadline to a full-TTL grace period on the\nnew leader — exactly etcd's failover behavior.\n\n=== LATENT BUG THIS ALSO FIXES ===\nexpiry_time is currently computed as clock.monotonic()+ttl and REPLICATED.\nclock.monotonic() is process-local (relative to boot), so the persisted value\nis meaningless on any other node. Today a promoted replica's fiber scans\ninherited expiry_time values from a different monotonic timeline — could\nexpire leases instantly or never. Rebuild-on-promote with the new leader's own\nclock removes this entirely."} | NULL | 2026-05-20T07:39:02Z |
| b6403489-01a6-5c93-bbf0-3d5892d7b3af | tarantool-etcd-lac | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m1-conformance | 2026-05-21T14:37:01Z |
| b6a99f2f-e9cc-5ff5-b1a8-807e832435f4 | tarantool-etcd-3c2 | created | Eugene Blikh | NULL | 2026-05-19T17:54:23Z | ||
| bb43477e-f05e-5f24-91b4-804a62100a69 | tarantool-etcd-35y | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m1-conformance | 2026-05-21T14:37:01Z |
| bb9963f5-e76e-5f17-b327-b7365865108b | tarantool-etcd-9lj | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:28Z |
| bbee3101-7104-5281-be2d-363a697a6cd9 | tarantool-etcd-1dk | created | Eugene Blikh | NULL | 2026-05-20T09:55:21Z | ||
| bbf149ce-2ac9-50c0-bdff-56477c0f2177 | tarantool-etcd-tf0 | created | Eugene Blikh | NULL | 2026-05-19T17:54:38Z | ||
| be2c2f97-97ea-57a7-bf62-190f95e0d293 | tarantool-etcd-acp | label_added | Eugene Blikh | NULL | NULL | Added label: milestone:m4-iproto-parity | 2026-05-21T14:37:35Z |
| bee57122-e8b8-5849-b1f6-8ad55f71dff9 | tarantool-etcd-7kb | created | Eugene Blikh | NULL | 2026-05-19T17:54:18Z |