tarantool-etcd-3sp LeaseKeepAlive fsyncs WAL on every renew Past Stand

On btrfs SSD bench (2026-05-19 disk run), LeaseKeepAlive/n_X drops from 4 234 ops/s (tmpfs) to 207 ops/s on tarantool while etcd holds at ~7 400 ops/s in both configs.

p50 goes from 224 µs (tmpfs) to 4 548 µs (disk) — exactly one btrfs fsync. Root cause at app/etcd/lease.lua:191:

    box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})

The leases space is is_sync=true so every keep-alive does a full WAL append + fsync. Etcd's KeepAlive is an in-memory TTL bump with no disk write.

Fix sketch: split expiry into an in-memory map (refreshed on every KeepAlive) and persist only on grant/revoke. The expiry fiber should read from the in-memory map. Trade-off: leases may extend further into the future than the on-disk record after a crash, which is fine — clients have to re-keep-alive after reconnect anyway.

See docs/BENCH.md Linux x86_64 disk section (Open follow-ups).

GOAL: lease keep-alive must not touch the WAL. Mirror etcd's split — durable
state = {lease record, granted TTL, attached keys}; ephemeral leader-local
state = the expiry deadline (refreshed on every keep-alive, never persisted).

=== STATE MODEL ===
Add a module-level in-memory map in app/etcd/lease.lua:
    local deadlines = {}   -- [lease_id] = monotonic deadline (clock.monotonic()+ttl)
This is leader-local. Followers never keep-alive (the gRPC write gate routes
LeaseKeepAlive to the leader), and only the leader runs the expiry fiber and
can revoke. So the countdown is structurally a leader concern.

The `leases` space (schema.lua:97-118) stays is_sync=true and keeps its
{id, ttl, expiry_time, granted_ttl} format. The `expiry_time` FIELD and its
TREE index (schema.lua:113) become vestigial — written once at grant, never
the source of liveness again. Leave them to avoid a space migration; an
optional follow-up can drop the field+index. Nothing reads expiry_time after
this change.

=== HANDLER CHANGES (app/etcd/lease.lua) ===
1. grant() ~L70: after `box.space.leases:insert(...)`, seed
   `deadlines[id] = now + ttl`. Insert still persists (durable, required).
2. keepalive() ~L191: REPLACE
       box.space.leases:update(id, {{'=', 'expiry_time', now + lease.ttl}})
   with
       deadlines[id] = clock.monotonic() + lease.ttl
   Keep the `box.space.leases:get(id)` existence check above it (cheap read,
   no fsync) — it supplies lease.ttl and preserves the NOT_FOUND-raises
   contract the gRPC stream handler depends on (see L169-171).
3. revoke() ~L150: after `box.space.leases:delete(id)`, clear
   `deadlines[id] = nil`.
4. time_to_live() ~L236: REPLACE `lease.expiry_time` read with `deadlines[id]`.
   If deadlines[id] is nil (lease loaded but not yet rebuilt), fall back to
   `clock.monotonic() + lease.granted_ttl`. remaining = max(0, floor(dl - now)).
5. expiry_loop() L287-338: REPLACE the `box.space.leases.index.expiry_time`
   scan (L302-309) with a scan of `deadlines`: collect ids where
   `deadline <= now`. Plain full-table scan is fine (numeric compares in Lua,
   runs every 500ms); a min-heap (etcd's leaseExpiredNotifier shape) is the
   optional optimization if lease counts get large. revoke() already nils the
   map entry on success.

=== REBUILD ON PROMOTE ===
Add lease.M.rebuild_deadlines():
    clears `deadlines`, then for each tuple in box.space.leases:pairs() sets
    deadlines[tuple.id] = clock.monotonic() + tuple.ttl
Call it in the RW branch of the box.watch('box.status', ...) callback in
app/roles/etcd.lua (L341-353), immediately before lease.start_expiry().
This resets every inherited lease's deadline to a full-TTL grace period on the
new leader — exactly etcd's failover behavior.

=== LATENT BUG THIS ALSO FIXES ===
expiry_time is currently computed as clock.monotonic()+ttl and REPLICATED.
clock.monotonic() is process-local (relative to boot), so the persisted value
is meaningless on any other node. Today a promoted replica's fiber scans
inherited expiry_time values from a different monotonic timeline — could
expire leases instantly or never. Rebuild-on-promote with the new leader's own
clock removes this entirely.

- keepalive issues ZERO WAL writes: box.info.lsn delta == 0 across N successive
  M.keepalive() calls on the same lease.
- BenchmarkLeaseKeepAlive/n_* on btrfs SSD (work.lab.local, real-disk run)
  recovers from 207 ops/s to within ~2x of etcd (~4000+ ops/s, near the tmpfs
  baseline). Re-run: TMPDIR=$HOME/bench-data go test -run=^$ \
  -bench=^BenchmarkLeaseKeepAlive$ -benchtime=5s ./bench/
- Lease still expires correctly: grant short TTL, no keep-alive, attached keys
  are deleted by the fiber after TTL elapses; keep-alive before TTL prevents it.
- TimeToLive reflects the latest keep-alive (remaining resets to ~ttl after a
  renew).
- Promote rebuilds deadlines: a freshly-promoted leader expires inherited
  leases using its own monotonic clock (no instant-expiry, no never-expiry).
- All existing lease conformance cells (tarantool, tarantool_json,
  tarantool_rs3) and Lua lease tests pass.

TESTS (add Lua-side tests pinning the fix per the project's regression rule):
- no-WAL: assert box.info.lsn unchanged across keepalive calls.
- liveness: keepalive refreshes deadline (TimeToLive resets, lease survives).
- promote rebuild: use the box.cfg{read_only=true}...{read_only=false}+
  box.ctl.promote() in-process follower-simulation pattern (see role_test.lua
  group role/write_gates_in_process) to verify rebuild_deadlines repopulates.
- cross-node: optionally exercise via the replicaset harness that a promoted
  replica expires an inherited lease.
Conformance already covers cross-wire correctness; this is about the Lua unit
pins + the bench recovery number.

DOCS: update docs/BENCH.md Linux x86_64 disk section once re-benched; remove
the LeaseKeepAlive "bug" annotation from the headline table.

SCOPE NOTE: leave the expiry_time field + index in place (no migration). Only
the keepalive write path and the fiber's liveness source change.