~bigbes/tarantool-etcd · parade

main · last commit 1 month ago · 2npkec0r

← Back to the parade

tarantool-etcd-7ey Replica route: connect as read-only replica to an existing etcd cluster Lined Up

status: open P2 feature milestone:m5-clustering
bd update tarantool-etcd-7ey --claim bd close tarantool-etcd-7ey
Created byEugene Blikh
Ownerbigbes@gmail.com
Created2026-05-20T05:48:15Z
Updated2026-07-17T14:27:36Z
Description
Add a mode where this server attaches to an existing/real etcd cluster as a read-only replica (mirror), serving reads locally while transparently keeping in sync with the upstream etcd.

Motivation: lets the Tarantool store act as a read scale-out / local cache in front of a real etcd deployment, and is a stepping stone toward migration (shadow the production cluster, compare, then cut over).

Design questions to resolve:
- Sync mechanism: use etcd's Watch (clientv3) from revision 0 to stream the full keyspace + tail live, persisting events into our etcd_* spaces and advancing our revision counter to match upstream. Need an initial Range snapshot (or Snapshot RPC) for bootstrap, then Watch with progress notifications + compaction handling (ErrCompacted -> re-snapshot).
- Revision alignment: our global revision sequence must track upstream's mod_revision exactly so reads return etcd-consistent revisions. This constrains the local revision allocator while in replica mode.
- Write handling: reject writes locally with a redirect/error, OR proxy them to the upstream leader and wait for the change to come back over the watch (read-your-writes). Pick one; proxying is closer to etcd's learner/proxy semantics.
- Lease/Watch/Auth: leases and auth state also need mirroring or pass-through. Scope what's in v1 (KV-only read replica) vs later.
- Config: new role config block (e.g. replica.upstream_endpoints, replica.mode = mirror|proxy, credentials).

Likely a multi-phase epic; this ticket captures the umbrella. Split bootstrap-snapshot, watch-tail-sync, revision-alignment, and write-proxying into child issues once the design is settled.
Notes
Reuses clientv3 Watch semantics we already understand from conformance work (ErrCompacted=11, progress notify). Consider relation to the Tarantool-native replicaset path (tarantool_rs3) — this is the opposite direction (follow a foreign etcd, not our own Raft).

Depends on

  • tarantool-etcd-16d — Implement Maintenance.Snapshot RPC (streaming DB snapshot) related open

Depended on by

Unblocks — everything waiting on this, transitively

  • tarantool-etcd-zws — Replica route: revision alignment with upstream parent-child open
  • tarantool-etcd-hlx — Replica route: watch-tail live sync blocks open
  • tarantool-etcd-75u — Replica route: write handling (reject vs proxy-to-leader) blocks open
  • tarantool-etcd-uee — Replica route: bootstrap snapshot import parent-child open
  • tarantool-etcd-tt6 — Replica route: lease / auth state mirroring (scope v1 vs later) parent-child open
  • tarantool-etcd-jg3 — Replica route: config block + role plumbing parent-child open
  • tarantool-etcd-cnj — [epic] Raft-learner replica — join an etcd cluster as a non-voting member related open
  • tarantool-etcd-umc — Learner: config block + role plumbing parent-child open
  • tarantool-etcd-axs — Learner: rafthttp peer transport over nghttp2 blocks open
  • tarantool-etcd-boq — Learner: raft state machine (follower/learner subset) blocks open
  • tarantool-etcd-6js — Learner: apply loop — InternalRaftRequest -> existing handlers blocks open
  • tarantool-etcd-q5f — Learner: snapshot receive (bbolt parse + bucket translation) blocks open
  • tarantool-etcd-zkg — Learner: conformance — join real etcd cluster, assert log replication blocks open
  • tarantool-etcd-ovo — Learner: durable applied_index (WAL-equivalent for restart) blocks open
  • tarantool-etcd-4yb — Learner: raftpb + InternalRaftRequest proto schemas parent-child open
  • tarantool-etcd-6zn — Learner: MemberAddAsLearner handshake + cluster version negotiation parent-child open

No comments.

Stored rows — what this pane was built from, as read
issues 1 row
id tarantool-etcd-7ey
content_hash be7b04f2cbf862eb22cfe43df8812d0bcb20c7b247f8edc485850c3fb44cbe34
title Replica route: connect as read-only replica to an existing etcd cluster
description Add a mode where this server attaches to an existing/real etcd cluster as a read-only replica (mirror), serving reads locally while transparently keeping in sync with the upstream etcd. Motivation: lets the Tarantool store act as a read scale-out / local cache in front of a real etcd deployment, and is a stepping stone toward migration (shadow the production cluster, compare, then cut over). Design questions to resolve: - Sync mechanism: use etcd's Watch (clientv3) from revision 0 to stream the full keyspace + tail live, persisting events into our etcd_* spaces and advancing our revision counter to match upstream. Need an initial Range snapshot (or Snapshot RPC) for bootstrap, then Watch with progress notifications + compaction handling (ErrCompacted -> re-snapshot). - Revision alignment: our global revision sequence must track upstream's mod_revision exactly so reads return etcd-consistent revisions. This constrains the local revision allocator while in replica mode. - Write handling: reject writes locally with a redirect/error, OR proxy them to the upstream leader and wait for the change to come back over the watch (read-your-writes). Pick one; proxying is closer to etcd's learner/proxy semantics. - Lease/Watch/Auth: leases and auth state also need mirroring or pass-through. Scope what's in v1 (KV-only read replica) vs later. - Config: new role config block (e.g. replica.upstream_endpoints, replica.mode = mirror|proxy, credentials). Likely a multi-phase epic; this ticket captures the umbrella. Split bootstrap-snapshot, watch-tail-sync, revision-alignment, and write-proxying into child issues once the design is settled.
design
acceptance_criteria
notes Reuses clientv3 Watch semantics we already understand from conformance work (ErrCompacted=11, progress notify). Consider relation to the Tarantool-native replicaset path (tarantool_rs3) — this is the opposite direction (follow a foreign etcd, not our own Raft).
status open
priority 2
issue_type feature
assignee NULL
estimated_minutes NULL
created_at 2026-05-20T05:48:15Z
created_by Eugene Blikh
owner bigbes@gmail.com
updated_at 2026-07-17T14:27:36Z
closed_at NULL
closed_by_session
external_ref NULL
spec_id
compaction_level 0
compacted_at NULL
compacted_at_commit NULL
original_size NULL
sender
ephemeral 0
wisp_type
pinned 0
is_template 0
mol_type
work_type
source_system
metadata <binary>
source_repo
close_reason
event_kind
actor
target
payload
await_type
await_id
timeout_ns 0
waiters
hook_bead
role_bead
agent_state
last_activity NULL
role_type
rig
due_at NULL
defer_until NULL
no_history 0
started_at NULL
is_blocked 0
labels 1 row
issue_id tarantool-etcd-7ey
label milestone:m5-clustering
dependencies 8 rows
id 0ec497b5-b954-52f2-853f-28088b75001e
issue_id tarantool-etcd-zws
type parent-child
created_at 2026-05-20T09:18:15Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-7ey
depends_on_wisp_id NULL
depends_on_external NULL
id 48652ac8-6e0e-5561-a798-7f3235f74ba3
issue_id tarantool-etcd-75u
type parent-child
created_at 2026-05-20T09:18:16Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-7ey
depends_on_wisp_id NULL
depends_on_external NULL
id 5a814ebe-5453-5f01-9812-89df538a0702
issue_id tarantool-etcd-uee
type parent-child
created_at 2026-05-20T09:18:14Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-7ey
depends_on_wisp_id NULL
depends_on_external NULL
id 5f327e65-dfdc-54aa-9495-f350a54e3af9
issue_id tarantool-etcd-tt6
type parent-child
created_at 2026-05-20T09:18:16Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-7ey
depends_on_wisp_id NULL
depends_on_external NULL
id 80a097c8-8dcf-53ed-9a90-909694bcb010
issue_id tarantool-etcd-jg3
type parent-child
created_at 2026-05-20T09:18:13Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-7ey
depends_on_wisp_id NULL
depends_on_external NULL
id 8d82773f-f842-5a59-b8cd-0485e5c180f0
issue_id tarantool-etcd-7ey
type related
created_at 2026-05-20T09:18:20Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-16d
depends_on_wisp_id NULL
depends_on_external NULL
id 9192386c-c056-5244-abe5-cbbb0bede32b
issue_id tarantool-etcd-cnj
type related
created_at 2026-05-20T09:47:29Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-7ey
depends_on_wisp_id NULL
depends_on_external NULL
id e59ef424-b68e-5f66-a10f-627c82f52942
issue_id tarantool-etcd-hlx
type parent-child
created_at 2026-05-20T09:18:15Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-7ey
depends_on_wisp_id NULL
depends_on_external NULL
events 2 rows
id 704155e7-0669-5bfa-8777-08c8a01e0a06
issue_id tarantool-etcd-7ey
event_type created
actor Eugene Blikh
old_value
new_value
comment NULL
created_at 2026-05-20T08:48:14Z
id d0d26b03-32e3-55e5-88d4-eff502e16069
issue_id tarantool-etcd-7ey
event_type label_added
actor Eugene Blikh
old_value NULL
new_value NULL
comment Added label: milestone:m5-clustering
created_at 2026-05-21T14:37:43Z