Beads — bigbes/tarantool-etcd

~bigbes/tarantool-etcd · parade

main · last commit 1 month ago · 2npkec0r

`tarantool-etcd-cnj` [epic] Raft-learner replica — join an etcd cluster as a non-voting member Lined Up

status: open P2 feature milestone:m5-clustering

bd update tarantool-etcd-cnj --claim bd close tarantool-etcd-cnj

Created by	Eugene Blikh
Owner	bigbes@gmail.com
Created	2026-05-20T06:39:59Z
Updated	2026-07-17T14:27:36Z

Description

Attach this server to a real etcd cluster as a Raft *learner* (non-voting member), receiving the replicated log over etcd's internal peer protocol (rafthttp) and applying it into our etcd_* spaces. Unlike the Watch-mirror route (tarantool-etcd-7ey), a learner is in the cluster's membership view (MemberList shows it), sees auth/lease/cluster/alarm state in the log automatically, and avoids the Watch compaction-resync gap.

Cost is ~60-70% of full-member work, NOT 20%: the election state machine, proposal path, leader responsibilities, and snapshot *send* drop off, but the rafthttp transport, log applier, and snapshot *receive* (bbolt parse) all remain. We track etcd-INTERNAL protos (raftpb.Message, etcdserverpb.InternalRaftRequest) — stable-ish but not a public contract; release cadence binds to upstream etcd versions.

Decision context (chat 2026-05-20): user explicitly chose the learner path over the Watch mirror. This epic is the alternative to 7ey, not a child of it.

Phasing is learner-first and bottom-up: config -> transport -> proto decode -> state machine -> apply loop -> snapshot receive -> durability -> membership handshake -> conformance. Split into child issues below.

Notes

Surface map: peer endpoints /raft, /raft/stream/message/<id>, /raft/stream/msgapp/<id>, /raft/snapshot on port 2380. Reuses our nghttp2 HTTP/2 layer for the transport and our pure-Lua pb engine for the gogoproto schemas. Easiest correctness oracle: tests/conformance harness can SpawnEtcd then MemberAddAsLearner us.

Depends on

tarantool-etcd-7ey — Replica route: connect as read-only replica to an existing etcd cluster related open

Depended on by

tarantool-etcd-umc — Learner: config block + role plumbing parent-child
tarantool-etcd-q5f — Learner: snapshot receive (bbolt parse + bucket translation) parent-child
tarantool-etcd-boq — Learner: raft state machine (follower/learner subset) parent-child
tarantool-etcd-axs — Learner: rafthttp peer transport over nghttp2 parent-child
tarantool-etcd-4yb — Learner: raftpb + InternalRaftRequest proto schemas parent-child
tarantool-etcd-ovo — Learner: durable applied_index (WAL-equivalent for restart) parent-child
tarantool-etcd-zkg — Learner: conformance — join real etcd cluster, assert log replication parent-child
tarantool-etcd-6zn — Learner: MemberAddAsLearner handshake + cluster version negotiation parent-child
tarantool-etcd-6js — Learner: apply loop — InternalRaftRequest -> existing handlers parent-child

Prerequisite chain — everything this waits on, transitively

tarantool-etcd-7ey — Replica route: connect as read-only replica to an existing etcd cluster related open
tarantool-etcd-16d — Implement Maintenance.Snapshot RPC (streaming DB snapshot) related open

Unblocks — everything waiting on this, transitively

tarantool-etcd-umc — Learner: config block + role plumbing parent-child open
tarantool-etcd-axs — Learner: rafthttp peer transport over nghttp2 blocks open
tarantool-etcd-boq — Learner: raft state machine (follower/learner subset) blocks open
tarantool-etcd-6js — Learner: apply loop — InternalRaftRequest -> existing handlers blocks open
tarantool-etcd-q5f — Learner: snapshot receive (bbolt parse + bucket translation) blocks open
tarantool-etcd-zkg — Learner: conformance — join real etcd cluster, assert log replication blocks open
tarantool-etcd-ovo — Learner: durable applied_index (WAL-equivalent for restart) blocks open
tarantool-etcd-4yb — Learner: raftpb + InternalRaftRequest proto schemas parent-child open
tarantool-etcd-6zn — Learner: MemberAddAsLearner handshake + cluster version negotiation parent-child open

Comments History

No comments.

Eugene Blikh created the issue · 2026-05-20T09:39:58Z
Eugene Blikh added subtask tarantool-etcd-umc · 2026-05-20T09:47:12Z
Eugene Blikh added subtask tarantool-etcd-q5f · 2026-05-20T09:47:29Z
Eugene Blikh added subtask tarantool-etcd-boq · 2026-05-20T09:47:29Z
Eugene Blikh added subtask tarantool-etcd-axs · 2026-05-20T09:47:29Z
Eugene Blikh linked tarantool-etcd-7ey (related) · 2026-05-20T09:47:29Z
Eugene Blikh added subtask tarantool-etcd-4yb · 2026-05-20T09:47:29Z
Eugene Blikh added subtask tarantool-etcd-ovo · 2026-05-20T09:47:29Z
Eugene Blikh added subtask tarantool-etcd-zkg · 2026-05-20T09:47:29Z
Eugene Blikh added subtask tarantool-etcd-6zn · 2026-05-20T09:47:29Z
Eugene Blikh added subtask tarantool-etcd-6js · 2026-05-20T09:47:29Z
Eugene Blikh added label milestone:m5-clustering · 2026-05-21T14:37:41Z

Stored rows — what this pane was built from, as read

issues 1 row

id	tarantool-etcd-cnj
content_hash	ed489cb32430e8d6cd2ea3b91464291e41a9c603992a7c8baf680dd56c313931
title	[epic] Raft-learner replica — join an etcd cluster as a non-voting member
description	Attach this server to a real etcd cluster as a Raft learner (non-voting member), receiving the replicated log over etcd's internal peer protocol (rafthttp) and applying it into our etcd_* spaces. Unlike the Watch-mirror route (tarantool-etcd-7ey), a learner is in the cluster's membership view (MemberList shows it), sees auth/lease/cluster/alarm state in the log automatically, and avoids the Watch compaction-resync gap. Cost is ~60-70% of full-member work, NOT 20%: the election state machine, proposal path, leader responsibilities, and snapshot send drop off, but the rafthttp transport, log applier, and snapshot receive (bbolt parse) all remain. We track etcd-INTERNAL protos (raftpb.Message, etcdserverpb.InternalRaftRequest) — stable-ish but not a public contract; release cadence binds to upstream etcd versions. Decision context (chat 2026-05-20): user explicitly chose the learner path over the Watch mirror. This epic is the alternative to 7ey, not a child of it. Phasing is learner-first and bottom-up: config -> transport -> proto decode -> state machine -> apply loop -> snapshot receive -> durability -> membership handshake -> conformance. Split into child issues below.
design
acceptance_criteria
notes	Surface map: peer endpoints /raft, /raft/stream/message/<id>, /raft/stream/msgapp/<id>, /raft/snapshot on port 2380. Reuses our nghttp2 HTTP/2 layer for the transport and our pure-Lua pb engine for the gogoproto schemas. Easiest correctness oracle: tests/conformance harness can SpawnEtcd then MemberAddAsLearner us.
status	open
priority	2
issue_type	feature
assignee	NULL
estimated_minutes	NULL
created_at	2026-05-20T06:39:59Z
created_by	Eugene Blikh
owner	bigbes@gmail.com
updated_at	2026-07-17T14:27:36Z
closed_at	NULL
closed_by_session
external_ref	NULL
spec_id
compaction_level	0
compacted_at	NULL
compacted_at_commit	NULL
original_size	NULL
sender
ephemeral	0
wisp_type
pinned	0
is_template	0
mol_type
work_type
source_system
metadata	<binary>
source_repo
close_reason
event_kind
actor
target
payload
await_type
await_id
timeout_ns	0
waiters
hook_bead
role_bead
agent_state
last_activity	NULL
role_type
rig
due_at	NULL
defer_until	NULL
no_history	0
started_at	NULL
is_blocked	0

labels 1 row

issue_id	tarantool-etcd-cnj
label	milestone:m5-clustering

dependencies 10 rows

id	06ac10bf-a45b-50c0-81c1-20ef00d34aa3
issue_id	tarantool-etcd-umc
type	parent-child
created_at	2026-05-20T09:47:12Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

id	0fbc3e46-3cce-57c1-bcb5-87c678116d33
issue_id	tarantool-etcd-q5f
type	parent-child
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

id	30803e2f-2f79-5e43-9ef4-6930bdbab635
issue_id	tarantool-etcd-boq
type	parent-child
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

id	8227e5b7-db14-5bc9-aeb8-26f9fde95dd5
issue_id	tarantool-etcd-axs
type	parent-child
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

id	9192386c-c056-5244-abe5-cbbb0bede32b
issue_id	tarantool-etcd-cnj
type	related
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-7ey
depends_on_wisp_id	NULL
depends_on_external	NULL

id	a2835c4a-d52d-5b54-af2d-6ccbdc823ff9
issue_id	tarantool-etcd-4yb
type	parent-child
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

id	a9139d4e-ecc4-5a12-94a5-7105b5a44b26
issue_id	tarantool-etcd-ovo
type	parent-child
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

id	c4afa884-aaa0-58c5-9029-b9e4720d2f46
issue_id	tarantool-etcd-zkg
type	parent-child
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

id	d080181a-8870-5118-b344-b342171ce2f8
issue_id	tarantool-etcd-6zn
type	parent-child
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

id	d6b0eb2c-9583-5791-9b8e-1de69b956d02
issue_id	tarantool-etcd-6js
type	parent-child
created_at	2026-05-20T09:47:29Z
created_by	Eugene Blikh
metadata	<binary>
thread_id
depends_on_issue_id	tarantool-etcd-cnj
depends_on_wisp_id	NULL
depends_on_external	NULL

events 2 rows

id	7e827059-0685-57e9-a0e4-f187205646f7
issue_id	tarantool-etcd-cnj
event_type	created
actor	Eugene Blikh
old_value
new_value
comment	NULL
created_at	2026-05-20T09:39:58Z

id	da619bb2-c632-55c3-b7e7-25e992a63cd1
issue_id	tarantool-etcd-cnj
event_type	label_added
actor	Eugene Blikh
old_value	NULL
new_value	NULL
comment	Added label: milestone:m5-clustering
created_at	2026-05-21T14:37:41Z