~bigbes/tarantool-etcd · parade

main · last commit 1 month ago · 2npkec0r

← Back to the parade

tarantool-etcd-q5f Learner: snapshot receive (bbolt parse + bucket translation) Lined Up

status: open P2 task milestone:m5-clustering
bd update tarantool-etcd-q5f --claim bd close tarantool-etcd-q5f
Created byEugene Blikh
Ownerbigbes@gmail.com
Created2026-05-20T06:45:16Z
Updated2026-07-17T14:27:37Z
Description
Handle /raft/snapshot. When we fall too far behind (or on first join), the leader ships a bbolt DB file dump, NOT raft entries. Two options: (a) parse the bbolt file and translate etcd's buckets — key (the mvcc keyspace), meta (consistent_index, scheduledCompactRev, finishedCompactRev), lease, auth/authUsers/authRoles, members/members_removed, cluster, alarm — into our etcd_* spaces; or (b) refuse snapshots and bootstrap via clientv3 Range/Watch instead, accepting that the cluster MAY force a snapshot mid-stream if we lag past its compaction window (it eventually will). Recommend (a) for a true learner. Needs a minimal bbolt page/leaf reader in Lua or via FFI. This is the phase most likely to surprise — it bypasses the raft proto entirely.
Notes
bbolt format: page-based B+tree, freelist, meta pages with txid. Bucket values for mvcc keys are mvccpb.KeyValue protobufs keyed by revision{main,sub}. This is the single biggest sub-effort in the epic.

Depends on

  • tarantool-etcd-cnj — [epic] Raft-learner replica — join an etcd cluster as a non-voting member parent-child open
  • tarantool-etcd-6js — Learner: apply loop — InternalRaftRequest -> existing handlers blocks open

Depended on by

  • tarantool-etcd-zkg — Learner: conformance — join real etcd cluster, assert log replication blocks

Prerequisite chain — everything this waits on, transitively

  • tarantool-etcd-cnj — [epic] Raft-learner replica — join an etcd cluster as a non-voting member parent-child open
  • tarantool-etcd-7ey — Replica route: connect as read-only replica to an existing etcd cluster related open
  • tarantool-etcd-16d — Implement Maintenance.Snapshot RPC (streaming DB snapshot) related open
  • tarantool-etcd-6js — Learner: apply loop — InternalRaftRequest -> existing handlers blocks open
  • tarantool-etcd-boq — Learner: raft state machine (follower/learner subset) blocks open
  • tarantool-etcd-axs — Learner: rafthttp peer transport over nghttp2 blocks open
  • tarantool-etcd-umc — Learner: config block + role plumbing blocks open
  • tarantool-etcd-4yb — Learner: raftpb + InternalRaftRequest proto schemas blocks open

No comments.

  • Eugene Blikh created the issue · 2026-05-20T09:45:16Z
  • Eugene Blikh added under epic tarantool-etcd-cnj · 2026-05-20T09:47:29Z
  • Eugene Blikh added dependency on tarantool-etcd-6js · 2026-05-20T09:47:29Z
  • Eugene Blikh tarantool-etcd-zkg now depends on this · 2026-05-20T09:47:29Z
  • Eugene Blikh added label milestone:m5-clustering · 2026-05-21T14:37:38Z
Stored rows — what this pane was built from, as read
issues 1 row
id tarantool-etcd-q5f
content_hash 722c317df5aebe05f32f99bf226d4b916604f1518089bea768b2ed206866ecd3
title Learner: snapshot receive (bbolt parse + bucket translation)
description Handle /raft/snapshot. When we fall too far behind (or on first join), the leader ships a bbolt DB file dump, NOT raft entries. Two options: (a) parse the bbolt file and translate etcd's buckets — key (the mvcc keyspace), meta (consistent_index, scheduledCompactRev, finishedCompactRev), lease, auth/authUsers/authRoles, members/members_removed, cluster, alarm — into our etcd_* spaces; or (b) refuse snapshots and bootstrap via clientv3 Range/Watch instead, accepting that the cluster MAY force a snapshot mid-stream if we lag past its compaction window (it eventually will). Recommend (a) for a true learner. Needs a minimal bbolt page/leaf reader in Lua or via FFI. This is the phase most likely to surprise — it bypasses the raft proto entirely.
design
acceptance_criteria
notes bbolt format: page-based B+tree, freelist, meta pages with txid. Bucket values for mvcc keys are mvccpb.KeyValue protobufs keyed by revision{main,sub}. This is the single biggest sub-effort in the epic.
status open
priority 2
issue_type task
assignee NULL
estimated_minutes NULL
created_at 2026-05-20T06:45:16Z
created_by Eugene Blikh
owner bigbes@gmail.com
updated_at 2026-07-17T14:27:37Z
closed_at NULL
closed_by_session
external_ref NULL
spec_id
compaction_level 0
compacted_at NULL
compacted_at_commit NULL
original_size NULL
sender
ephemeral 0
wisp_type
pinned 0
is_template 0
mol_type
work_type
source_system
metadata <binary>
source_repo
close_reason
event_kind
actor
target
payload
await_type
await_id
timeout_ns 0
waiters
hook_bead
role_bead
agent_state
last_activity NULL
role_type
rig
due_at NULL
defer_until NULL
no_history 0
started_at NULL
is_blocked 1
labels 1 row
issue_id tarantool-etcd-q5f
label milestone:m5-clustering
dependencies 3 rows
id 0fbc3e46-3cce-57c1-bcb5-87c678116d33
issue_id tarantool-etcd-q5f
type parent-child
created_at 2026-05-20T09:47:29Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-cnj
depends_on_wisp_id NULL
depends_on_external NULL
id 180d56bc-568d-5b7c-910c-cd894fca5bdd
issue_id tarantool-etcd-q5f
type blocks
created_at 2026-05-20T09:47:29Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-6js
depends_on_wisp_id NULL
depends_on_external NULL
id b3faee82-6b59-5203-872f-d3fbcde31538
issue_id tarantool-etcd-zkg
type blocks
created_at 2026-05-20T09:47:29Z
created_by Eugene Blikh
metadata <binary>
thread_id
depends_on_issue_id tarantool-etcd-q5f
depends_on_wisp_id NULL
depends_on_external NULL
events 2 rows
id e315dfd4-c556-5638-b364-ccc96f555f16
issue_id tarantool-etcd-q5f
event_type created
actor Eugene Blikh
old_value
new_value
comment NULL
created_at 2026-05-20T09:45:16Z
id e8e98e27-de06-534a-9e96-9a4910f0be8a
issue_id tarantool-etcd-q5f
event_type label_added
actor Eugene Blikh
old_value NULL
new_value NULL
comment Added label: milestone:m5-clustering
created_at 2026-05-21T14:37:38Z