~bigbes/tarantool-etcd · parade

main · last commit 1 month ago · 2npkec0r

← Back to the parade

tarantool-etcd-fv9 gRPC connections closed after 30s idle; no server keepalive PINGs Lined Up

status: open P2 bug etcd-changelog-sweep
bd update tarantool-etcd-fv9 --claim bd close tarantool-etcd-fv9
Created byEugene Blikh
Ownerbigbes@gmail.com
Created2026-07-17T13:19:34Z
Updated2026-07-17T13:19:34Z
Description
lib/nghttp2/grpc.lua:394-398 hardcodes 'local readable = self.sock:readable(30)' and breaks the read loop when it returns false — the comment reads 'Idle timeout — close the connection.' So any gRPC connection with no INBOUND bytes for 30s is dropped. etcd does the opposite: --grpc-keepalive-interval (default 2h) PINGs idle connections to keep them alive.

Supporting evidence: lib/nghttp2/session.lua:611-620 defines submit_ping with zero callers; ENHANCE_YOUR_CALM (lib/nghttp2/ffi.lua:892) is declared and never used; no keepalive knob exists in app/roles/etcd.lua.

IMPACT NOTE (deliberately downgraded from the sweep's 'client-breaking'): clientv3 auto-reconnects and resumes watches from the last revision, so the real cost is reconnect churn, dropped idle streams, and latency spikes — not breakage. An idle Tarantool EE config.etcd watch would bounce every 30s. Worth fixing; not an outage.

The conformance suite cannot see this: no test idles for 30s.
Design
Cheapest correct fix is to stop treating 'no inbound bytes' as death — loop on readable() rather than breaking, and only close on EOF/error. Optionally send a PING on idle and expose grpc.keepalive.{interval,timeout} in roles_cfg. Consider whether any deployment relies on the 30s reap to shed dead sockets before removing it outright.
Acceptance criteria
A connection idle >30s stays open and serves the next request; an idle Watch keeps delivering after 60s+; keepalive interval/timeout configurable; a test covers the idle case.

Depends on

No outgoing dependencies.

Depended on by

Nothing depends on this issue.

No comments.

  • Eugene Blikh created the issue · 2026-07-17T16:19:34Z
  • Eugene Blikh added label etcd-changelog-sweep · 2026-07-17T16:19:34Z
Stored rows — what this pane was built from, as read
issues 1 row
id tarantool-etcd-fv9
content_hash 831466004ea8e32443ae2880646ec03b2fe3e529023e793ebbb93b3622b84f99
title gRPC connections closed after 30s idle; no server keepalive PINGs
description lib/nghttp2/grpc.lua:394-398 hardcodes 'local readable = self.sock:readable(30)' and breaks the read loop when it returns false — the comment reads 'Idle timeout — close the connection.' So any gRPC connection with no INBOUND bytes for 30s is dropped. etcd does the opposite: --grpc-keepalive-interval (default 2h) PINGs idle connections to keep them alive. Supporting evidence: lib/nghttp2/session.lua:611-620 defines submit_ping with zero callers; ENHANCE_YOUR_CALM (lib/nghttp2/ffi.lua:892) is declared and never used; no keepalive knob exists in app/roles/etcd.lua. IMPACT NOTE (deliberately downgraded from the sweep's 'client-breaking'): clientv3 auto-reconnects and resumes watches from the last revision, so the real cost is reconnect churn, dropped idle streams, and latency spikes — not breakage. An idle Tarantool EE config.etcd watch would bounce every 30s. Worth fixing; not an outage. The conformance suite cannot see this: no test idles for 30s.
design Cheapest correct fix is to stop treating 'no inbound bytes' as death — loop on readable() rather than breaking, and only close on EOF/error. Optionally send a PING on idle and expose grpc.keepalive.{interval,timeout} in roles_cfg. Consider whether any deployment relies on the 30s reap to shed dead sockets before removing it outright.
acceptance_criteria A connection idle >30s stays open and serves the next request; an idle Watch keeps delivering after 60s+; keepalive interval/timeout configurable; a test covers the idle case.
notes
status open
priority 2
issue_type bug
assignee NULL
estimated_minutes NULL
created_at 2026-07-17T13:19:34Z
created_by Eugene Blikh
owner bigbes@gmail.com
updated_at 2026-07-17T13:19:34Z
closed_at NULL
closed_by_session
external_ref NULL
spec_id
compaction_level 0
compacted_at NULL
compacted_at_commit NULL
original_size NULL
sender
ephemeral 0
wisp_type
pinned 0
is_template 0
mol_type
work_type
source_system
metadata <binary>
source_repo
close_reason
event_kind
actor
target
payload
await_type
await_id
timeout_ns 0
waiters
hook_bead
role_bead
agent_state
last_activity NULL
role_type
rig
due_at NULL
defer_until NULL
no_history 0
started_at NULL
is_blocked 0
labels 1 row
issue_id tarantool-etcd-fv9
label etcd-changelog-sweep
events 2 rows
id 019f703b-7afc-76e9-bab8-377ef485e936
issue_id tarantool-etcd-fv9
event_type created
actor Eugene Blikh
old_value
new_value
comment NULL
created_at 2026-07-17T16:19:34Z
id 019f703b-7afc-7e15-93aa-8802114e62f9
issue_id tarantool-etcd-fv9
event_type label_added
actor Eugene Blikh
old_value NULL
new_value NULL
comment Added label: etcd-changelog-sweep
created_at 2026-07-17T16:19:34Z