main · last commit
1 month ago ·
2npkec0r
tarantool-etcd-fv9 gRPC connections closed after 30s idle; no server keepalive PINGs
Lined Up
bd update tarantool-etcd-fv9 --claim
bd close tarantool-etcd-fv9
| Created by | Eugene Blikh |
| Owner | bigbes@gmail.com |
| Created | 2026-07-17T13:19:34Z |
| Updated | 2026-07-17T13:19:34Z |
lib/nghttp2/grpc.lua:394-398 hardcodes 'local readable = self.sock:readable(30)' and breaks the read loop when it returns false — the comment reads 'Idle timeout — close the connection.' So any gRPC connection with no INBOUND bytes for 30s is dropped. etcd does the opposite: --grpc-keepalive-interval (default 2h) PINGs idle connections to keep them alive. Supporting evidence: lib/nghttp2/session.lua:611-620 defines submit_ping with zero callers; ENHANCE_YOUR_CALM (lib/nghttp2/ffi.lua:892) is declared and never used; no keepalive knob exists in app/roles/etcd.lua. IMPACT NOTE (deliberately downgraded from the sweep's 'client-breaking'): clientv3 auto-reconnects and resumes watches from the last revision, so the real cost is reconnect churn, dropped idle streams, and latency spikes — not breakage. An idle Tarantool EE config.etcd watch would bounce every 30s. Worth fixing; not an outage. The conformance suite cannot see this: no test idles for 30s.
Cheapest correct fix is to stop treating 'no inbound bytes' as death — loop on readable() rather than breaking, and only close on EOF/error. Optionally send a PING on idle and expose grpc.keepalive.{interval,timeout} in roles_cfg. Consider whether any deployment relies on the 30s reap to shed dead sockets before removing it outright.
A connection idle >30s stays open and serves the next request; an idle Watch keeps delivering after 60s+; keepalive interval/timeout configurable; a test covers the idle case.
No outgoing dependencies.
Nothing depends on this issue.
| id | tarantool-etcd-fv9 |
| content_hash | 831466004ea8e32443ae2880646ec03b2fe3e529023e793ebbb93b3622b84f99 |
| title | gRPC connections closed after 30s idle; no server keepalive PINGs |
| description | lib/nghttp2/grpc.lua:394-398 hardcodes 'local readable = self.sock:readable(30)' and breaks the read loop when it returns false — the comment reads 'Idle timeout — close the connection.' So any gRPC connection with no INBOUND bytes for 30s is dropped. etcd does the opposite: --grpc-keepalive-interval (default 2h) PINGs idle connections to keep them alive. Supporting evidence: lib/nghttp2/session.lua:611-620 defines submit_ping with zero callers; ENHANCE_YOUR_CALM (lib/nghttp2/ffi.lua:892) is declared and never used; no keepalive knob exists in app/roles/etcd.lua. IMPACT NOTE (deliberately downgraded from the sweep's 'client-breaking'): clientv3 auto-reconnects and resumes watches from the last revision, so the real cost is reconnect churn, dropped idle streams, and latency spikes — not breakage. An idle Tarantool EE config.etcd watch would bounce every 30s. Worth fixing; not an outage. The conformance suite cannot see this: no test idles for 30s. |
| design | Cheapest correct fix is to stop treating 'no inbound bytes' as death — loop on readable() rather than breaking, and only close on EOF/error. Optionally send a PING on idle and expose grpc.keepalive.{interval,timeout} in roles_cfg. Consider whether any deployment relies on the 30s reap to shed dead sockets before removing it outright. |
| acceptance_criteria | A connection idle >30s stays open and serves the next request; an idle Watch keeps delivering after 60s+; keepalive interval/timeout configurable; a test covers the idle case. |
| notes | |
| status | open |
| priority | 2 |
| issue_type | bug |
| assignee | NULL |
| estimated_minutes | NULL |
| created_at | 2026-07-17T13:19:34Z |
| created_by | Eugene Blikh |
| owner | bigbes@gmail.com |
| updated_at | 2026-07-17T13:19:34Z |
| closed_at | NULL |
| closed_by_session | |
| external_ref | NULL |
| spec_id | |
| compaction_level | 0 |
| compacted_at | NULL |
| compacted_at_commit | NULL |
| original_size | NULL |
| sender | |
| ephemeral | 0 |
| wisp_type | |
| pinned | 0 |
| is_template | 0 |
| mol_type | |
| work_type | |
| source_system | |
| metadata | <binary> |
| source_repo | |
| close_reason | |
| event_kind | |
| actor | |
| target | |
| payload | |
| await_type | |
| await_id | |
| timeout_ns | 0 |
| waiters | |
| hook_bead | |
| role_bead | |
| agent_state | |
| last_activity | NULL |
| role_type | |
| rig | |
| due_at | NULL |
| defer_until | NULL |
| no_history | 0 |
| started_at | NULL |
| is_blocked | 0 |
| issue_id | tarantool-etcd-fv9 |
| label | etcd-changelog-sweep |
| id | 019f703b-7afc-76e9-bab8-377ef485e936 |
| issue_id | tarantool-etcd-fv9 |
| event_type | created |
| actor | Eugene Blikh |
| old_value | |
| new_value | |
| comment | NULL |
| created_at | 2026-07-17T16:19:34Z |
| id | 019f703b-7afc-7e15-93aa-8802114e62f9 |
| issue_id | tarantool-etcd-fv9 |
| event_type | label_added |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Added label: etcd-changelog-sweep |
| created_at | 2026-07-17T16:19:34Z |
No comments.