main · last commit
1 month ago ·
2npkec0r
tarantool-etcd-j1c /debug/pprof/heap — pprof-shape converter on top of /debug/memprof
Lined Up
bd update tarantool-etcd-j1c --claim
bd close tarantool-etcd-j1c
| Created by | Eugene Blikh |
| Owner | bigbes@gmail.com |
| Created | 2026-05-19T13:34:30Z |
| Updated | 2026-05-19T13:34:30Z |
/debug/memprof ships the raw Tarantool memprof binary trace. Open work: a pprof-shape converter — pure-Lua parser for the memprof binary, aggregate events by allocation site, emit a four-dimension perftools.profiles.Profile (alloc_objects/count, alloc_space/bytes, inuse_objects/count, inuse_space/bytes — Go heap profile shape).
Implementation sketch:
1. Pure-Lua parser for the memprof binary format (header magic + tagged event stream: alloc / realloc / free / symtab). Reference is tools/memprof.lua + tools/utils/binreader.lua in upstream tarantool/tarantool (not bundled with the homebrew binary).
2. Aggregate into (stack_key → {alloc_objs, alloc_bytes, inuse_objs, inuse_bytes}) with inuse_* from correlating alloc events against later free events inside the same window. Stack keys are the (chunkname, line) chain from symtab events.
3. Encode via lib/protobuf/encoder.lua + lib/pprof/defs.lua — schema already supports multi-sample-type profiles, declare four sample_type entries.
4. New endpoint /debug/pprof/heap?seconds=N on the JSON gateway. Same shape as /debug/pprof/profile (600 s cap, 503 on memprof-unavailable). Existing /debug/memprof stays for raw-trace use cases.
Known frictions: - misc.memprof.available == false on Apple Silicon (same arm64 constraint as the JIT — see reference_apple_silicon_no_jit). start() returns true but stop() says "profiler is disabled" and writes nothing. /debug/memprof already detects this and 503s; pprof-heap should do the same. Linux/x86_64 unaffected. - inuse_* is "alive at end-of-window", not "alive right now" — long-lived objects allocated before the window started never appear. Document this; Go heap profile has the same caveat. - Writing the binary parser without ground-truth output is the main risk — develop on a Linux box where memprof actually fires, validate against tools/memprof.lua textual report.
1. /debug/pprof/heap?seconds=N returns valid pprof-shaped protobuf, parseable by `go tool pprof`. 2. On Apple Silicon, returns 503 with clear "memprof unavailable on arm64" message. 3. Round-trip tests in test/pprof_test.lua (same shape as the cpu profile tests).
Source: docs/TODO.md "Open / Pprof-shape /debug/pprof/heap". Referenced from app/etcd/debug.lua:16, lib/pprof/memprof.lua:12.
No outgoing dependencies.
Nothing depends on this issue.
| id | tarantool-etcd-j1c |
| content_hash | d72af58dd67917cd0d6f45c09ebc8410746a5a5fa3ce0f2fc547a0845aa4d6da |
| title | /debug/pprof/heap — pprof-shape converter on top of /debug/memprof |
| description | /debug/memprof ships the raw Tarantool memprof binary trace. Open work: a pprof-shape converter — pure-Lua parser for the memprof binary, aggregate events by allocation site, emit a four-dimension perftools.profiles.Profile (alloc_objects/count, alloc_space/bytes, inuse_objects/count, inuse_space/bytes — Go heap profile shape). Implementation sketch: 1. Pure-Lua parser for the memprof binary format (header magic + tagged event stream: alloc / realloc / free / symtab). Reference is tools/memprof.lua + tools/utils/binreader.lua in upstream tarantool/tarantool (not bundled with the homebrew binary). 2. Aggregate into (stack_key → {alloc_objs, alloc_bytes, inuse_objs, inuse_bytes}) with inuse_* from correlating alloc events against later free events inside the same window. Stack keys are the (chunkname, line) chain from symtab events. 3. Encode via lib/protobuf/encoder.lua + lib/pprof/defs.lua — schema already supports multi-sample-type profiles, declare four sample_type entries. 4. New endpoint /debug/pprof/heap?seconds=N on the JSON gateway. Same shape as /debug/pprof/profile (600 s cap, 503 on memprof-unavailable). Existing /debug/memprof stays for raw-trace use cases. |
| design | Known frictions: - misc.memprof.available == false on Apple Silicon (same arm64 constraint as the JIT — see reference_apple_silicon_no_jit). start() returns true but stop() says "profiler is disabled" and writes nothing. /debug/memprof already detects this and 503s; pprof-heap should do the same. Linux/x86_64 unaffected. - inuse_* is "alive at end-of-window", not "alive right now" — long-lived objects allocated before the window started never appear. Document this; Go heap profile has the same caveat. - Writing the binary parser without ground-truth output is the main risk — develop on a Linux box where memprof actually fires, validate against tools/memprof.lua textual report. |
| acceptance_criteria | 1. /debug/pprof/heap?seconds=N returns valid pprof-shaped protobuf, parseable by `go tool pprof`. 2. On Apple Silicon, returns 503 with clear "memprof unavailable on arm64" message. 3. Round-trip tests in test/pprof_test.lua (same shape as the cpu profile tests). |
| notes | Source: docs/TODO.md "Open / Pprof-shape /debug/pprof/heap". Referenced from app/etcd/debug.lua:16, lib/pprof/memprof.lua:12. |
| status | open |
| priority | 3 |
| issue_type | feature |
| assignee | NULL |
| estimated_minutes | NULL |
| created_at | 2026-05-19T13:34:30Z |
| created_by | Eugene Blikh |
| owner | bigbes@gmail.com |
| updated_at | 2026-05-19T13:34:30Z |
| closed_at | NULL |
| closed_by_session | |
| external_ref | NULL |
| spec_id | |
| compaction_level | 0 |
| compacted_at | NULL |
| compacted_at_commit | NULL |
| original_size | NULL |
| sender | |
| ephemeral | 0 |
| wisp_type | |
| pinned | 0 |
| is_template | 0 |
| mol_type | |
| work_type | |
| source_system | |
| metadata | <binary> |
| source_repo | |
| close_reason | |
| event_kind | |
| actor | |
| target | |
| payload | |
| await_type | |
| await_id | |
| timeout_ns | 0 |
| waiters | |
| hook_bead | |
| role_bead | |
| agent_state | |
| last_activity | NULL |
| role_type | |
| rig | |
| due_at | NULL |
| defer_until | NULL |
| no_history | 0 |
| started_at | NULL |
| is_blocked | 0 |
| issue_id | tarantool-etcd-j1c |
| label | milestone:m1-conformance |
| id | cf5dbef3-f48b-5f9b-b3da-8d431e68bb67 |
| issue_id | tarantool-etcd-j1c |
| event_type | label_added |
| actor | Eugene Blikh |
| old_value | NULL |
| new_value | NULL |
| comment | Added label: milestone:m1-conformance |
| created_at | 2026-05-21T14:37:01Z |
| id | ead456d0-04f2-59a4-b2e7-b47f8b774fba |
| issue_id | tarantool-etcd-j1c |
| event_type | created |
| actor | Eugene Blikh |
| old_value | |
| new_value | |
| comment | NULL |
| created_at | 2026-05-19T16:34:30Z |
No comments.