A flight recorder for Postgres

Here is how most PostgreSQL incidents end. The dashboard spike flattens, the pager quiets down, and someone writes in the postmortem: “the database was slow between 14:10 and 14:40; cause not fully determined; we will monitor.” Everyone nods. Nothing was learned. Three weeks later it happens again, and the investigation starts from zero, again, because the evidence evaporated the moment the incident stopped happening.

I’ve been building Tracegres to attack exactly this, and the design premise fits in one sentence: database incidents should be reconstructable, not anecdotal.

What aviation figured out

Aviation solved this problem decades ago, and the solution wasn’t better dashboards in the cockpit. It was the flight recorder: an instrument that is always recording, cheap to run, ignored 99.9% of the time, and utterly decisive when something goes wrong. Nobody watches a black box in real time. Its entire value is that when investigators arrive after the fact, the past is still there.

Database observability mostly took the opposite path: walls of live charts, optimised for watching now. Charts tell you that latency rose. They are strikingly bad at telling you why, after the fact, at the level of “which query’s plan changed and what did it look like before.” The signal you need for reconstruction (plans, fingerprints, what deployed when) is exactly the signal most stacks don’t keep.

So Tracegres is built as the black box, not the cockpit: a lightweight Rust collector that continuously records what future-you will need for the investigation:

• operational metrics, the baseline vital signs;
• query fingerprints, so “this query” means something precise across time;
• safe EXPLAIN plans, captured without query literals, because plans are the why behind most database mysteries and nobody keeps them;
• deployment markers, because “what changed” is the first question of every incident;
• trace context, so a database anomaly links to the requests that experienced it.

“Privacy-safe” is a design constraint, not a feature checkbox: fingerprints and plan structure, not your customers’ data. The collector should be boring to approve.

The querying-the-past test

The bar I hold Tracegres to: five minutes to a defensible explanation, even after the incident has stopped happening. Not a wall of charts that correlates with the outage, but a narrative with evidence: at 14:07 this deployment landed; at 14:10 this query’s plan flipped from an index scan to a sequential scan; here is the before plan, here is the after plan, structurally diffed. That’s the difference between “the database was slow” and a sentence you can defend in a postmortem.

The deeper opinion here: most teams don’t need more observability, they need answerable questions. A tool’s output should be an explanation or a decision, not another surface to stare at.

The part I’m most excited about: plans as release gates

Reconstruction is the defensive half. The offensive half: if plan changes cause incidents, catch the plan change before the release ships.

Tracegres keeps plan history per query fingerprint and does structural comparison between a baseline and a candidate: not “is it slower on my laptop,” which is noise, but “did the shape of the plan regress”: index scan became sequential scan, a join strategy flipped, an estimate went haywire. That comparison becomes a CI gate with evidence attached. The repo’s proof of it is my favourite part: a deterministic regression lab that spins up a disposable Postgres 16 with representative data, captures a baseline plan through the real collector, drops an index, captures the candidate, and asserts that the real CLI gate blocks the release, showing the structural diff as its reasoning.

Determinism is the point. Plan regressions in the wild feel spooky and irreproducible; the lab makes one happen on demand, which is what lets you trust the gate. Schema change reviews today are mostly vibes (“this migration looks fine”). Plans-as-CI-artifacts replaces vibes with evidence, the same movement types brought to refactoring.

Dogfood first, product later

Tracegres is an experimental prototype, and I use it in my own development workflow, which I’ve come to believe is the only honest way to build developer tools: if the flight recorder can’t explain my incidents, it has no business near yours. The collector, CLI, plan-comparison engine, and regression lab are headed for open source; the longer-horizon ideas (retention, anomaly detection, incident intelligence) can stay ideas until the core has earned trust.

It’s aimed at the teams I keep meeting: no database specialist on call, no appetite for operating a full observability stack, but real production Postgres and real unexplained Tuesdays. The pitch to them isn’t “more monitoring.” It’s: when your database has a bad afternoon, the evidence should still exist in the evening.

Airplanes don’t fly with investigators on board. They fly with a recorder, and that’s why crashes become knowledge instead of mysteries. Databases deserve the same courtesy.