石頭的coding之路 — PostgreSQL

pg_dbmigrator: Near-Zero-Downtime PostgreSQL Migration with Online Mode

2026-05-03T16:00:00.000Z

pg_dbmigrator: Near-Zero-Downtime PostgreSQL Migration with Online Mode

pg_dbmigrator is a Rust-based CLI and library for moving PostgreSQL databases between two endpoints. It supports two strategies under one tool:

Offline mode — a one-shot pg_dump → pg_restore, suitable for maintenance windows.
Online mode — a streaming pipeline built on PostgreSQL’s logical replication that copies the initial snapshot, then continuously applies WAL changes until you trigger cutover.

If you’ve ever had to migrate a busy PostgreSQL database between hosts, regions, or cloud providers (on-prem → RDS, RDS → Aurora, Azure → AWS, major-version upgrades, etc.), the online mode is what makes pg_dbmigrator interesting: the source database stays writable during the entire copy, and downtime is reduced to the few seconds it takes to flip the application’s connection string.

This post walks through how the tool works, how to operate it end-to-end, and what to watch for in production.

Why Another Migration Tool?

PostgreSQL ships with the building blocks — pg_dump, pg_restore, CREATE PUBLICATION, CREATE SUBSCRIPTION, replication slots — but stitching them together correctly is surprisingly easy to get wrong. Common mistakes:

Dumping before creating the slot, so you have no consistent starting LSN.
Creating the slot after the dump, missing changes that occurred during the dump.
Forgetting to use EXPORT_SNAPSHOT so the dump and the replication start point don’t align.
Cutting over while the subscriber is still lagging behind, losing writes.

pg_dbmigrator encodes the correct order as a state machine, surfaces lag metrics, and lets the operator drive the cutover when ready.

Migration Modes at a Glance

Offline mode — one shot, with downtime

The source must be quiesced (writes stopped) before the dump starts. Anything written after the dump begins will be lost. The whole migration is a single linear pipe:

┌────────┐   pg_dump    ┌──────┐   pg_restore   ┌────────┐
│ Source │ ───────────▶ │ Dump │ ─────────────▶ │ Target │
└────────┘              └──────┘                └────────┘
 (writes stopped for the entire duration)

Downtime ≈ time to dump + time to restore. Simple, but every minute is user-visible.

Online mode — copy + replay, cutover on demand

Two things run back-to-back on the same timeline. First, an initial snapshot copy (just like offline). Then, while the target catches up, a logical-replication stream keeps replaying every change that happened on the source after the snapshot — including writes that arrived during the dump. The source stays writable the whole time.

          Phase 1: initial copy
┌────────┐  snapshot dump   ┌────────┐
│ Source │ ───────────────▶ │ Target │
└────────┘                  └────────┘
     │  (slot created BEFORE dump,
     │   so we know the exact LSN
     │   the dump corresponds to)
     │
     ▼  Phase 2: streaming catch-up (runs continuously)
┌────────┐  WAL changes via             ┌────────┐
│ Source │  logical replication ─────▶ │ Target │
└────────┘  (from the slot's LSN)       └────────┘
     │
     │  Source stays writable. Lag shrinks over time.
     │
     ▼  Phase 3: cutover (operator-driven)
 1. Stop app writes on the source
 2. Wait until lag = 0
 3. Press Ctrl+C  →  pg_dbmigrator stops cleanly
 4. Repoint the app at the target

The trick that makes this work: the replication slot is created before the dump and exports a snapshot. The dump uses that snapshot, and the stream starts at the slot’s LSN — so the stream picks up exactly where the dump left off. No gap, no duplicates.

Rule of thumb: pick offline when you can afford a maintenance window the size of (dump + restore). Pick online when you can’t.

Online Mode: The State Machine

Internally pg_dbmigrator runs the online migration as a sequence of phases:

1
2
3

Validate → SourceVacuum → PrepareSnapshot → Dump → Restore → Analyze
   → StreamApply → (Lag heartbeat …) → CaughtUp → Cutover
   → SourceCleanup → Complete

Each phase has a specific job:

Phase	What happens
Validate	Checks connection strings, `wal_level`, replication slot/sender capacity, target reachability.
SourceVacuum	Runs `VACUUM ANALYZE` on the source to reclaim dead tuples and refresh planner statistics before the dump. Skip with `--skip-source-vacuum`.
PrepareSnapshot	Creates the logical replication slot with `EXPORT_SNAPSHOT`. Streaming is deferred until after the dump completes; the exported snapshot ID is what makes the dump consistent with the slot’s starting LSN.
Dump	Runs `pg_dump --snapshot=<exported_id>` so the dump reflects the database state exactly at the slot’s starting LSN.
Restore	`pg_restore` into the target. Parallelized with `--jobs` and split-sections (indexes rebuilt in parallel) by default.
Analyze	Runs `ANALYZE` on the target so the first application queries have fresh statistics. Skip with `--skip-analyze`.
StreamApply	Starts `START_REPLICATION` from the slot’s LSN and applies decoded changes onto the target.
Lag heartbeat	Polls `pg_current_wal_flush_lsn()` on the source on a tunable interval, logs the delta against the receiver/applied LSN.
CaughtUp	Fires once lag falls below `--lag-threshold-bytes`, signalling the operator that cutover is safe. The threshold is advisory only — it does not auto-trigger cutover.
Cutover	Triggered by SIGINT (Ctrl+C). Flushes the final LSN feedback, syncs sequences on the target, then disables and drops the subscription.
SourceCleanup	After cutover, drops auto-created publications and the replication slot on the source. Failures are logged as warnings but don’t abort.

The key invariant: the slot is created before the dump, and the dump uses the slot’s exported snapshot. That way the WAL stream picks up exactly where the dump ended — no gap, no overlap.

Installation

pg_dbmigrator is published on crates.io:

1 2	cargo install pg_dbmigrator pg_dbmigrator --help

Or build from source:

1
2
3

git clone https://github.com/isdaniel/pg_dbmigrator
cd pg_dbmigrator
cargo install --path crates/pg_dbmigrator-cli

You also need pg_dump and pg_restore on the PATH — the tool shells out to them rather than reimplementing the dump format. Make sure their major version is >= the source database’s major version, otherwise you’ll hit compatibility errors during dump.

Source Prerequisites

Online mode requires a few server settings on the source:

-- postgresql.conf (requires restart)
ALTER SYSTEM SET wal_level = 'logical';
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;

That’s it for setup — pg_dbmigrator will auto-create the publication (FOR ALL TABLES) on first run if it doesn’t already exist, and clean it up after cutover. If you’d rather manage the publication yourself (e.g. to scope it to a subset of tables), pre-create it and pass --no-auto-create-publication:

1	CREATE PUBLICATION pg_dbmigrator_pub FOR TABLE schema.t1, schema.t2;

The connecting role needs REPLICATION and the ability to read every table in the publication. On managed services:

AWS RDS: set rds.logical_replication = 1 in the parameter group, then reboot.
Azure Database for PostgreSQL: enable the logical wal_level server parameter.
Google Cloud SQL: enable cloudsql.logical_decoding.

The target doesn’t need wal_level = logical unless you plan to chain replication out of it.

Resource Lifecycle

By default, pg_dbmigrator owns the resources it creates and cleans them up when the migration finishes. Override the cleanup if you need the resource to outlive the migration:

Resource	Created by	Default cleanup	Override
Publication	Auto-created if missing	Dropped if auto-created	`--no-auto-create-publication`
Replication slot	Always by migrator	Dropped after cutover	`--keep-slot`
Subscription	Always by migrator	Dropped after cutover	`--keep-subscription`

Offline Mode: the Simple Case

pg_dbmigrator \
  --mode offline \
  --source 'postgres://user:pw@src.example.com:5432/appdb' \
  --target 'postgres://user:pw@dst.example.com:5432/appdb' \
  --jobs 4 \
  --drop-target-first

What this does:

Drops the target database contents first (--drop-target-first).
Runs pg_dump against the source.
Pipes into pg_restore with 4 parallel workers.
Exits.

Use it for dev/staging clones, small databases, or when an outage window is acceptable.

Online Mode: the Full Workflow

A typical online migration command:

pg_dbmigrator \
  --mode online \
  --source 'postgres://user:pw@src.example.com:5432/appdb' \
  --target 'postgres://user:pw@dst.example.com:5432/appdb' \
  --slot-name pg_dbmigrator_slot \
  --publication pg_dbmigrator_pub \
  --subscription-name pg_dbmigrator_sub \
  --jobs 4 \
  --lag-threshold-bytes 8192 \
  --cutover-poll-secs 5

Walking through what each flag controls:

Flag	Purpose
`--mode online`	Selects the streaming pipeline.
`--source` / `--target`	Standard libpq connection strings.
`--slot-name`	Name of the logical replication slot created on the source. Pick something recognizable so you can clean it up if the migration is aborted.
`--publication`	Publication name. Defaults to `pg_dbmigrator_pub`; auto-created `FOR ALL TABLES` if missing.
`--subscription-name`	Name pg_dbmigrator gives the subscription it creates on the target.
`--jobs`	Parallelism for the initial `pg_restore`. Auto-detected from CPU count and clamped to `[1, 8]` if omitted.
`--lag-threshold-bytes`	When the gap between source flush LSN and applied LSN drops below this, the tool emits a “ready to cut over” signal. Advisory only — it does not auto-trigger cutover.
`--cutover-poll-secs`	How often the lag heartbeat polls `pg_current_wal_flush_lsn()`.

Useful additional flags:

--drop-target-first — recreate the target schema cleanly. Necessary on most fresh runs.
--no-auto-create-publication — skip auto-creation; use a publication you manage yourself (also skips its cleanup).
--keep-slot / --keep-subscription — leave the slot or subscription in place after cutover instead of dropping them. Handy for keeping a rollback window.
--no-sequence-sync — skip the automatic setval() pass at cutover. Use when the target role lacks the privileges; you’ll need to advance sequences manually.
--exclude-schema / --exclude-table — repeat to omit specific schemas or tables from the dump.
--skip-source-vacuum / --skip-analyze — disable the pre-dump VACUUM ANALYZE on the source or the post-restore ANALYZE on the target. Useful when the source is already well-vacuumed or when you want to control the maintenance window yourself.
--dump-compress — archive compression spec. Defaults to lz4:1; alternatives include zstd:3 or none.
--no-split-sections — disable parallel section restore. Split sections are 30–60% faster on index-heavy schemas, so only flip this off if the parallel restore is misbehaving.
--allow-restore-errors — treat pg_restore errors as warnings. Useful when the target is a managed service that reserves extension names (Azure-reserved extensions, missing pg_cron, etc.).

What You’ll See in the Terminal

Once StreamApply starts, the tool prints periodic heartbeats:

[INFO] StreamApply: replication lag 1,245,184 bytes
       (source LSN 0/1A2B3C4D, received LSN 0/1A1F0000, applied LSN 0/1A1E8000)
[INFO] StreamApply: replication lag 524,288 bytes
       (source LSN 0/1A2C1000, received LSN 0/1A2B9000, applied LSN 0/1A2B8000)
[INFO] StreamApply: replication lag 4,096 bytes
       (source LSN 0/1A2D5000, received LSN 0/1A2D4000, applied LSN 0/1A2D4000)
[INFO] CaughtUp: lag below 8,192 bytes — safe to cut over

The lag is the byte distance between the source’s current flush LSN and the target’s applied LSN. Watch it converge before cutting over.

Driving the Cutover

Cutover is operator-driven, not automatic. This is intentional — only you know when the application is in a state where it can be flipped. The flow:

Wait for CaughtUp to appear (or for the lag to be acceptable).
Stop application writes to the source (put the app in maintenance, revoke writes, or fail over the load balancer).
Wait one more heartbeat to confirm lag is 0 bytes.
Press Ctrl+C (single SIGINT). pg_dbmigrator:
- Stops the apply loop at the current LSN and flushes the final feedback message.
- Syncs sequences on the target via setval() (skip with --no-sequence-sync).
- Disables and drops the subscription on the target (unless --keep-subscription).
- Drops the replication slot and any auto-created publication on the source (unless --keep-slot or --no-auto-create-publication).
- Exits cleanly.
Repoint the application’s connection string to the target.

A second Ctrl+C is an escape hatch — if shutdown is stuck, hitting it again forces termination. Use only as a last resort, since the slot may be left behind on the source (drop manually with SELECT pg_drop_replication_slot('pg_dbmigrator_slot');).

A Realistic Online Migration Recipe

Putting it together, here’s the sequence I’d run for a real migration from on-prem to RDS:

# 1. On the source, one-time setup
psql "$SOURCE" <<'SQL'
ALTER SYSTEM SET wal_level = 'logical';
SQL
# Restart source PostgreSQL.
# (No need to pre-create the publication — pg_dbmigrator will auto-create
#  pg_dbmigrator_pub FOR ALL TABLES on first run, and clean it up at cutover.)

# 2. Kick off the migration in a tmux/screen session you can leave running.
pg_dbmigrator \
  --mode online \
  --source "$SOURCE" \
  --target "$TARGET" \
  --slot-name pg_dbmigrator_slot \
  --publication pg_dbmigrator_pub \
  --subscription-name pg_dbmigrator_sub \
  --jobs 8 \
  --lag-threshold-bytes 16384 \
  --cutover-poll-secs 5 \
  --drop-target-first

# 3. (In another terminal) sanity-check row counts as the dump completes,
#    and watch lag heartbeats.

# 4. When CaughtUp fires and you're ready:
#      a. Put app in maintenance mode (stop writes to source).
#      b. Wait until lag heartbeat shows 0 bytes.
#      c. Ctrl+C the pg_dbmigrator process — it will sync sequences, drop
#         the subscription, and clean up the slot/publication on the source.
#      d. Update DNS / connection string to point at TARGET.
#      e. Bring app out of maintenance.

# 5. Cleanup on the source — by default pg_dbmigrator drops the slot and
#    the auto-created publication itself. Only run these if you used
#    --keep-slot / --no-auto-create-publication, or aborted the run:
psql "$SOURCE" -c "SELECT pg_drop_replication_slot('pg_dbmigrator_slot');"
psql "$SOURCE" -c "DROP PUBLICATION pg_dbmigrator_pub;"

Total user-visible downtime is steps 4a–4e, which is typically tens of seconds.

Operational Gotchas

A few things that bite people the first time:

Replication slots hold WAL. If you start an online migration and abandon it, the slot will keep the source’s WAL pinned indefinitely, eventually filling the source’s disk. A clean Ctrl+C drops the slot for you, but if the process is killed or crashes, drop the slot manually (pg_drop_replication_slot).

Sequences are synced automatically at cutover. pg_dbmigrator runs setval() on every sequence in the included schemas as part of cutover, so you don’t need the old “advance every sequence by hand” ritual. Per-sequence failures (typically a missing privilege on the target role) are logged as warnings but won’t abort cutover. If your target role can’t setval, pass --no-sequence-sync and run the SQL yourself:

1 2	SELECT setval('public.users_id_seq', (SELECT max(id) FROM public.users));

DDL during migration is not replicated. If someone runs an ALTER TABLE on the source mid-stream, the target won’t see it and apply will eventually fail when a row references the new column. Freeze schema changes for the duration of the migration. If a DDL change really must happen, you’ll need to refresh the publication and restart the migration.

Restore errors on managed targets. Azure reserves extension names; pg_cron is unavailable on most managed services; some superuser-only objects can’t be restored by an unprivileged role. --allow-restore-errors lets the migration proceed, but inspect the warnings — anything important needs to be applied manually.

Custom column transforms aren’t supported. The streaming apply binds replicated values as text and casts them server-side. If you need to transform data during migration (e.g. re-encrypt a column), do it before or after, not inline.

Major-version differences. If source is PG14 and target is PG16, run pg_dump/pg_restore from the higher version’s binaries. pg_dbmigrator inherits whatever’s on the PATH.

When to Pick Offline vs. Online

Situation	Mode
Database is small (< few GB) and you have a maintenance window	Offline
Dev / staging clones	Offline
Production database, can’t take more than a few seconds of write downtime	Online
Cross-cloud or cross-region migrations	Online
Major-version upgrade on a live system	Online
Source can’t be set to `wal_level = logical`	Offline (no choice)

Summary

pg_dbmigrator wraps a careful, correct online migration recipe — slot-before-dump, snapshot-consistent copy, streaming apply, operator-driven cutover — into a single Rust CLI. For straightforward jobs the offline mode is enough; for production cutovers where downtime matters, the online mode reduces the outage to the moment you press Ctrl+C.

The most important habits when operating it:

Always confirm wal_level = logical on the source before starting.
Watch the lag heartbeats and only cut over from a CaughtUp state with 0 bytes lag and writes stopped — the threshold is advisory, not automatic.
Let the tool exit cleanly via Ctrl+C so it can drop the slot, the auto-created publication, and sync sequences for you. If the process dies hard, drop the slot manually so abandoned WAL doesn’t fill the source’s disk.
Freeze schema changes for the duration of the migration — DDL isn’t replicated.

References

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/pg-migrator-online-postgresql-migration/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！

pg_pidstat: Real-Time Per-Backend CPU, Memory, and I/O Monitoring for PostgreSQL

2026-04-27T02:00:00.000Z

pg_pidstat: Real-Time Per-Backend CPU, Memory, and I/O Monitoring for PostgreSQL

When a PostgreSQL server slows down, the first question is usually: which backend is consuming resources? pg_stat_activity tells you what each backend is doing, but not how much CPU, memory, or disk I/O it uses. You end up jumping between top, pidstat, and pg_stat_activity, manually correlating PIDs.

pg_pidstat solves this by embedding OS-level process metrics directly into PostgreSQL. A single SQL view gives you per-backend CPU percentage, memory consumption, I/O throughput, and context switch rates — all joined with the connection context you already know from pg_stat_activity.

The Problem: Blind Spots in PostgreSQL Monitoring

PostgreSQL’s built-in statistics views (pg_stat_activity, pg_stat_user_tables, pg_stat_io) provide query-level and table-level insights, but they lack process-level OS metrics. When you need to answer questions like:

Which backend is burning 100% CPU?
Is that long-running query memory-hungry or I/O-bound?
Are context switches causing latency spikes?

You typically resort to external tools — pidstat, htop, or custom /proc scrapers — and then manually join their output with pg_stat_activity by PID. This workflow is tedious, error-prone, and impossible to automate in pure SQL.

pg_pidstat eliminates that gap entirely.

Key Features

Per-backend CPU usage as a percentage of total system capacity
Memory tracking — both percentage and absolute MB (resident set size)
I/O throughput — read/write bytes per second and IOPS
I/O wait detection — flags backends blocked on disk
Context switch rates — voluntary and non-voluntary, per second
Full pg_stat_activity integration — all standard columns (pid, datname, usename, state, query, etc.) included
Background worker sampling — metrics updated every 1 second automatically
Lock-free reads — minimal contention with brief exclusive writes only during sampling

Architecture

pg_pidstat is built with pgrx (Rust) and consists of four modules:

┌─────────────────────────────────────────────────────┐
│                    SQL Interface                     │
│          pg_pidstat view + helper functions           │
│                    (activity.rs)                      │
└──────────────────────┬──────────────────────────────┘
                       │  per-second deltas
┌──────────────────────▼──────────────────────────────┐
│               Background Worker                      │
│        Samples all backends every 1 second           │
│                   (bgworker.rs)                      │
└──────────────────────┬──────────────────────────────┘
                       │  raw counters
┌──────────────────────▼──────────────────────────────┐
│           Double-Buffered Shared Memory              │
│        Lock-free reads, up to 1024 backends          │
│                    (shmem.rs)                         │
└──────────────────────┬──────────────────────────────┘
                       │  reads /proc/[pid]/*
┌──────────────────────▼──────────────────────────────┐
│               /proc Filesystem Parser                │
│   /proc/[pid]/stat, statm, io, status, /proc/meminfo│
│                 (proc_stats.rs)                      │
└─────────────────────────────────────────────────────┘

How it works:

A background worker wakes every second and iterates over all PostgreSQL backend PIDs.
For each PID, it reads /proc/[pid]/stat, /proc/[pid]/statm, /proc/[pid]/io, and /proc/[pid]/status to collect raw CPU ticks, memory pages, I/O byte counters, and context switch counts.
The raw readings are stored in a double-buffered shared memory region. The double-buffer design lets SQL queries read the previous-second snapshot without blocking the worker’s current write.
When you query the pg_pidstat view, the activity module computes per-second deltas (rates) from the raw counters and joins them with pg_stat_activity columns.

This design ensures zero lock contention on reads — the only exclusive lock is a brief window when the background worker swaps the active buffer.

Metrics Reference

The pg_pidstat view includes all standard pg_stat_activity columns plus these monitoring columns:

Column	Type	Description
`cpu_percent`	`float8`	CPU usage as % of total system capacity
`memory_percent`	`float8`	Resident memory as % of total system RAM
`memory_usage_mb`	`float8`	Resident memory in megabytes
`io_read_bytes_per_sec`	`float8`	Disk read throughput (bytes/sec)
`io_write_bytes_per_sec`	`float8`	Disk write throughput (bytes/sec)
`io_read_ops_per_sec`	`float8`	Read IOPS
`io_write_ops_per_sec`	`float8`	Write IOPS
`io_wait`	`bool`	`true` if the process is in disk sleep state
`voluntary_ctxt_switches_per_sec`	`float8`	Voluntary context switches per second
`nonvoluntary_ctxt_switches_per_sec`	`float8`	Non-voluntary context switches per second

Usage Examples

Find CPU-Heavy Backends

SELECT pid, datname, usename, state, query,
       cpu_percent, memory_percent, memory_usage_mb
FROM pg_pidstat
ORDER BY cpu_percent DESC;

Identify I/O-Intensive Queries

SELECT pid, usename, state, query,
       io_read_bytes_per_sec,
       io_write_bytes_per_sec,
       io_read_ops_per_sec,
       io_write_ops_per_sec,
       io_wait
FROM pg_pidstat
WHERE state = 'active'
ORDER BY io_read_bytes_per_sec + io_write_bytes_per_sec DESC;

Detect Context Switch Hotspots

High non-voluntary context switches indicate CPU contention — too many backends fighting for CPU time:

SELECT pid, usename, query,
       voluntary_ctxt_switches_per_sec,
       nonvoluntary_ctxt_switches_per_sec
FROM pg_pidstat
WHERE state = 'active'
ORDER BY nonvoluntary_ctxt_switches_per_sec DESC;

Full Troubleshooting Dashboard

Combine all metrics for a comprehensive view of backend resource usage:

SELECT pid, datname, usename, state,
       left(query, 80) AS query_preview,
       round(cpu_percent::numeric, 2) AS cpu_pct,
       round(memory_usage_mb::numeric, 1) AS mem_mb,
       round((io_read_bytes_per_sec / 1024 / 1024)::numeric, 2) AS read_mb_s,
       round((io_write_bytes_per_sec / 1024 / 1024)::numeric, 2) AS write_mb_s,
       io_wait,
       round(voluntary_ctxt_switches_per_sec::numeric, 0) AS vol_cs,
       round(nonvoluntary_ctxt_switches_per_sec::numeric, 0) AS nvol_cs
FROM pg_pidstat
WHERE state = 'active'
ORDER BY cpu_percent DESC;

Monitoring Over Time with pg_cron

Pair pg_pidstat with pg_cron to build a historical resource usage table:

CREATE TABLE backend_metrics_history (
    captured_at timestamptz DEFAULT now(),
    pid int,
    datname name,
    usename name,
    state text,
    cpu_percent float8,
    memory_usage_mb float8,
    io_read_bytes_per_sec float8,
    io_write_bytes_per_sec float8
);

-- Sample every 10 seconds via pg_cron
SELECT cron.schedule('backend-metrics', '10 seconds',
$$
INSERT INTO backend_metrics_history
    (pid, datname, usename, state, cpu_percent, memory_usage_mb,
     io_read_bytes_per_sec, io_write_bytes_per_sec)
SELECT pid, datname, usename, state, cpu_percent, memory_usage_mb,
       io_read_bytes_per_sec, io_write_bytes_per_sec
FROM pg_pidstat WHERE state = 'active';
$$);

Why pg_pidstat Over External Monitoring?

Aspect	External tools (pidstat, top)	pg_pidstat
PID-to-query mapping	Manual correlation	Automatic (joined with `pg_stat_activity`)
Query interface	Shell parsing / custom scripts	Standard SQL
Alerting integration	Requires glue code	Use any SQL-based alerting
Historical storage	External time-series DB	`INSERT INTO ... SELECT` from the view
Granularity	Per-process only	Per-backend with database context
Deployment	Agent on every host	PostgreSQL extension — no extra process

The key advantage is context: knowing that PID 12345 uses 90% CPU is useful, but knowing it’s the analytics user running a sequential scan on the orders table in the production database is actionable.

Performance Design

pg_pidstat is designed for minimal overhead in production:

Release-mode optimization: opt-level = 3, fat LTO, single codegen unit
Lock-free reads: Double-buffered shared memory means SELECT from the view never blocks
Brief exclusive writes: Only during the buffer swap (microseconds)
Bounded memory: Fixed-size shared memory for up to 1024 backends
1-second sampling: Background worker reads /proc once per second — negligible system load

Conclusion

pg_pidstat brings OS-level process monitoring into PostgreSQL itself. Instead of correlating PIDs across top, pidstat, and pg_stat_activity, you get CPU, memory, I/O, and context switch metrics in a single SQL view — with full connection context attached.

For anyone running PostgreSQL on Linux and needing to quickly identify resource-heavy backends, pg_pidstat turns a multi-tool investigation into one query.

References

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/pg-pidstat-real-time-postgresql-backend-monitoring/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！

pg-walstream: High-Performance PostgreSQL WAL Streaming in Rust

2026-04-21T14:00:00.000Z

pg-walstream: High-Performance PostgreSQL WAL Streaming in Rust

pg-walstream is a Rust library for parsing and streaming PostgreSQL Write-Ahead Log (WAL) messages through logical and physical replication protocols. It provides a type-safe, async-first interface for building real-time Change Data Capture (CDC) pipelines.

If you need to react to database changes in real-time — event-driven architectures, data pipelines, audit logging, cache invalidation, or search index syncing — pg-walstream abstracts the complex PostgreSQL replication protocol into a clean Rust API.

crates.io | API Docs

Background: PostgreSQL WAL and Replication

Before diving into pg-walstream, it helps to understand how PostgreSQL replication works at a fundamental level.

What is WAL (Write-Ahead Logging)?

WAL is PostgreSQL’s mechanism for ensuring data durability. The core idea: write the transaction log first, then write the actual data. Every INSERT, UPDATE, DELETE, and DDL change is recorded sequentially in WAL files before the corresponding data pages are flushed to disk.

This design provides two key benefits:

Crash recovery — if the system crashes, PostgreSQL can replay (REDO) the WAL to recover committed transactions that weren’t yet flushed to disk.
I/O efficiency — sequential WAL writes are much faster than random page writes, so PostgreSQL doesn’t need to flush dirty pages on every commit.

Each WAL record is identified by a Log Sequence Number (LSN) — a monotonically increasing pointer into the WAL stream. LSN is the backbone of replication: it tells both the sender and receiver exactly where they are in the change history.

               WAL Stream
┌───────┬───────┬───────┬───────┬───────┐
│ BEGIN │INSERT │UPDATE │DELETE │COMMIT │  ...
│LSN: 1 │LSN: 2 │LSN: 3 │LSN: 4 │LSN: 5 │
└───────┴───────┴───────┴───────┴───────┘
                      │
           Replayed by standby / consumed by CDC

For a deeper dive into WAL internals, see my earlier post: PostgreSQL WAL (Write-Ahead Logging) mechanism.

Physical vs. Logical Replication

PostgreSQL supports two replication modes, each serving different use cases:

Physical Replication streams raw WAL bytes — the exact disk-level changes — to a standby server. The standby replays these byte-for-byte, producing an identical copy of the primary. This is what powers read replicas and high-availability setups.

Logical Replication decodes WAL into higher-level change events (INSERT, UPDATE, DELETE) using an output plugin (e.g., pgoutput). Instead of raw disk blocks, consumers receive structured messages like “row X was inserted into table Y with these column values.” This is what powers CDC pipelines.

Physical Replication:
  Primary ──[raw WAL bytes]──▶ Standby (byte-identical copy)

Logical Replication:
  Primary ──[pgoutput plugin]──▶ Decoded Change Events ──▶ CDC Consumer
                                  (INSERT/UPDATE/DELETE)

pg-walstream supports both modes — physical replication for standby/backup scenarios and logical replication for CDC.

The Logical Replication Protocol

When a client connects to PostgreSQL in replication mode, they communicate via the Streaming Replication Protocol. For logical replication, the flow works as follows:

Client creates a replication slot — this tells PostgreSQL to retain WAL segments needed by this consumer, preventing them from being recycled.
Client starts replication from a slot, specifying the output plugin (pgoutput) and which publication to subscribe to.
PostgreSQL streams messages — the server continuously sends WAL data messages (XLogData) containing decoded change events.
Client sends feedback — periodically, the client reports its progress (flushed LSN, applied LSN) back to the server. This lets PostgreSQL know which WAL segments can be safely recycled.

The decoded messages follow the logical replication message format, which has evolved across four protocol versions:

Protocol Version	PostgreSQL	Key Additions
v1	14+	Core messages: BEGIN, COMMIT, INSERT, UPDATE, DELETE, TRUNCATE, RELATION, TYPE, ORIGIN
v2	14+	Streaming transactions: STREAM_START, STREAM_STOP, STREAM_COMMIT, STREAM_ABORT — allows consuming large, in-progress transactions before COMMIT
v3	14+	Two-phase commit: BEGIN_PREPARE, PREPARE, COMMIT_PREPARED, ROLLBACK_PREPARED, STREAM_PREPARE
v4	17+	Parallel streaming, `abort_lsn` field for more precise abort handling

Each message type carries specific data. For example, an INSERT message contains:

Relation ID — which table the row belongs to
Tuple data — the column values of the new row, typed by OID

The RELATION message (sent once per table, or when a schema changes) maps the relation ID to a table name, namespace, and column definitions — so the consumer can interpret the tuple data.

Why Build a Library for This?

Implementing the replication protocol from scratch involves:

Managing the PostgreSQL wire protocol and authentication (cleartext, MD5, SCRAM-SHA-256)
Parsing binary WAL messages with protocol-version-specific formats
Tracking LSN positions and sending periodic feedback to avoid WAL bloat
Handling connection drops, retries, and replication slot lifecycle
Dealing with streaming transactions that may arrive interleaved

pg-walstream encapsulates all of this complexity into a type-safe Rust API, so you can focus on what to do with the change events rather than how to receive them.

Key Features

Protocol v1–v4 support including streaming transactions (v2), two-phase commit (v3), and parallel streaming (v4)
Two connection backends: libpq (C FFI, default) and rustls-tls (pure Rust, no runtime C deps)
Zero-copy buffers via the bytes crate — no unnecessary data cloning
Serde-based deserialization — map WAL events directly to Rust structs
Automatic retry with exponential backoff for transient failures
Async/await with tokio and futures::Stream integration
Memory efficient — all configurations stay under 18 MB RSS

Architecture

┌──────────────────────────────────────────┐
│          Application Layer               │
│  (Your CDC / Replication Logic)          │
└──────────────┬───────────────────────────┘
               │
┌──────────────▼───────────────────────────┐
│    LogicalReplicationStream              │
│  - Connection management & retry         │
│  - Event processing & LSN feedback       │
│  - Snapshot export support               │
└──────────────┬───────────────────────────┘
               │
┌──────────────▼───────────────────────────┐
│  LogicalReplicationParser                │
│  - Protocol v1-v4 parsing                │
│  - Zero-copy message deserialization     │
│  - Streaming transaction support         │
└──────────────┬───────────────────────────┘
               │
┌──────────────▼───────────────────────────┐
│     PgReplicationConnection              │
│  ┌─────────────────┬──────────────────┐  │
│  │  libpq backend  │ rustls-tls       │  │
│  │  (C FFI)        │ (pure Rust)      │  │
│  └─────────────────┴──────────────────┘  │
│  Compile-time feature flag selection     │
└──────────────┬───────────────────────────┘
               │
┌──────────────▼───────────────────────────┐
│     BufferReader / BufferWriter          │
│  - Zero-copy operations (bytes crate)    │
│  - Binary protocol handling              │
└──────────────────────────────────────────┘

The library has two connection backends selected at compile time:

Backend	Feature Flag	Dependencies	Description
libpq (default)	`libpq`	`libpq-dev`, `libclang-dev`	FFI wrapper around PostgreSQL’s C client library
rustls-tls	`rustls-tls`	`cmake` (build-time only)	Pure-Rust TLS via rustls + aws-lc-rs (hardware-accelerated)

When both features are enabled, rustls-tls takes priority automatically.

Getting Started

Installation

# Default (libpq backend)
[dependencies]
pg_walstream = "0.6.2"

# Pure-Rust backend (no C runtime deps)
pg_walstream = { version = "0.6.2", default-features = false, features = ["rustls-tls"] }

PostgreSQL Setup

Enable logical replication in postgresql.conf:

1
2
3

wal_level = logical
max_replication_slots = 4
max_wal_senders = 4

Create a publication and replication user:

CREATE PUBLICATION my_publication FOR TABLE users, orders;

CREATE USER replication_user WITH REPLICATION PASSWORD 'secure_password';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO replication_user;

Basic Streaming Example

use pg_walstream::{
    LogicalReplicationStream, ReplicationStreamConfig, RetryConfig,
    StreamingMode, CancellationToken,
};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ReplicationStreamConfig::new(
        "my_slot".to_string(),
        "my_publication".to_string(),
        2,                              // Protocol version
        StreamingMode::On,
        Duration::from_secs(10),        // Feedback interval
        Duration::from_secs(30),        // Connection timeout
        Duration::from_secs(60),        // Health check interval
        RetryConfig::default(),
    );

    let mut stream = LogicalReplicationStream::new(
        "postgresql://postgres:password@localhost:5432/mydb?replication=database",
        config,
    ).await?;

    stream.start(None).await?;
    let cancel_token = CancellationToken::new();

    loop {
        match stream.next_event_with_retry(&cancel_token).await {
            Ok(event) => {
                println!("Received: {:?}", event);
                stream.shared_lsn_feedback.update_applied_lsn(event.lsn.value());
            }
            Err(e) if matches!(e, pg_walstream::ReplicationError::Cancelled(_)) => {
                println!("Shutting down gracefully");
                break;
            }
            Err(e) => {
                eprintln!("Error: {}", e);
                break;
            }
        }
    }

    Ok(())
}

Typed Deserialization

pg-walstream supports mapping WAL events directly to Rust structs via Serde:

use serde::Deserialize;
use pg_walstream::EventType;

#[derive(Debug, Deserialize)]
struct User {
    id: i64,
    username: String,
    email: Option<String>,
    score: f64,
    active: bool,
}

// Inside your event loop:
match &event.event_type {
    EventType::Insert { .. } => {
        let user: User = event.deserialize_insert()?;
        println!("New user: {:?}", user);
    }
    EventType::Update { .. } => {
        let (old, new): (Option<User>, User) = event.deserialize_update()?;
        println!("Updated: {:?} -> {:?}", old, new);
    }
    EventType::Delete { .. } => {
        let user: User = event.deserialize_delete()?;
        println!("Deleted: {:?}", user);
    }
    _ => {}
}

Producer/Consumer Pattern with tokio::spawn

For high-throughput scenarios, decouple WAL reading from event processing using channels:

use tokio::sync::mpsc;
use pg_walstream::{ChangeEvent, ReplicationError};

async fn run_producer(
    mut stream: LogicalReplicationStream,
    cancel_token: CancellationToken,
    tx: mpsc::Sender<ChangeEvent>,
) {
    stream.start(None).await.unwrap();
    loop {
        match stream.next_event_with_retry(&cancel_token).await {
            Ok(event) => {
                stream.shared_lsn_feedback.update_applied_lsn(event.lsn.value());
                if tx.send(event).await.is_err() { break; }
            }
            Err(ReplicationError::Cancelled(_)) => break,
            Err(e) => {
                eprintln!("Fatal: {e}");
                break;
            }
        }
    }
    stream.stop().await.ok();
}

Load Testing Results

The library was benchmarked on an 8-core Intel Xeon Platinum 8370C (16 GB RAM, Ubuntu 22.04) across three PostgreSQL configurations:

PG16 with protocol v4 + parallel streaming (rustls-tls backend)
PG18 with binary mode + direct TLS negotiation
PG18 + COPY with COPY generator optimization

DML Throughput (events/sec)

Scenario	PG16	PG18 Binary+DirectTLS	PG18 +COPY
Baseline	148,533	168,205	209,193
Batch-5000	132,623	151,820	190,687
4-Writers	135,036	159,233	193,354
Mixed-DML	42,198	176,580	186,019
Batch-100	22,270	141,597	199,780
Wide-20col	18,917	172,772	173,283
Payload-2KB	14,017	114,884	134,323

Peak throughput: 209,193 events/sec (PG18 + COPY, Baseline scenario).

Data Throughput (MB/s)

Scenario	PG16	PG18 Binary+DirectTLS	PG18 +COPY
Baseline	30.4	31.1	38.7
Wide-20col	21.3	50.7	51.5
Payload-2KB	28.3	43.9	57.1
Mixed-DML	7.9	32.1	33.8
Batch-100	4.6	26.2	37.0

Best data throughput: 57.1 MB/s (PG18 + COPY, Payload-2KB scenario).

Stress Scaling: 16 to 192 Concurrent Writers

Writers	PG16	PG18 Binary+DirectTLS	PG18 +COPY
16	125,657	130,625	185,044
32	111,970	133,880	184,718
64	103,937	125,082	182,349
128	87,352	109,594	160,293
192	71,316	98,482	171,585

Under high concurrency (16 to 192 writers), PG16 degrades by 43% while PG18 + COPY only degrades by ~7%, demonstrating significantly better scalability.

CPU Efficiency (events/sec per 1% CPU)

Scenario	PG16	PG18 Binary+DirectTLS	PG18 +COPY
Baseline	5,689	5,637	5,920
Batch-5000	5,379	5,733	5,440
Wide-20col	2,369	5,059	5,517
Batch-100	3,966	5,572	5,693

PG18 variants deliver consistently higher CPU efficiency, averaging 5,200+ events/sec per 1% CPU compared to PG16’s ~4,700.

Memory Usage

All configurations remain extremely lightweight — between 15–18 MB RSS regardless of load. Memory stays flat even under 192 concurrent writers, demonstrating the zero-copy buffer design pays off.

Key Takeaways from Load Tests

PG18 + COPY + binary mode is the clear winner, peaking at 209K events/sec
Stress resilience — PG18 + COPY maintains throughput under heavy concurrency where PG16 degrades sharply
CPU efficient — the rustls-tls backend uses ~3x less CPU than libpq in prior benchmarks (4,252 vs 1,628 events/sec per 1% CPU)
Memory stable — sub-18 MB footprint under all tested conditions
Binary mode + direct TLS provide significant improvements even without COPY optimization

Performance Design Decisions

Several design choices contribute to pg-walstream’s performance:

SmallVec for tuple data — up to 16 columns stored inline on the stack, avoiding heap allocation for common cases
Custom OidHasher — eliminates SipHash overhead for 32-bit OID integer keys
Arc<str> for column/namespace names — shared immutable strings across events
CachePadded atomics for LSN feedback — avoids false sharing in concurrent scenarios
Feedback throttling — time checks only every 128 events via bitmask (count & 0x7F == 0)

TCP Tuning for Production

For high-throughput deployments, the following Linux kernel parameters are recommended:

net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 262144 67108864
net.ipv4.tcp_wmem = 4096 262144 67108864
net.ipv4.tcp_congestion_control = bbr
net.core.netdev_max_backlog = 5000

Conclusion

pg-walstream fills a gap in the Rust ecosystem for a production-grade PostgreSQL WAL streaming library. With protocol v1–v4 support, dual connection backends, zero-copy parsing, and throughput exceeding 200K events/sec, it provides a solid foundation for building CDC pipelines, event-driven systems, and real-time data synchronization.

The load testing results demonstrate that pairing pg-walstream with PostgreSQL 18’s binary mode and COPY optimization delivers exceptional performance and scalability — maintaining high throughput even under 192 concurrent writers while keeping memory usage under 18 MB.

References

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/pg-walstream-rust-postgresql-wal-streaming/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！

pg2any: A Rust CDC Library for Streaming PostgreSQL Changes to Any Database

2025-09-09T13:10:43.000Z

What is pg2any?

pg2any is a Rust library (published as pg2any_lib on crates.io) that builds production-ready Change Data Capture (CDC) pipelines. It reads PostgreSQL’s Write-Ahead Log (WAL) through logical replication and replays changes — inserts, updates, deletes, and truncates — to destination databases in real time.

Supported destinations:

MySQL (via SQLx)
SQL Server (via Tiberius)
SQLite (via SQLx)

Each destination is behind a Cargo feature flag (mysql, sqlserver, sqlite), so you compile only the drivers you need.

A ready-to-run example application lives at pg2any-example, and the underlying PostgreSQL streaming replication protocol is handled by the companion crate pg_walstream.

Architecture

pg2any follows a producer–consumer pattern with file-based transaction persistence as the intermediary. This design gives crash safety: if the process dies mid-stream, committed-but-unexecuted transactions survive on disk and are replayed on restart.

┌─────────────────┐        ┌──────────────────────────────────────┐       ┌──────────────────┐
│   PostgreSQL    │        │          pg2any CDC Engine            │       │   Destination    │
│                 │  WAL   │                                      │  SQL  │  MySQL / MSSQL   │
│  Logical Repli- │───────▶│  ┌──────────┐     ┌───────────────┐ │──────▶│  / SQLite        │
│  cation Stream  │        │  │ Producer  │────▶│ File Storage  │ │       │                  │
│                 │        │  └──────────┘     └───────┬───────┘ │       └──────────────────┘
└─────────────────┘        │                          │         │
                           │                   ┌──────▼───────┐ │       ┌──────────────────┐
                           │                   │   Consumer   │ │       │   Prometheus     │
                           │                   │ (Priority Q) │ │       │   /metrics       │
                           │                   └──────────────┘ │       │   /health        │
                           └──────────────────────────────────────┘       └──────────────────┘

Three-Directory Transaction Lifecycle

Transactions flow through three directories during their lifetime:

Directory	Purpose
`sql_data_tx/`	Stores actual SQL content. Files are append-only and rotate at 64 MB segments for large transactions.
`sql_received_tx/`	Metadata for in-progress transactions (created at `BEGIN`).
`sql_pending_tx/`	Metadata for committed transactions ready for the consumer (atomically moved from `sql_received_tx/` on `COMMIT`).

This three-phase approach means that only fully committed transactions ever reach the consumer, and incomplete transactions are cleaned up on restart.

Producer

The producer reads the logical replication stream event by event:

On BEGIN — creates a metadata file in sql_received_tx/ and a data file in sql_data_tx/.
On INSERT / UPDATE / DELETE / TRUNCATE — converts each event to destination-dialect SQL and appends it to the data file via a BufferedEventWriter.
On COMMIT — atomically moves metadata from sql_received_tx/ to sql_pending_tx/, making the transaction visible to the consumer.

For protocol version 2+, the producer also handles streaming transactions (StreamStart / StreamStop / StreamCommit), which allow PostgreSQL to send chunks of large in-progress transactions before the final commit.

Consumer

The consumer maintains a priority queue ordered by commit LSN to guarantee correct replay order:

Reads pending transaction metadata from sql_pending_tx/.
Parses SQL from sql_data_tx/ using a streaming SQL parser (constant memory regardless of transaction size).
Executes statements atomically in a destination-side database transaction.
Invokes a PreCommitHook — a callback that runs inside the destination transaction before COMMIT, used to atomically persist the LSN checkpoint alongside the data. This eliminates the window where data is committed but the checkpoint is not (or vice versa).
Commits, then deletes processed files.

Crash Recovery

On startup, pg2any scans:

sql_received_tx/ for incomplete transactions → aborts them.
sql_pending_tx/ for committed-but-unexecuted transactions → replays them.

The LsnTracker persists the last successfully applied LSN, so replication resumes exactly where it left off.

Key Features

DML Coalescing

One of pg2any’s most impactful optimizations. Instead of executing individual DML statements one by one, the coalescing engine merges consecutive same-table operations:

Multiple INSERTs → multi-value INSERT INTO ... VALUES (...), (...), (...).
Multiple UPDATEs → CASE-WHEN batch updates.
Multiple DELETEs → combined WHERE clauses with OR.

This is applied across all three destination types, with dialect-aware identifier quoting (backticks for MySQL, brackets for SQL Server, double quotes for SQLite) and respects max_allowed_packet limits (MySQL) with an 80% safety margin.

Compressed Storage

When enabled via PG2ANY_ENABLE_COMPRESSION=true, transaction files are stored as .sql.gz with accompanying .sql.gz.idx index files. Sync points are created every 1,000 statements, enabling O(1) seeking to arbitrary positions without decompressing the entire file — critical for efficient crash recovery of large transactions.

Monitoring

With the metrics feature enabled, pg2any exposes a Prometheus-compatible HTTP server (default port 8080):

GET /metrics — Prometheus text format with event counters, LSN progress, processing rates, error counts, and transaction statistics.
GET /health — JSON health status.

Metrics use AtomicU64 counters (lock-free) to minimize overhead on the hot path. When compiled without the metrics feature, all metric calls become zero-cost no-ops.

Protocol Version Support

Version	Capabilities
v1	Basic logical replication (BEGIN, INSERT, UPDATE, DELETE, TRUNCATE, COMMIT)
v2	Adds streaming transactions for large in-progress transactions
v3	Adds two-phase commit support
v4	Additional protocol capabilities

Quick Start

Prerequisites

PostgreSQL 10+ with wal_level = logical
A destination database (MySQL 8.0+, SQL Server, or SQLite)

PostgreSQL Setup

-- Verify logical replication is enabled
SHOW wal_level;           -- must be 'logical'
SHOW max_replication_slots;
SHOW max_wal_senders;

-- Create a publication for the tables you want to replicate
CREATE PUBLICATION cdc_pub FOR ALL TABLES;
-- Or for specific tables:
-- CREATE PUBLICATION cdc_pub FOR TABLE orders, customers;

Using pg2any as a Library

Add pg2any_lib to your Cargo.toml with the destination features you need:

1
2
3

[dependencies]
pg2any_lib = { version = "0.9", features = ["mysql", "metrics"] }
tokio = { version = "1", features = ["full"] }

Configuration via Environment Variables

Variable	Description	Default
`CDC_SOURCE_CONNECTION_STRING`	PostgreSQL URI with `?replication=database`	Required
`CDC_DEST_TYPE`	`MySQL`, `SqlServer`, or `SQLite`	Required
`CDC_DEST_URI`	Destination connection string	Required
`CDC_REPLICATION_SLOT`	Replication slot name	Required
`CDC_PUBLICATION`	Publication name	Required
`CDC_SCHEMA_MAPPING`	Comma-separated `source:dest` pairs (e.g., `public:cdc_db`)	None
`CDC_PROTOCOL_VERSION`	Protocol version (1–4)	`1`
`CDC_STREAMING_MODE`	Enable streaming transactions (requires v2+)	`false`
`CDC_BINARY_MODE`	Binary format for protocol	`false`
`CDC_CONNECTION_TIMEOUT`	Connection timeout (seconds)	`30`
`CDC_QUERY_TIMEOUT`	Query timeout (seconds)	`60`
`CDC_BUFFER_SIZE`	Transaction channel queue capacity	`1000`
`CDC_BATCH_SIZE`	Batch size for destination inserts	`1000`
`PG2ANY_ENABLE_COMPRESSION`	Enable gzip compression for SQL files	`false`
`PG2ANY_METRICS_PORT`	Prometheus HTTP port	`8080`

Run with Docker Compose

The example project includes a full Docker Compose stack with PostgreSQL, MySQL, the CDC application, and Prometheus:

git clone https://github.com/isdaniel/rust_playground.git
cd rust_playground/pg2any-example

# Start all services
docker-compose up -d

# Watch CDC logs
docker-compose logs -f cdc_app

Design Decisions Worth Noting

File-based persistence over in-memory queues — Using the filesystem as the intermediary between producer and consumer trades some latency for crash safety. If the process is killed, no committed transaction data is lost.

PreCommitHook for atomic checkpoints — Executing the LSN checkpoint update inside the same destination transaction as the data changes eliminates an entire class of consistency bugs where the checkpoint and data can diverge.

Feature-gated compilation — Database drivers and monitoring are behind Cargo features, so the binary only includes what you actually use. This reduces compile time, binary size, and attack surface.

Transaction segmentation at 64 MB — Large transactions (e.g., bulk imports) are split across multiple files to prevent unbounded memory and disk usage.

Testing

pg2any has 104+ tests across 16 test files, covering:

Integration tests for all three destination types
Streaming transaction correctness
Compression and large file handling
WHERE clause generation for UPDATE/DELETE with various replica identity configurations
Position tracking for crash recovery
Metrics logic

Beyond unit and integration tests, the project runs chaos testing in CI — randomly restarting the CDC application during pgbench workloads to validate graceful shutdown and recovery under real conditions.

Links

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/pg2any-rust-introduce/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！

Building a PostgreSQL Foreign Data Wrapper (FDW) in Rust with pgrx

2025-06-30T14:30:11.000Z

🚀 Building a Simple PostgreSQL FDW with Rust and pgrx

PostgreSQL Foreign Data Wrappers (FDW) enable PostgreSQL to query external data sources as if they were regular tables. Traditionally, FDWs are written in C, but with pgrx, we can now build PostgreSQL extensions — including FDWs — in Rust, unlocking safety and modern tooling.

In this post, we’ll walk through creating a simple FDW using Rust and pgrx that simulates reading rows from an external source (e.g., Redis or API). It is deliberately minimal so the core FDW lifecycle stays visible end to end.

🛠️ What Can You Build with `pgrx`?

✅ SQL Functions: Scalar, aggregate, and set-returning functions.
✅ Custom Types: Define composite types or enums in Rust.
✅ Foreign Data Wrappers (FDWs): Like the one in your example — connect PostgreSQL to external systems (Redis, APIs, file systems, etc.).
✅ Index Access Methods: Implement new index types.
✅ Background Workers: Run tasks in the background inside PostgreSQL.
✅ Hooks: Intercept or modify PostgreSQL internal behavior (like planner or executor hooks).

🌐 Under the hood:

PostgreSQL communicates via C APIs.
pgrx provides Rust-safe bindings to these APIs.
Memory management is handled carefully via PgMemoryContexts, matching PostgreSQL’s memory context model.
Rust functions are exposed to PostgreSQL as SQL-callable functions with the #[pg_extern] macro.

🏗️ Key Components of an default_fdw

PostgreSQL FDWs consist of several callback functions that handle different phases of query planning and execution:

Example extension default_fdw

Planning Phase:
- GetForeignRelSize: Estimate rows.
- GetForeignPaths: Generate access paths.
- GetForeignPlan: Create the scan plan.
Execution Phase:
- BeginForeignScan: Initialize the scan.
- IterateForeignScan: Produce each row.
- ReScanForeignScan: Restart the scan if needed.
- EndForeignScan: Cleanup.

How to use

create foreign data wrapper default_wrapper
  handler default_fdw_handler;

create server my_default_server
  foreign data wrapper default_wrapper
  options (
    foo 'bar'
  );

create foreign table hello (
  id bigint,
  col text
)
server my_default_server options (
	foo 'bar'
);

Then we can select hello table.

1	select * from hello;

🔧 Setting Up the FDW Handler

The entry point is the FDW handler function, which PostgreSQL calls to retrieve a set of function pointers.

#[pg_extern(create_or_replace)]
pub extern "C" fn default_fdw_handler() -> PgBox<pg_sys::FdwRoutine> {
    log!("> default_fdw_handler");
    unsafe {
        let mut fdw_routine = PgBox::<pg_sys::FdwRoutine, AllocatedByRust>::alloc_node(pg_sys::NodeTag::T_FdwRoutine);

        // Planning callbacks
        fdw_routine.GetForeignRelSize = Some(get_foreign_rel_size);
        fdw_routine.GetForeignPaths = Some(get_foreign_paths);
        fdw_routine.GetForeignPlan = Some(get_foreign_plan);
        fdw_routine.ExplainForeignScan = Some(explain_foreign_scan);

        // Execution callbacks
        fdw_routine.BeginForeignScan = Some(begin_foreign_scan);
        fdw_routine.IterateForeignScan = Some(iterate_foreign_scan);
        fdw_routine.ReScanForeignScan = Some(re_scan_foreign_scan);
        fdw_routine.EndForeignScan = Some(end_foreign_scan);

        fdw_routine.into_pg_boxed()
    }
}

📦 Extracting Foreign Table Options

PostgreSQL allows specifying options like hostnames or credentials when creating a foreign table. This function retrieves those options:

1
2
3

unsafe fn get_foreign_table_options(relid: pg_sys::Oid) -> HashMap<String, String> {
    ...
}

This is crucial when your FDW needs to connect to external systems like Redis, REST APIs, or filesystems.

📊 Planner Callbacks

1️⃣ GetForeignRelSize

Estimates the number of rows in the foreign table.

#[pg_guard]
extern "C" fn get_foreign_rel_size(..., baserel: *mut pg_sys::RelOptInfo, ...) {
    log!("> get_foreign_rel_size");
    unsafe {
        (*baserel).rows = 1000.0;
    }
}

2️⃣ GetForeignPaths

Defines possible access paths for the planner.

#[pg_guard]
extern "C" fn get_foreign_paths(..., baserel: *mut pg_sys::RelOptInfo, ...) {
    log!("> get_foreign_paths");
    unsafe {
        let path = pg_sys::create_foreignscan_path(
            ...,
            (*baserel).rows,
            10.0,   // startup cost
            100.0,  // total cost
            ...
        );
        pg_sys::add_path(baserel, path as *mut pg_sys::Path);
    }
}

3️⃣ GetForeignPlan

Generates the actual execution plan.

#[pg_guard]
extern "C" fn get_foreign_plan(...) -> *mut pg_sys::ForeignScan {
    log!("> get_foreign_plan");
    unsafe {
        pg_sys::make_foreignscan(...)
    }
}

▶️ Execution Callbacks

🏁 BeginForeignScan

Initializes the scan.

#[pg_guard]
extern "C" fn begin_foreign_scan(node: *mut pg_sys::ForeignScanState, ...) {
    log!("> begin_foreign_scan");
    unsafe {
        let relid = (*(*node).ss.ss_currentRelation).rd_id;
        let options = get_foreign_table_options(relid);
        log!("Foreign table options: {:?}", options);

        let state = PgMemoryContexts::CurrentMemoryContext
            .leak_and_drop_on_delete(RedisFdwState { row: 0 });

        (*node).fdw_state = state as *mut std::ffi::c_void;
    }
}

🔁 IterateForeignScan

Produces rows one at a time.

#[pg_guard]
extern "C" fn iterate_foreign_scan(node: *mut pg_sys::ForeignScanState) -> *mut pg_sys::TupleTableSlot {
    log!("> iterate_foreign_scan");

    unsafe {
        let state = &mut *((*node).fdw_state as *mut RedisFdwState);
        let slot = (*node).ss.ss_ScanTupleSlot;
        let tupdesc = (*slot).tts_tupleDescriptor;
        let natts = (*tupdesc).natts as usize;

        if state.row >= 5 {
            exec_clear_tuple(slot);
            return slot;
        }

        exec_clear_tuple(slot);

        let values_ptr = PgMemoryContexts::For((*slot).tts_mcxt)
            .palloc(std::mem::size_of::<pg_sys::Datum>() * natts) as *mut pg_sys::Datum;

        let nulls_ptr = PgMemoryContexts::For((*slot).tts_mcxt)
            .palloc(std::mem::size_of::<bool>() * natts) as *mut bool;

        *values_ptr.add(0) = (state.row + 1).into();
        let name = format!("hello_{}", state.row + 1);
        let cstring = CString::new(name).unwrap();
        *values_ptr.add(1) = Datum::from(pg_sys::cstring_to_text(cstring.as_ptr()));

        *nulls_ptr.add(0) = false;
        *nulls_ptr.add(1) = false;

        (*slot).tts_values = values_ptr;
        (*slot).tts_isnull = nulls_ptr;

        pg_sys::ExecStoreVirtualTuple(slot);

        state.row += 1;

        slot
    }
}

This example emits five rows with (id, name) pairs like (1, hello_1).

🔄 ReScanForeignScan

Handles rescan requests.

#[pg_guard]
extern "C" fn re_scan_foreign_scan(_node: *mut pg_sys::ForeignScanState) {
    log!("> re_scan_foreign_scan");
}

🛑 EndForeignScan

Frees resources.

#[pg_guard]
extern "C" fn end_foreign_scan(node: *mut pg_sys::ForeignScanState) {
    log!("> end_foreign_scan");
    unsafe {
        if !(*node).fdw_state.is_null() {
            (*node).fdw_state = std::ptr::null_mut();
        }
    }
}

🗒️ Utilities

Tuple clearing is handled by this helper:

unsafe fn exec_clear_tuple(slot: *mut pg_sys::TupleTableSlot) {
    if let Some(clear) = (*(*slot).tts_ops).clear {
        clear(slot);
    }
}

🏁 Conclusion

This post walked you through the basics of building a PostgreSQL FDW using Rust and pgrx. While this example generates dummy data, the same structure can be extended to connect with real-world systems like Redis, REST APIs, or message queues.

🚀 Next Steps

Add connection logic to Redis or any backend.
Support INSERT, UPDATE, DELETE by implementing the modification callbacks.
Package and distribute as a PostgreSQL extension.

📚 References

If you’d like, I can help refine this post further, format it for Medium/Dev.to, or extend it with Redis connection examples. Would you like that?

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/rust-pgrx-extension-fdw/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！

AI-Powered PostgreSQL Performance Tuning with MCP - Introducing pgtuner_mcp

2025-01-13T02:30:00.000Z

AI-Powered PostgreSQL Performance Tuning with MCP - Introducing pgtuner_mcp

Database performance optimization is one of the most critical yet challenging aspects of maintaining production systems. Identifying slow queries, optimizing indexes, and monitoring database health requires deep expertise and constant vigilance. Today, I’m excited to introduce pgtuner_mcp, a Model Context Protocol (MCP) server that brings AI-powered PostgreSQL performance tuning capabilities directly into your development workflow.

What is pgtuner_mcp?

pgtuner_mcp is an intelligent PostgreSQL performance analysis server built on the Model Context Protocol (MCP). It bridges the gap between AI assistants (like Claude) and your PostgreSQL database, enabling natural language interactions for complex database optimization tasks.

GitHub Repository

Key Capabilities

Intelligent Query Analysis: Identify slow queries with detailed statistics from pg_stat_statements
AI-Powered Index Recommendations: Get smart indexing suggestions based on actual workload patterns
Hypothetical Index Testing: Test indexes without creating them using HypoPG
Comprehensive Health Checks: Monitor connections, cache efficiency, locks, and replication
Bloat Detection: Identify and quantify table/index bloat for maintenance
Vacuum Monitoring: Track vacuum operations and autovacuum effectiveness
I/O Analysis: Analyze disk read/write patterns and identify bottlenecks
Configuration Review: Get recommendations for memory, checkpoint, and connection settings

Architecture Overview

pgtuner_mcp leverages several PostgreSQL extensions and Python libraries to provide comprehensive analysi

┌─────────────────┐
│  AI Assistant   │
│  (Claude, etc)  │
└────────┬────────┘
         │ MCP Protocol
         ▼
┌─────────────────┐    ┌──────────────────┐
│  pgtuner_mcp    │───▶│   PostgreSQL     │
│  (Python MCP    │    │   + Extensions   │
│   Server)       │    │  - pg_stat_      │
└─────────────────┘    │    statements    │
         │              │  - hypopg        │
         │              │  - pgstattuple   │
         ▼              └──────────────────┘
┌─────────────────┐
│  Analysis &     │
│  Recommendations│
└─────────────────┘

Core Components

MCP Server: Provides tools, prompts, and resources via Model Context Protocol
Query Analyzer: Parses and analyzes SQL using pglast library
Performance Metrics: Collects statistics from PostgreSQL system views
AI Recommendations: Generates intelligent suggestions based on workload patterns
Multiple Transport Modes: Supports stdio, SSE, and streamable HTTP

Installation and Setup

Prerequisites

Before installing pgtuner_mcp, ensure you have:

Python 3.10+
PostgreSQL 12+ (recommended: 14+)
Access to install PostgreSQL extensions

Quick Installation

# Install via pip
pip install pgtuner_mcp

# Or using uv (faster)
uv pip install pgtuner_mcp

PostgreSQL Extensions Setup

pgtuner_mcp requires specific PostgreSQL extensions for full functionality:

1. pg_stat_statements (Required)

This extension tracks query execution statistics:

-- Add to postgresql.conf
shared_preload_libraries = 'pg_stat_statements'
compute_query_id = o
pg_stat_statements.max = 10000
pg_stat_statements.track = top
pg_stat_statements.track_utility = on

-- Restart PostgreSQL, then:
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Verify installation
SELECT * FROM pg_stat_statements LIMIT 1;

2. HypoPG (Optional, Recommended)

Enables hypothetical index testing without disk usage:

CREATE EXTENSION IF NOT EXISTS hypopg;

-- Verify installation
SELECT * FROM hypopg_list_indexes();

3. pgstattuple (Optional, for Bloat Detection)

Provides tuple-level statistics for bloat analysis:

CREATE EXTENSION IF NOT EXISTS pgstattuple;

-- Verify installation
SELECT * FROM pgstattuple('pg_class') LIMIT 1;

User Permissions

Create a dedicated monitoring user with minimal required permissions:

-- Create monitoring user
CREATE USER pgtuner_monitor WITH PASSWORD 'secure_password';

-- Grant connection and schema access
GRANT CONNECT ON DATABASE your_database TO pgtuner_monitor;
GRANT USAGE ON SCHEMA public TO pgtuner_monitor;

-- Grant read access to user tables
GRANT SELECT ON ALL TABLES IN SCHEMA public TO pgtuner_monitor;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT SELECT ON TABLES TO pgtuner_monitor;

-- Grant system statistics access (PostgreSQL 10+)
GRANT pg_read_all_stats TO pgtuner_monitor;

-- Grant access to pg_stat_statements
GRANT SELECT ON pg_stat_statements TO pgtuner_monitor;
GRANT SELECT ON pg_stat_statements_info TO pgtuner_monitor;

-- For bloat detection (PostgreSQL 14+)
GRANT pg_stat_scan_tables TO pgtuner_monitor;

-- For HypoPG functions
GRANT SELECT ON hypopg_list_indexes TO pgtuner_monitor;
GRANT EXECUTE ON FUNCTION hypopg_create_index(text) TO pgtuner_monitor;
GRANT EXECUTE ON FUNCTION hypopg_drop_index(oid) TO pgtuner_monitor;
GRANT EXECUTE ON FUNCTION hypopg_reset() TO pgtuner_monitor;

Configuration

Server Modes

pgtuner_mcp supports three deployment modes:

1. Standard MCP Mode (stdio)

Best for MCP clients like Claude Desktop or Cline:

# Default mode
python -m pgtuner_mcp

# Explicit stdio mode
python -m pgtuner_mcp --mode stdio

Configuration for Claude Desktop (cline_mcp_settings.json):

{
  "mcpServers": {
    "pgtuner_mcp": {
      "command": "python",
      "args": ["-m", "pgtuner_mcp"],
      "env": {
        "DATABASE_URI": "postgresql://pgtuner_monitor:password@localhost:5432/mydb"
      },
      "disabled": false,
      "autoApprove": []
    }
  }
}

2. HTTP SSE Mode (Legacy Web Applications)

Server-Sent Events for web-based MCP communication:

# Start SSE server
python -m pgtuner_mcp --mode sse --host 0.0.0.0 --port 8080

# With debug mode
python -m pgtuner_mcp --mode sse --debug

Endpoints:

GET /sse - SSE connection endpoint
POST /messages - Send messages/requests

3. Streamable HTTP Mode (Recommended for Web)

Modern MCP protocol with single /mcp endpoint:

# Stateful mode (maintains session state)
python -m pgtuner_mcp --mode streamable-http

# Stateless mode (serverless-friendly)
python -m pgtuner_mcp --mode streamable-http --stateless

# Custom host/port
python -m pgtuner_mcp --mode streamable-http --host localhost --port 8080

Configuration:

{
  "mcpServers": {
    "pgtuner_mcp": {
      "type": "http",
      "url": "http://localhost:8080/mcp"
    }
  }
}

Environment Variables

Variable	Description	Required
`DATABASE_URI`	PostgreSQL connection string	Yes
`PGTUNER_EXCLUDE_USERIDS`	Comma-separated user OIDs to exclude	No

Connection String Format:

1	postgresql://user:password@host:port/database

Available Tools

pgtuner_mcp provides 15+ specialized tools organized into categories:

Performance Analysis Tools

Tool	Description
`get_slow_queries`	Retrieve slow queries with detailed statistics (time, calls, cache hit ratio)
`analyze_query`	Analyze execution plans with EXPLAIN ANALYZE and automated issue detection
`get_table_stats`	Get table statistics: size, row counts, dead tuples, access patterns
`analyze_disk_io_patterns`	Analyze I/O patterns, identify hot tables and bottlenecks

Index Tuning Tools

Tool	Description
`get_index_recommendations`	AI-powered index recommendations based on workload analysis
`explain_with_indexes`	Test hypothetical indexes without creating them
`manage_hypothetical_indexes`	Create, list, drop, or reset HypoPG hypothetical indexes
`find_unused_indexes`	Find unused and duplicate indexes for cleanup

Database Health Tools

Tool	Description
`check_database_health`	Comprehensive health check with scoring
`get_active_queries`	Monitor active queries and find long-running transactions
`analyze_wait_events`	Identify I/O, lock, or CPU bottlenecks
`review_settings`	Review PostgreSQL configuration with recommendations

Bloat Detection Tools

Tool	Description
`analyze_table_bloat`	Analyze table bloat using pgstattuple extension
`analyze_index_bloat`	Analyze B-tree index bloat (also supports GIN/Hash)
`get_bloat_summary`	Comprehensive bloat overview with maintenance priorities

Vacuum Monitoring Tools

Tool	Description
`monitor_vacuum_progress`	Track VACUUM, VACUUM FULL, and autovacuum operations

Docker Deployment

pgtuner_mcp is available as a Docker image for easy deployment:

# Pull the image
docker pull dog830228/pgtuner_mcp

# Run in streamable HTTP mode (recommended)
docker run -p 8080:8080 \
  -e DATABASE_URI=postgresql://user:pass@host:5432/db \
  dog830228/pgtuner_mcp --mode streamable-http

# Run in stateless mode (serverless-friendly)
docker run -p 8080:8080 \
  -e DATABASE_URI=postgresql://user:pass@host:5432/db \
  dog830228/pgtuner_mcp --mode streamable-http --stateless

# Run in stdio mode for MCP clients
docker run -i \
  -e DATABASE_URI=postgresql://user:pass@host:5432/db \
  dog830228/pgtuner_mcp --mode stdio

Real-World Use Cases

1. Slow Query Investigation

Scenario: Application experiencing slow response times.

Workflow:

User: "Find the slowest queries in my database"
AI + pgtuner_mcp:
  1. Calls get_slow_queries(limit=10, order_by="total_time")
  2. Identifies top 3 problematic queries
  3. Calls analyze_query() for each
  4. Detects sequential scans and missing indexes
  5. Calls get_index_recommendations()
  6. Provides CREATE INDEX statements with impact estimates

2. Index Optimization

Scenario: Database growing, need to optimize indexes.

Workflow:

User: "Help me optimize my database indexes"
AI + pgtuner_mcp:
  1. Calls find_unused_indexes() to identify cleanup candidates
  2. Calls get_index_recommendations() for new index suggestions
  3. Calls explain_with_indexes() to test hypothetical indexes
  4. Estimates storage savings and performance improvements
  5. Provides prioritized action plan

3. Health Check Before Production Deploy

Scenario: Pre-deployment database health validation.

Workflow:

User: "Is my database ready for production traffic?"
AI + pgtuner_mcp:
  1. Calls check_database_health(verbose=True)
  2. Analyzes connection pool capacity
  3. Checks cache hit ratios
  4. Reviews vacuum and autovacuum status
  5. Analyzes wait events
  6. Reviews configuration settings
  7. Provides comprehensive health report with recommendations

4. Performance Regression Investigation

Scenario: Performance degraded after recent changes.

Workflow:

User: "Why is my database slower than last week?"
AI + pgtuner_mcp:
  1. Calls get_table_stats() to identify growth patterns
  2. Calls analyze_disk_io_patterns() for I/O bottlenecks
  3. Calls get_bloat_summary() to detect table/index bloat
  4. Calls monitor_vacuum_progress() to check maintenance
  5. Calls analyze_wait_events() to find resource contention
  6. Identifies root causes and provides remediation steps

Performance Considerations

Extension Overhead

Extension	Performance Impact	Recommendation
`pg_stat_statements`	Low (~1-2%)	Always enable
`track_io_timing`	Low-Medium (~2-5%)	Enable in production, test first
`track_functions = all`	Low	Enable for function-heavy workloads
`pgstattuple` functions	Varies by table size	Use `_approx` for large tables
`HypoPG`	Zero (in-memory only)	Safe for all environments

Tip: Use pg_test_timing to measure timing overhead on your specific hardware:

1	SELECT pg_test_timing();

Best Practices

Use Approximate Analysis: For large tables (>5GB), use pgstattuple_approx instead of pgstattuple
Filter System Users: Exclude monitoring/replication users using PGTUNER_EXCLUDE_USERIDS
Limit Query History: Configure pg_stat_statements.max based on your workload
Regular Maintenance: Use vacuum monitoring tools to ensure optimal performance
Test Hypothetical Indexes: Always test with HypoPG before creating real indexes

Conclusion

pgtuner_mcp represents a paradigm shift in database performance optimization. By combining the power of AI assistants with deep PostgreSQL expertise through the Model Context Protocol, it makes advanced database tuning accessible to developers at all skill levels.

The tool doesn’t replace database administrators—it augments their capabilities and democratizes access to expert-level analysis. Whether you’re debugging a slow query, planning index strategies, or conducting pre-deployment health checks, pgtuner_mcp provides intelligent, context-aware assistance.

Key Takeaways

AI-Native Performance Tuning: Natural language interface to complex database operations
Risk-Free Testing: HypoPG enables index testing without disk usage
Comprehensive Analysis: 15+ tools covering queries, indexes, health, bloat, vacuum, and I/O
Flexible Deployment: stdio, HTTP SSE, or streamable HTTP modes
Production-Ready: Minimal overhead, proper permissions, comprehensive monitoring

Whether you’re a seasoned DBA looking to leverage AI for faster workflows or a developer seeking to understand and optimize database performance, pgtuner_mcp offers a powerful, modern approach to PostgreSQL tuning.

Resources

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/pgtuner-mcp-ai-powered-postgresql-performance/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！

Building Safe PostgreSQL Extensions with Rust - Introducing pg_where_guard

2025-01-02T23:20:00.000Z

Building Safe PostgreSQL Extensions with Rust - Introducing pg_where_guard

Database safety is a critical concern for any production system. Accidental data loss from DELETE or UPDATE statements without WHERE clauses can be catastrophic. Today, I’ll introduce pg_where_guard, a PostgreSQL extension built with Rust and the pgrx framework that prevents these dangerous operations.

What is pg_where_guard?

pg_where_guard is a PostgreSQL extension that acts as a safety net for your database by intercepting and blocking potentially dangerous SQL operations:

DELETE Protection: Prevents DELETE FROM table without WHERE clause
UPDATE Protection: Prevents UPDATE table SET ... without WHERE clause
CTE Support: Recursively checks Common Table Expressions
Hook Integration: Uses PostgreSQL’s post_parse_analyze_hook for query interception
Memory Safe: Written in Rust with pgrx for safety and performance

GitHub Repository

Why Rust for PostgreSQL Extensions?

Building PostgreSQL extensions traditionally meant working with C and dealing with manual memory management, potential segmentation faults, and complex debugging. Rust changes this paradigm by offering:

Performance

Zero-cost abstractions mean Rust code performs as well as equivalent C code while being much safer.

pgrx Framework

The pgrx framework provides:

Type-safe PostgreSQL API bindings
Automatic SQL schema generation
Comprehensive testing support
Easy development workflow

Technical Architecture

Hook-Based Implementation

pg_where_guard leverages PostgreSQL’s hook system to intercept queries after parsing:

1
2
3

// Hook registration in _PG_init
PREV_POST_PARSE_ANALYZE_HOOK = pg_sys::post_parse_analyze_hook;
pg_sys::post_parse_analyze_hook = Some(delete_needs_where_check);

Query Analysis Engine

The extension examines the parsed query tree to detect dangerous operations:

// Query checking logic
match query.commandType {
    pg_sys::CmdType::CMD_DELETE => {
        if !query.jointree.is_null() {
            let jointree = &*query.jointree;
            if jointree.quals.is_null() {
                error!("DELETE requires a WHERE clause");
            }
        }
    }
    pg_sys::CmdType::CMD_UPDATE => {
        if !query.jointree.is_null() {
            let jointree = &*query.jointree;
            if jointree.quals.is_null() {
                error!("UPDATE requires a WHERE clause");
            }
        }
    }
    _ => {
        // Other command types are allowed
    }
}

Key Components

Hook Function (delete_needs_where_check):
- Intercepts queries via post_parse_analyze_hook
- Checks command types (DELETE/UPDATE)
- Validates presence of WHERE clauses
- Handles Common Table Expressions recursively
Query Analysis (check_query_for_where_clause):
- Examines the query’s jointree structure
- Looks for quals (qualification/WHERE conditions)
- Throws errors for unqualified modifications
Extension Functions:
- pg_where_guard_is_enabled(): Check if protection is active
- pg_where_guard_enable(): Enable protection

Installation and Setup

Prerequisites

Before installing pg_where_guard, ensure you have:

Rust toolchain (1.70+)
pgrx framework
PostgreSQL development headers
cargo-pgrx

Build and Install

# Clone the repository
git clone https://github.com/isdaniel/pg_where_guard.git
cd pg_where_guard

# Install cargo-pgrx if not already installed
cargo install cargo-pgrx

# Initialize pgrx for your PostgreSQL version
cargo pgrx init

# Install the extension
cargo pgrx install

Database Setup

-- Create the extension
CREATE EXTENSION pg_where_guard;

-- Verify installation
SELECT pg_where_guard_is_enabled();  -- Returns: true

Usage Examples

Safe Operations (Allowed)

-- Create a test table
CREATE TABLE employees (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    department TEXT,
    salary INTEGER
);

-- Insert test data
INSERT INTO employees (name, department, salary) VALUES
    ('Alice Johnson', 'Engineering', 75000),
    ('Bob Smith', 'Marketing', 65000),
    ('Charlie Brown', 'Engineering', 80000);

-- These operations work fine (have WHERE clauses)
UPDATE employees SET salary = 78000 WHERE name = 'Alice Johnson';
DELETE FROM employees WHERE department = 'Marketing';

Dangerous Operations (Blocked)

-- These commands will FAIL due to pg_where_guard protection:

-- This will fail: UPDATE without WHERE clause
UPDATE employees SET salary = 100000;
-- ERROR: UPDATE requires a WHERE clause

-- This will fail: DELETE without WHERE clause
DELETE FROM employees;
-- ERROR: DELETE requires a WHERE clause

Common Table Expression Support

The extension also protects CTEs:

-- This would also be blocked
WITH department_update AS (
    UPDATE employees SET salary = salary * 1.1  -- No WHERE clause!
    RETURNING *
)
SELECT * FROM department_update;

Performance Considerations

Minimal Overhead

pg_where_guard adds minimal performance overhead because it:

Only analyzes DELETE and UPDATE statements
Performs lightweight checks on the parsed query tree
Uses efficient Rust code with zero-cost abstractions
Operates at parse time, not execution time

Production Readiness

The extension is designed for production use with:

Comprehensive error handling
Memory-safe implementation
Minimal system resource usage
Support for PostgreSQL 12-16

Development and Testing

Project Structure

pg_where_guard/
├── Cargo.toml              # Rust project configuration
├── pg_where_guard.control  # PostgreSQL extension control file
├── src/
│   ├── lib.rs             # Main extension code
│   └── bin/
│       └── pgrx_embed.rs  # pgrx schema generation
├── sql/                   # SQL test scripts
└── tests/                 # Test files

Running Tests

# Run the test suite
cargo pgrx test

# Test with specific PostgreSQL version
cargo pgrx test pg15

Development Workflow

# Start a development PostgreSQL instance
cargo pgrx run

# Install the extension in development
cargo pgrx install --debug

Benefits of the Rust + pgrx Approach

Developer Experience

Type Safety: Compile-time guarantees prevent runtime errors
Modern Tooling: Cargo ecosystem and excellent IDE support
Testing: Built-in unit testing and integration testing
Documentation: Automatic documentation generation

Safety Guarantees

Memory Safety: No buffer overflows or memory leaks
Thread Safety: Rust’s ownership model prevents data races
Error Handling: Explicit error handling with Result types
Null Safety: No null pointer dereferences

Performance Benefits

Zero-Cost Abstractions: High-level code without runtime overhead
Optimized Compilation: LLVM backend generates efficient machine code
Minimal Dependencies: Small runtime footprint
Efficient Resource Usage: Predictable memory usage patterns

Comparison with Traditional C Extensions

Aspect	C Extension	Rust + pgrx Extension
Memory Safety	Manual management	Automatic, compile-time guaranteed
Development Speed	Slow, error-prone	Fast, safe development
Debugging	GDB, complex	Standard Rust tooling
Testing	Manual, limited	Built-in unit/integration tests
Maintenance	High overhead	Low overhead
Performance	Optimal	Near-optimal with safety

Integration Possibilities

pg_where_guard can be integrated with:

Database Migration Tools: Validate migrations before execution
ORM Frameworks: Add safety checks to generated queries
Monitoring Systems: Alert on attempted dangerous operations
Audit Systems: Log blocked operations for compliance

Conclusion

pg_where_guard demonstrates the power of modern Rust tooling for PostgreSQL extension development. By combining Rust’s safety guarantees with pgrx’s ease of use, we can build robust database tools that protect against common but dangerous operations.

The extension serves as both a practical safety tool and an example of how Rust is revolutionizing systems programming beyond traditional applications. As the PostgreSQL ecosystem continues to evolve, Rust-based extensions like pg_where_guard pave the way for safer, more maintainable database tools.

Key Takeaways

Safety First: Rust eliminates entire classes of bugs that plague C extensions
Developer Productivity: pgrx makes PostgreSQL extension development accessible
Performance: Memory safety doesn’t require sacrificing performance
Future-Proof: Rust’s growing ecosystem ensures long-term maintainability

Whether you’re looking to protect your database from accidental data loss or explore modern PostgreSQL extension development, pg_where_guard offers a compelling example of what’s possible with Rust and pgrx.

Resources

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/pg-where-guard-rust-postgresql-extension/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！

How can we use vscode debug PostgreSQL running process by source code?

2023-06-02T14:30:11.000Z

Foreword

Using gdb for command-line debugging still feels inconvenient. I initially wanted to find a simpler way to directly debug the PostgreSQL source code under Windows. After searching for a while, I found that Visual Studio (VS) was the only option available, but it is heavy and the steps are quite complex. Since most real environments run on Linux, it is better to debug the PostgreSQL source code under Linux.

How to build & install PostgreSQL from source code.

I used Ubuntu Linux environment, the first step we might need to install pre-requirement tool for PostgreSQL build.

1 2	sudo apt-get update sudo apt-get install build-essential libreadline-dev zlib1g-dev flex bison libxml2-dev libxslt-dev libssl-dev libxml2-utils xsltproc ccache libsystemd-dev -y

download PostgreSQL source code.

1
2
3

wget https://ftp.postgresql.org/pub/source/v14.8/postgresql-14.8.tar.gz
tar xvfz postgresql-14.8.tar.gz
cd postgresql-14.8

we would need to make sure the path (--prefix) exist in your system.

./configure --prefix=/home/daniel/postgresql-14.8/pgsql --with-icu --with-openssl --with-systemd --with-libxml --enable-debug

#or debug -g3 mode

./configure --prefix=/home/daniel/postgresql-14.8/pgsql --with-icu --with-openssl --with-systemd --with-libxml --enable-debug CFLAGS="-DGCC_HASCLASSVISIBILITY -O0 -Wall -W -g3 -gdwarf-2"

make -j 8
make install

we must build with --enable-debug parameter, otherwise we can’t debug with our source code.

Here are commands we would use later.

1
2
3

/home/daniel/postgresql-14.8/pgsql/bin/psql
/home/daniel/postgresql-14.8/pgsql/bin/initdb -D /home/daniel/postgresql-14.8/pgsql/data
/home/daniel/postgresql-14.8/pgsql/bin/pg_ctl -D /home/daniel/postgresql-14.8/pgsql/data -l logfile start

setup PostgreSQL environment path

I would recommend setup PostgreSQL environment path, after we build & install PostgreSQL program from source code that would assist with us easier our next steps.

Identify your shell: Determine which shell you are using by running the following command:

1	echo $SHELL

It will display the path to your current shell.

Locate the configuration file: The configuration file you need to modify depends on your shell. Here are the common ones:

Bash: ~/.bashrc or ~/.bash_profile
Zsh: ~/.zshrc or ~/.zprofile
Fish: ~/.config/fish/config.fish

Open the configuration file: Use a text editor, such as nano or vim, to open the configuration file. For example, if you are using Bash and the file is ~/.bashrc, you can

1	vim ~/.bashrc

please modify with your environment PGDATA & PATH setting align with your PostgreSQL build bin & data path.

because my build parameter used --prefix=/home/daniel/postgresql-14.8/pgsql so the setting would be align with that.

1 2	export PGDATA="/home/daniel/postgresql-14.8/pgsql/data" export PATH="/home/daniel/postgresql-14.8/pgsql/bin:$PATH"

Update the environment: To apply the changes to your current session, run the following command in your terminal:

1	source ~/.bashrc # or the appropriate file for your shell

build pem file to a create user

To create the keys, a preferred command is ssh-keygen, which is available with OpenSSH utilities.

ssh-keygen -m PEM -t rsa -b 4096

danielss@postgresql-debuger:~$ ssh-keygen -m PEM -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/danielss/.ssh/id_rsa): /home/danielss/.ssh/danielkey
Created directory '/home/danielss/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/danielss/.ssh/danielkey
Your public key has been saved in /home/danielss/.ssh/danielkey.pub

When you create an Azure VM by specifying the public key, Azure copies the public key (in the .pub format) to the ~/.ssh/authorized_keys folder on the VM. SSH keys in ~/.ssh/authorized_keys ensure that connecting clients present the corresponding private key during an SSH connection.

danielss@postgresql-debuger:~/.ssh$ ll
total 20
drwx------ 2 danielss danielss 4096 Jun  9 02:51 ./
drwxr-xr-x 4 danielss danielss 4096 Jun  9 02:52 ../
-rw-rw-r-- 1 danielss danielss  753 Jun  9 02:51 authorized_keys
-rw------- 1 danielss danielss 3247 Jun  9 02:50 danielkey
-rw-r--r-- 1 danielss danielss  753 Jun  9 02:50 danielkey.pub

copy danielkey (pem file) to your local machine, then you might login via the pem file.

permission problem on pem file

Confirm that the permissions of the .ssh directory and the authorized_keys file on the server are set correctly. Run the following commands on the server:

1 2	chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys

These commands ensure that the directory has read, write, and execute permissions only for the user, and the authorized_keys file has read and write permissions only for the user.

setup vscode

Please follow below config

Modify “program” column with PostgreSQL binary file path from json file

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "dbg postgresql",
            "type": "cppdbg",
            "request": "attach",
            "program": "/home/daniel/postgresql-14.8/pgsql/bin/postgres",
            "processId": "${command:pickProcess}",
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
          ]
        }
    ]
}

Final used ssh with pem file to login your VM via VsCode.

demo

Here is the demo for vscode debug PostgreSQL running process by sourcecode.

How can solve “ptrace:Operation not permitted” problem?

Regarding to Troubleshoot attaching to processes using GDB

Run the following command as super user: echo 0| sudo tee /proc/sys/kernel/yama/ptrace_scope

This will set the ptrace level to 0, after this just with user permissions you can attach to processes which are not launched by the debugger.

1
2
3

postgres@postgresql-debuger:~$ echo 0| sudo tee /proc/sys/kernel/yama/ptrace_scope
[sudo] password for postgres:
0

More information

https://github.com/microsoft/MIEngine/wiki/Troubleshoot-attaching-to-processes-using-GDB

https://learn.microsoft.com/en-us/azure/virtual-machines/linux/create-ssh-keys-detailed#overview-of-ssh-and-keys

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/postgresql-vscode-sourcecode-debugger/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！

Azure PostgreSQL Flexible long term backup with Managed Identity

2022-12-25T14:30:11.000Z

Introduction

This ariticle will guide us how to do long-term backup (more than 35 days) on Azure PostgreSQL Flexible.

Prerequisites:

A Postgres flexible server and an Azure VM (Linux (ubuntu 20.04)) that has access to it.
A MI (Managed Identity) in your subscription.
Please kindly make sure backup Postgres flexible server version align with pg_dump version.

Here is the sample code of Azure PostgreSQL Flexible long term backup with Managed Identity.

echo 'start program'
echo '====================='
echo 'start installing postgresql client suit'
sudo apt update
sudo apt -y install postgresql-client
echo '======================'
echo 'postgresql client install ends'
echo 'start mount storage'

wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get update
sudo apt-get install blobfuse -y
sudo apt-get install jq -y

if [ ! -f blob.conf  ]; then
    echo 'accountName <<Your blob accountName>>' >> blob.conf 
	echo 'authType MSI' >> fuse_connection.cfg
	echo 'identityObjectId <<Your blob accountKey>>' >> blob.conf 
	echo 'containerName <<Your blob Container Name>>' >> blob.conf 
fi
#create a folder which can mount to blob
mkdir ~/data
sudo blobfuse ~/data --tmp-path=/mnt/resource/mycontainer  --config-file=./blob.conf -o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120

echo '======= starting backup============'
DNAME=`date +%Y%m%d%H%M%S`
export PGPASSWORD=`curl -s 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<<MI client ID>>' -H Metadata:true | jq -r .access_token`
pg_dump --host=test-conn.postgres.database.azure.com --username='MI-Demo' -Fc -c  testdb > ~/data/dump$DNAME.sql

Experiments:

Here is the guideline for using managed identity to connect to your DB server in VM.

We would make sure the Authentication setting that was PostgreSQL and Azure Active Directory authentication.

Selected which managed identity you want to add to the PostgreSQL flexible server.

When we had added managed identity in Azure portal that will add the managed identity in the PostgreSQL server.

In below script, we need to replace <<MI client ID>> by your MI Client Id and replace by your login MI as red frames that would tell Azure PostgreSQL server verify MI user by access token instead of user password.

After we replace all content by your really connect information that I pointed out, we can try to run the script and the backup file would be back up by pg_dump without password.

Here is guideline to guide us how to mount Linux VM from blob storage by MI.

If you’re using User assigned managed identity, please add the identity in User assigned configuration of your Linux VM as shown below (choose which MI you want to use to verify.)

Please following the step “Managed Identities -> Azure role assignments -> Add role assignment (Preview)” to choose which Blob storage you want to mounted and verified by the MI.

Edit the storage account information which mark as red frame from the script which I provided to you.

AccountName: blob Account Name
AuthType: auth type must be MSI
IdentityObjectId: Filled with your managed identity object it as below red arrow.
ContainerName: blob Container Name

After the execution, you could see the dump file has successfully uploaded to storage account.

More information

https://techcommunity.microsoft.com/t5/azure-paas-blog/mount-blob-storage-on-linux-vm-using-managed-identities-or/ba-p/1821744

https://github.com/Azure/azure-storage-fuse/blob/master/README.md#valid-authentication-setups

此文作者：Daniel Shih(石頭)
此文地址： https://isdaniel.github.io/azure-flexible-longterm-backup/
版權聲明：本博客所有文章除特別聲明外，均採用 CC BY-NC-SA 3.0 TW 許可協議。轉載請註明出處！