One of our larger customers had a Postgres 16.2 cluster with three async replicas.
For two weeks they were chasing a phantom 8–12 second lag on one specific replica.
The other two were flat. The network was fine, disk was fine, and
pg_stat_replication insisted everything had been sent.
The obvious metric — write_lag — was near zero. So was flush_lag.
Only replay_lag was climbing, and only during a window that moved around the
clock without lining up with backups, vacuum, or their traffic peak.
That combination narrows the field considerably. If bytes arrive and are flushed on time but are not replayed, the replica is not behind on I/O — it is blocked from applying.
Nine times out of ten this is a conflict between recovery and a long-running query on
the replica: replay wants a lock, a read holds it, and with
max_standby_streaming_delay set generously, replay simply waits.
We checked pg_stat_database_conflicts. Nothing. No cancelled queries, no
deadlock counters moving. That was the first surprise.
The replica was doing double duty as the target of a logical replication slot feeding an analytics warehouse. The slot had been created months earlier, tested, and forgotten. When the warehouse job ran, the walsender on the replica competed for the same buffers that recovery needed, and replay yielded.
The window moved around the clock because the warehouse job was scheduled relative to a job queue that drifted, not to a fixed cron. That is why it never lined up with anything on the dashboards.
replay_lag separately from write_lag — averaging them hides exactly this case.Managed clusters on Norvik now surface the three lag components separately in the project dashboard instead of a single "replication lag" number, and we warn when a replica serving reads also hosts an active logical slot. It is a small thing that would have saved this customer two weeks.