New high availability architecture for Aiven for PostgreSQL in limited availability

At Aiven, we use and contribute to open source to build our managed data platform. In the early days of Aiven for PostgreSQL, robust cloud-native tools for high availability and streaming backups did not exist at the level we needed. So we built our own.

We created pglookout to monitor replication and handle failovers, and we built pghoard for backup and restore using object storage. These tools served us and thousands of our customers well for years. But the open-source PostgreSQL ecosystem has grown. Tools that were once new are now battle-tested.

With the release of our new high availability architecture in limited availability (LA), we are replacing pglookout with the Patroni project for cluster management and replacing pghoard with pgBackRest for backup management.

High Availability with Patroni

Pglookout, like other observer tools made decisions based on external connectivity checks. In a complex network split, the observer could think a primary node is dead when it is just isolated. This could cause a split-brain situation where two nodes both think they are the primary writer.

Patroni solves this problem using a Distributed Consensus Store (DCS) to pick the leader. A node can only be the primary PostgreSQL instance if it holds the leader lock in the store. If a primary node gets isolated from the network, it loses its connection and its lock expires. Patroni instantly demotes that local instance. At the same time, the remaining connected nodes elect a new leader.

Patroni also dynamically manages your configuration files. When a standby node becomes a primary node, Patroni reconfigures the node instantly without needing extra scripts. By providing a REST API on every node, this allows the Aiven control plane to run highly reliable health checks and manual switchovers.

Backup and Recovery with pgBackRest

Our previous tool pghoard was great at shipping the Write-Ahead Logs (WAL) to object storage, but was a single-threaded design which slowed down base backups and restores.

pgBackRest uses parallel processing for backups, restores, and WAL archiving. This generally leads to faster completion times, especially for large databases. Your workloads will also benefit from native support for backups taken directly from standby instances.

We are also introducing incremental backups and you will be able to run databases with faster backup performance and a lower load on your primary instances.

Major and Minor Version Upgrades

PostgreSQL version upgrades are now much smoother. First, we upgrade the database replica. Once that is done, Aiven fails over to the replica and promotes it to a primary node. We then upgrade the old primary in the background. We build any additional primaries from the new replica.

This rolling upgrade approach limits your version upgrade downtime to just a few seconds. You can freely upgrade your database to the latest version to unlock new performance improvements and features without causing application downtime.

Summary

Adopting Partroni and pgBackRest unlocks a new architecture for Aiven for PostgreSQL. This comes with the benefits of:

• Increased Reliability: Patroni’s distributed consensus actively prevents split-brain scenarios where two database nodes think they are in charge at the same time.

• Better Backup Performance: Incremental backups to give you up to 4x better backup performance.

• Faster Maintenance: You get near-zero downtime of under 5 seconds for maintenance updates and plan scaling. You can also upgrade to the newest PostgreSQL versions without headaches using managed logical replication.

This change will make your databases more resilient to failure and faster to recover in a disaster. If you are interested in enabling this LA upgrade, reach out to us. Once enabled for a project or an organization, it becomes the default for all new Aiven for PostgreSQL services.