Auto scaling Azure Cosmos DB for PostgreSQL with Citus, Grafana, & Azure Functions

This blog post explains how to build an auto scaling setup for the Citus database extension running as a managed service on Azure - called Azure Cosmos DB for PostgreSQL, using Azure Functions, ARM templates, and Grafana. The post highlights the components required for auto scaling, which include Azure Functions, Azure Resource Manager, Grafana, and Azure Cosmos DB for PostgreSQL.

Overview of the components for Auto Scaling

The auto scaling architecture combines multiple components, including Azure Functions, Azure Resource Manager (ARM), Grafana, and Azure Cosmos DB for PostgreSQL. The blog post uses the managed version of Citus on Azure and outlines the steps for creating and populating the cluster, using Azure Functions and ARM templates to scale out the Citus cluster, configuring Grafana data sources, creating Grafana alerts, and testing the auto scaling setup.

Step 1: Creating & Populating the cluster

The first step is to create a Citus database cluster, which is used to keep track of the number of nodes in the cluster and to trigger shard rebalancing operations after adding new workers.

Step 2: Use Azure Functions & ARM templates to scale out your Citus cluster

To trigger Azure Functions that add new nodes to the Citus cluster and execute a shard rebalancing operation, functions should be created separately for each step. Azure Function Core Tools can be used to create the initial boilerplates and publish the code to the Azure cloud. After building a new ARM template containing the desired state, the ARM API is used to update the Citus cluster.

Step 3: Configuring Grafana data sources

Grafana data sources need to be configured to monitor the Citus cluster and trigger alerts if CPU usage spikes to set off the auto scaling process.

Step 4: Creating Grafana alerts

With Grafana data sources in place, alerts can be created to trigger the Azure Functions based on thresholds set for CPU usage.

Step 5: Testing the auto scaling setup

Auto scaling setup can be tested by creating CPU usage spikes and observing the cluster scale out.

Using Grafana & Citus 11.3 to monitor tenants

The blog post, along with a talk given at Citus Con, demonstrates how Grafana and Citus 11.3 can be used to monitor tenants and notice hotspots of activity across the cluster.

Key takeaways for auto scaling Azure Cosmos DB for PostgreSQL

The blog post outlines the steps for building an auto scaling setup with Azure Functions, ARM templates, Grafana, and Azure Cosmos DB for PostgreSQL. Additionally, it highlights the importance of using separate functions for adding new nodes and executing shard rebalancing.