Federating System and User metrics to Azure Blob storage in Azure Red Hat OpenShift

By default Azure Red Hat OpenShift (ARO) stores metrics in Ephemeral volumes, and its advised that users do not change this setting. However its not unreasonable to expect that metrics should be persisted for a set amount of time.

This guide shows how to set up Thanos to federate both System and User Workload Metrics to a Thanos gateway that stores the metrics in Azure Blob Container and makes them available via a Grafana instance (managed by the Grafana Operator).

Pre-Prequsites

1. An ARO cluster

2. Set some environment variables to use throughout to suit your environment

   Note: AZR_STORAGE_ACCOUNT_NAME must be unique

Azure Preparation

1. Create an Azure storage account

   modify the arguments to suit your environment

2. Get the account key and update the secret in thanos-store-credentials.yaml

3. Create the Azure Blob container that Thanos will use to store metrics

4. Create a namespace to use

5. Add the MOBB chart repository to your Helm

6. Update your repositories

7. Use the mobb/operatorhub chart to deploy the grafana operator

8. Wait for the Operator to be ready

9. Use Helm deploy the OpenShift Patch Operator

10. Wait for the Operator to be ready

11. Deploy ARO Thanos Azure Blob container Helm Chart (mobb/aro-thanos-af)

    Note: enableUserWorkloadMetrics=true will overwrite configs for cluster and userworkload metrics. If you have customized them already, you may need to modify patch-monitoring-configs.yaml in the Helm chart to include your changes.

    If your cluster version is not 4.17, add --set "aro.clusterVersion=4.xx" to the command below.

Validate Grafana is installed and seeing metrics from Azure Blob storage

1. Get the Route URL for Grafana (remember its https) and login using username admin and the default password from the chart values (or the one you set via --set "grafana-cr.basicAuthPassword=<your-password>" during install).

2. Once logged in go to Dashboards and expand the aro-thanos-af folder and you should see the cluster metrics dashboards. Click on the Use Method / Cluster Dashboard and you should see metrics. \o/.

   Note: If it complains about a missing datasource run the following: oc annotate -n $NAMESPACE grafanadatasource aro-thanos-af-prometheus "retry=1"

   

Cleanup

1. Uninstall the aro-thanos-af chart

2. Uninstall the grafana-operator chart

3. Uninstall the patch-operator chart

4. Delete the namespaces

5. Delete the storage account