How to add a PVC to Prometheus on Portworx?
KB0016124
Question
How to add a PVC to Prometheus on Portworx?
Answer
You need a recent PX Operator version with a recent CR which supports this feature, like 23.4 onwards.
You have first to validate if the StorageCluster CR supports volumes on our Prometheus monitoring.
kubectl explain stc.spec.monitoring.prometheus
You should see content about adding a volume to Prometheus:
% kubectl explain stc.spec.monitoring.prometheus
KIND: StorageCluster
VERSION: core.libopenstorage.org/v1
RESOURCE: prometheus <Object>
DESCRIPTION:
Contains configuration of Prometheus to monitor the storage cluster.
FIELDS:
alertManager <Object>
Contains configuration of AlertManager for the storage cluster.
enabled <boolean>
Flag indicating whether Prometheus stack needs to be enabled and deployed
by the Storage operator.
exportMetrics <boolean>
If this flag is enabled it will expose the storage cluster metrics to
Prometheus.
remoteWriteEndpoint <string>
Specifies the remote write endpoint for Prometheus.
replicas <integer>
Total number of non-terminated pods targeted by this Prometheus deployment
(their labels match the selector).
resources <Object>
Define resources requests and limits for single Pods.
retention <string>
Time duration Prometheus shall retain data for. Default is '24h' if
retentionSize is not set, and must match the regular expression
`[0-9]+(ms|s|m|h|d|w|y)` (milliseconds seconds minutes hours days weeks
years).
retentionSize <string>
Maximum amount of disk space used by blocks.
securityContext <Object>
Security options the pod should run with. More info:
https://kubernetes.io/docs/concepts/policy/security-context/ More info:
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
storage <Object>
Storage spec to specify how storage shall be used.
volumeMounts <[]Object>
VolumeMounts allows configuration of additional VolumeMounts on the output
StatefulSet definition. VolumeMounts specified will be appended to other
VolumeMounts in the prometheus container, that are generated as a result of
StorageSpec objects.
volumes <[]Object>
Volumes allows configuration of additional volumes on the output
StatefulSet definition. Volumes specified will be appended to other volumes
that are generated as a result of StorageSpec objects.
Once you’ve verified that we can indeed create a volume, we can change Prometheus to consume a PVC:
Change the monitoring part of the StorageCluster:
spec:
...
monitoring:
telemetry:
enabled: true
prometheus:
enabled: true
exportMetrics: true
retentionSize: "17GiB"
storage:
volumeClaimTemplate:
apiVersion: v1
spec:
storageClassName: px-csi-db
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "20Gi"
This will restart Prometheus (any previous metrics will be gone), but you will have retention onwards, until reaching the retentionSize defined.