Concatenating Meta Labels for Prometheus in Kubernetes using ServiceMonitor Relabelings

In this article, we explore how to relabel the __param_target label within a ServiceMonitor configuration on your prometheus setup inside Kubernetes. This approach helps simulate interactions with services as if they were running internally (in-cluster), while also including service names and port specifics in our request path or target URL structure, without needing to hardcode the full endpoint details.

Understanding ServiceMonitor Relabelings for __param_target Formatting

To construct an appropriate ServiceMonitor, we must define a series of relabelings that combine various metadata labels into one comprehensive Prometheus target URL:

spec:
  endpoints:
    - interval: '15s'
      port: http-prometheus   # Ensure this matches the service’s HTTP protocol/port.
      path: /metrics       # Typically, monitoring agents expose metrics at their base URL endpoint (e.g., `/metrics`).
  relabelings:
    - sourceLabels: [__scheme__, __meta_kubernetes_service_name]   // Start of the target URI with scheme and service name components.
      targetLabel: instance       # The final component needed for a full Prometheus label-based path (e.g., `blackbox/myServiceName`).
    - sourceLabels: [__meta_kubernetes_namespace, __address__]   // Add the namespace and address to form `.svc.<NAME>.<NAME>`.
      targetLabel: instance       # We're building this up as our final component. 
    ... (continued relabeling rules)

The key here is that each source label must correspond accurately with a meta-label in your service specification to form the right URL path or query string for monitoring requests correctly within Prometheus. The __scheme__ and http://, while not directly used as part of this example, are essential when building URLs especially if you’re targeting an external API endpoint that requires it (common with some blackbox monitors).

Crafted ServiceMonitor Relabelings in YAML Configuration

Below is a configuration snippet for the ServiceMonitor showing how to construct and concatenate metadata labels into our desired URL form:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:  # Metadata describing service monitor's purpose, namespace etc.
... (rest of the spec)
spec:   # Specification for creating a new `ServiceMonitor`.
    endpoints:      
      - interval: '15s'     // Set how often Prometheus will probe this endpoint/service every 15 seconds.
        port: http         # Assuming we’re dealing with HTTP traffic here, set the proper protocol and service exposed on Kubernetes.  
        path: /metrics       # This is where your application's metrics are published (default).
    relabelings:     // Specify how to manipulate target labels for correct endpoint referencing in Prometheus queries/alerting rules later downstream. 
      - sourceLabels: ['__scheme__', 'http://serviceName']   # Combine with metadata namespace and port number from service's annotations or selectors (if available). Here, we assume the use of `http` protocol without SSL termination within Kubernetes services for simplicity in this example.
        targetLabel: instance         // The ultimate label that Prometheus will utilize to identify unique monitored targets uniquely named after each service and namespace combination—this forms our complete URL path/endpoint, i.e., `serviceName.namespace`.
      # Here we may need additional rules if specific port number relabeling is necessary for target ports other than the default (80 or 443). For example:  
        - sourceLabels: ['__meta_kubernetes_pod_container_port_number', 'http']    // Combine pod container's exposed servicePort with scheme and protocol. Here, we would replace `serviceName` placeholder accordingly based on the actual value of `__metas__.namespace`. 
      # Continue to expand relabelings as necessary for specific cases or configurations...  

In this configuration:

• The interval is set at every 15 seconds (a common probe frequency). You may adjust it according to your application’s needs.
• We target the /metrics endpoint typically exposed by applications designed with monitoring in mind, which includes Prometheus itself or other metrics systems compatible with http.

By integrating this service monitor into our prometheus setup within Kubernetes environment, we enable it to automatically detect and probe each named instance of services for their performance data according the crafted relabeling rules. Each monitoring request can thus be constructed in an automated fashion by Prometheus based on these labels without any additional manual configuration required from us after deployments or service changes within our cluster’s DNS namespaces, leading to more maintainable and less error-prone configurations for external blackbox monitors that rely heavily upon target URL specificity.