Choose the configuration authority

For an Operator-managed Serving installation, change the existing KnativeServing resource through spec.config; do not edit the generated config-observability ConfigMap. For an installation managed directly through YAML, update its authoritative ConfigMap manifest or use the labeled direct-edit route below. The monitoring stack itself is separate from the Knative Operator. If you already have a metrics backend, adapt the example to it rather than installing a second stack.

Select the cluster before using the monitoring setup. Replace these values before any Helm installation or port-forward command:

CONTEXT=my-cluster-context
SERVING_NAMESPACE=knative-serving
SERVING_RESOURCE=knative-serving
kubectl config get-contexts
kubectl --context "$CONTEXT" get services.serving.knative.dev -n "$SERVING_NAMESPACE"

Confirm the named context is the intended cluster and the read succeeds. Stop on a connection or permission error; do not continue using a different default context. SERVING_RESOURCE is the Operator resource name and is needed only for that route. Keep these variables available when using the shared setup and commands below; set the same CONTEXT in any additional terminal before port-forwarding. Cluster-targeting Helm commands must use --kube-context "$CONTEXT", and cluster-targeting kubectl commands must use --context "$CONTEXT"; do not run them against an implicit default context. This does not change your kubeconfig's defaults. Inspect the monitoring example's values and target namespaces before installing it.

These examples target a same-cluster installation; use the multi-cluster Operator guide for remote targets. You need list permission on Knative Services for the preflight, and the installation permissions required by your monitoring stack.

Collecting Metrics in Knative

Knative leverages OpenTelemetry for exporting metrics. We currently support the following export protocols:

• OTel (OTLP) over HTTP or gRPC
• Prometheus

You can also set up the OpenTelemetry Collector to receive metrics from Knative components and distribute them to other metrics providers that support OpenTelemetry.

Note

The following monitoring setup is for illustrative purposes. Support is best-effort and changes are welcome in the Knative Monitoring repository By default metrics are exporting is off.

About the Prometheus Stack

Prometheus is an open-source tool for collecting, aggregating timeseries metrics and alerting. It can also be used to scrape the OpenTelemetry Collector that is demonstrated below when Prometheus is used.

Grafana is an open-source platform for data analytics and visualization, enabling users to create customizable dashboards for monitoring and analyzing metrics from various data sources.

Prometheus Stack is a preconfigured collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules, combined to provide end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. The stack includes by default some Prometheus packages and Grafana.

Setting up the Prometheus Stack

This is optional platform setup, not a prerequisite for inspecting a monitoring provider that already serves your cluster. If one exists, use its access and query instructions and continue to the component-specific configuration below.

The component guide establishes the intended CONTEXT before this section. Confirm that it names the cluster you administer before running these commands. Review the observability namespace, Helm release name, and linked values file; do not install this demonstration stack over an existing monitoring installation.

1. Install the Prometheus Stack by using Helm:

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install knative prometheus-community/kube-prometheus-stack \
     --kube-context "$CONTEXT" \
     --create-namespace \
     --namespace observability \
     -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/promstack-values.yaml
   

Access the Prometheus instance locally

By default, the Prometheus instance is only exposed on a private service named prometheus-operated.

To access the console in your web browser:

1. Enter the command:

   kubectl --context "$CONTEXT" port-forward -n observability svc/prometheus-operated 9090:9090
   

2. Access the console in your browser via http://localhost:9090.

Access the Grafana instance locally

By default, the Grafana instance is only exposed on a private service named prometheus-grafana.

To access the dashboards in your web browser:

1. Enter the command:

   kubectl --context "$CONTEXT" port-forward -n observability svc/knative-grafana 3000:80
   

2. Access the dashboards in your browser via http://localhost:3000.

3. Use the default credentials to login:

   username: admin
   password: prom-operator
   

Before changing Serving configuration

• Identify your installation's configuration authority and retain its previous values for recovery. If GitOps manages it, update and synchronize that source instead of creating competing live changes.
• You need permission to read the component configuration and update either the KnativeServing resource or the ConfigMap, according to that authority. Installing monitor resources also requires permissions for the monitoring stack.
• The backend must support the chosen protocols. This example scrapes control-plane Prometheus endpoints and pushes request metrics through OTLP HTTP to Prometheus's enabled OTLP receiver. Verify that your backend has that receiver enabled and is reachable from the Serving workloads; configuring an endpoint alone does not establish connectivity.

Continue using the context and namespace selected before the monitoring setup.

Enabling Metric Collection

Use the following values for the example monitoring stack, or substitute your backend endpoints. Merge only the intended settings into the existing configuration; the tracing entries shown here are retained from the monitoring example, not required merely to enable metrics. Preserve your existing tracing policy if you are not changing it.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-serving
data:
  metrics-protocol: prometheus
  request-metrics-protocol: http/protobuf
  request-metrics-endpoint: http://knative-kube-prometheus-st-prometheus.observability.svc:9090/api/v1/otlp/v1/metrics
  # Optional tracing settings from the example stack:
  tracing-protocol: http/protobuf
  tracing-endpoint: http://jaeger-collector.observability.svc:4318/v1/traces
  tracing-sampling-rate: "1"

For protocol identifiers and other settings, consult the v1.24 Serving observability ConfigMap and configuration parser. In v1.24, metrics-protocol and request-metrics-protocol default to none; OTLP HTTP/gRPC require an endpoint, while Prometheus scraping does not. The metrics configuration validator defines these constraints. The YAML above is an explicit example policy, not the defaults. Check the matching release for another installation.

Operator-managed ServingYAML-managed Serving

Inspect the current custom resource and its authoritative source manifest:

kubectl --context "$CONTEXT" get knativeserving "$SERVING_RESOURCE" -n "$SERVING_NAMESPACE" -o yaml

Check spec.config for config-observability and observability before selecting an entry to change. Use the existing winning entry; do not add a shortened entry that is shadowed by a full-name entry, even if the full-name entry is empty. Do not delete an existing entry merely to bypass its precedence. Consult the ConfigMap mapping reference for the lookup rules.

Merge the intended data values from the example into that entry under spec.config, not as a complete ConfigMap resource. Preserve unrelated keys. If neither naming form exists, choose one form according to the mapping reference. After selecting the entry through the mapping reference, follow the generic Serving Operator apply/verify workflow for authoritative application, resource status, and recovery. Check config-observability as the affected resource and perform the backend checks below. Do not follow the separate domain example or its config-domain guards, and do not run the direct-edit command in the next tab.

Update the intended data keys in your authoritative config-observability manifest and apply it through your normal configuration workflow, preserving unrelated values and the actual component namespace.

For an installation not managed by either the Operator or GitOps, you can edit the ConfigMap directly:

kubectl --context "$CONTEXT" edit configmap config-observability -n "$SERVING_NAMESPACE"

Merge the intended example values into data. See Working with ConfigMaps for the distinction between active values and documentation in _example.

Apply the ServiceMonitors/PodMonitors to collect Serving control-plane metrics:

kubectl --context "$CONTEXT" apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/config/serving-monitors.yaml

These are resources for the example monitoring stack, not values to put under the Knative Operator's spec.config. Check their namespaces and selectors against your stack before applying them.

Verify collection

1. Inspect the active configuration after your chosen workflow has propagated it:

   kubectl --context "$CONTEXT" get configmap config-observability -n "$SERVING_NAMESPACE" -o yaml
   

   Confirm that active data contains your intended metric protocols and endpoint, not merely an _example description. For the Operator route, also confirm that the custom resource reports Ready through the linked generic apply/verify workflow. These checks verify configuration propagation, not receipt of metrics.

2. Open the Prometheus console using the access instructions above, or your existing backend's interface. In Prometheus's Targets view, confirm that the expected Serving control-plane targets are present and UP, and that queries return recent samples. Missing targets suggest monitor selection/discovery problems; targets that are present but down require checking reachability and scrape errors.

3. Verify request-metric delivery separately. Send requests to a known Ready Service using the deployment request checks. In your backend, select an exposed request metric for that Service/Revision and confirm fresh samples and request counts reflecting the test traffic. Use the Serving metrics reference to identify the relevant instruments and attributes; exported names depend on the backend's format. A healthy control-plane scrape does not prove that request metrics reached the OTLP receiver.

If request samples remain absent, check the receiver configuration, endpoint, and network access from the workloads before assuming collection succeeded. To recover configuration, restore the recorded values through the same authoritative workflow and repeat both propagation and backend checks; do not restore Operator-managed values by editing the generated ConfigMap.

Import Grafana dashboards

1. Grafana dashboards can be imported from the monitoring repository.

2. If you are using the Grafana Helm Chart with the dashboard sidecar enabled (the default), you can load the dashboards by applying the following ConfigMaps:

   kubectl --context "$CONTEXT" apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/config/configmap-serving-dashboard.json
   

   Confirm that the dashboards appear in Grafana and show recent data from the intended backend. An imported dashboard without data does not verify metric collection.