Skip to content

Configuring Knative cert-manager integration

Knative Serving relies on a bridging component to use cert-manager for automated certificate provisioning. If you intend to use that feature, you need to enable the Knative cert-manager integration.

Prerequisites

The following must be installed on your Knative cluster:

Warning

Make sure you have installed cert-manager. Otherwise, the Serving controller will not start up correctly.

You also need permission to read component configuration and update its authoritative source, plus the issuer/certificate resources required by the chosen example. Retain prior issuer-reference values for recovery. If GitOps manages the installation, update and synchronize that source rather than competing with live edits.

Configuration authority and destination

Identify whether Serving is Operator-managed or directly managed through YAML before changing config-certmanager. Operator users change the KnativeServing resource under spec.config; they must not directly edit the generated ConfigMap. The issuer and Certificate resources are separate cert-manager resources, not ConfigMap values to place under spec.config.

Select the intended context and component namespace for the commands below:

CONTEXT=my-cluster-context
SERVING_NAMESPACE=knative-serving
SERVING_RESOURCE=knative-serving
kubectl config get-contexts

Replace the example values and confirm the named context is the intended cluster. SERVING_RESOURCE is needed only for the Operator route. These examples address a same-cluster installation; use the multi-cluster Operator guide for remote targets.

Issuer configuration

The Knative cert-manager integration defines three references to cert-manager issuers to configure different CAs for the three Knative Serving encryption features:

  • issuerRef: issuer for external-domain certificates used for ingress.
  • clusterLocalIssuerRef: issuer for cluster-local-domain certificates used for ingress.
  • systemInternalIssuerRef: issuer for certificates for system-internal-tls certificates used by Knative internal components.

The following example uses a self-signed ClusterIssuer and the Knative cert-manager integration references that ClusterIssuer for all three configurations. As this should not be used in production (and does not support rotating the CA without downtime), you should think about which CA should be used for each use case and how trust will be distributed to the clients calling the encrypted services. For the Knative system components, Knative provides a way to specify a bundle of CAs that should be trusted (more on this below).

There is no general answer on how to structure this, here an example on how it could look like:

Feature Certificate Authority Trusted via
external-domain-tls Let's encrypt Browser clients have the Let's encrypt chain already, all the root CAs will be added in company-wide Docker base image by DevOps team.
cluster-local-domain-tls CA provided by cluster operator The CA is managed by the DevOps team and will be added in company-wide Docker base image.
system-internal-tls Self-signed Cert-Manager ClusterIssuer The CA will be populated by cert-manager. DevOps team will use trust-manager to distribute the CA to Knative system components.

Issuer selection

In general, you can refer to the cert-manager documentation. There are examples available for:

Important

Please note, that not all issuer types work for each Knative feature.

cluster-local-domain-tls needs to be able to sign certificates for cluster-local domains like myapp.<namespace>, myapp.<namespace>.svc and myapp.<namespace>.svc.cluster.local. The CA is usually outside the cluster, so verification via ACME protocol (DNS01/HTTP01) is impossible. You can use an issuer that allows the creation of these certificates (e.g., a CA issuer).

system-internal-tls needs to be able to sign specific SANs that Knative validates for. The defined set of SANs is:

  • kn-routing
  • kn-user-<namespace> ( is each namespace where Knative Services are/will be created)
  • data-plane.knative.dev

As this is also not possible via ACME protocol (DNS01/HTTP01), you need to configure an issuer that allows creating the these certificates (e.g. CA issuer).

Configuring issuers

Warning

The self-signed cluster issuer should not be used in production, please see Issuer configuration above for more information.

  1. Create and apply the following self-signed ClusterIssuer to your cluster:

    # this issuer is used by cert-manager to sign all certificates
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cluster-selfsigned-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer # this issuer is specifically for Knative, it will use the CA stored in the secret created by the Certificate below
metadata:
  name: knative-selfsigned-issuer
spec:
  ca:
    secretName: knative-selfsigned-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate # this creates a CA certificate, signed by cluster-selfsigned-issuer and stored in the secret knative-selfsigned-ca
metadata:
  name: knative-selfsigned-ca
  namespace: cert-manager #  If you want to use it as a ClusterIssuer the secret must be in the cert-manager namespace.
spec:
  secretName: knative-selfsigned-ca
  commonName: knative.dev
  usages:
    - server auth
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: cluster-selfsigned-issuer

  2. Ensure that the ClusterIssuer is ready:

    kubectl --context "$CONTEXT" get clusterissuer cluster-selfsigned-issuer -o yaml
kubectl --context "$CONTEXT" get clusterissuer knative-selfsigned-issuer -o yaml
    Result: The Status.Conditions should include Ready=True.

Set the intended issuer references through the selected configuration authority. The example below retains the self-signed issuer for all three features; change only references appropriate to your intended encryption features and preserve unrelated configuration. The references are serialized YAML strings in v1.24 Serving's config-certmanager. NewCertManagerConfigFromConfigMap parses all three keys and defaults each absent reference to the knative-selfsigned-issuer ClusterIssuer. Parsing a reference does not verify that its issuer is present, Ready, or suitable for the requested domains.

Inspect both the live resource and its authoritative source manifest before choosing the entry to change:

kubectl --context "$CONTEXT" get knativeserving "$SERVING_RESOURCE" -n "$SERVING_NAMESPACE" -o yaml

Check spec.config for config-certmanager and certmanager. A full-name entry wins even if it is empty or does not contain the issuer key you want; it is not merged with the shortened entry. Do not create a shadow entry or remove the full-name entry to bypass its precedence. Use the existing winning entry, or choose one form if neither exists, according to the ConfigMap mapping reference.

Merge the intended data values from the example into that entry under spec.config, preserving their serialized string form and unrelated values. Do not put the complete ConfigMap, ClusterIssuer, or Certificate resource under spec.config.

Follow the generic Serving Operator apply/verify workflow for authoritative application, resource status, and recovery. Inspect config-certmanager as the affected resource and continue with the propagation check below. Do not substitute these values into the separate domain example or follow its config-domain guards. Do not run the direct-edit command in the next tab.

Merge the intended issuer references into your authoritative config-certmanager manifest and apply it through your existing workflow, targeting the selected context and actual component namespace.

Only for a directly managed installation with no Operator or GitOps controller, edit the ConfigMap:

kubectl --context "$CONTEXT" edit configmap config-certmanager -n "$SERVING_NAMESPACE"

Example ConfigMap values:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-certmanager
  namespace: knative-serving
  labels:
    networking.knative.dev/certificate-provider: cert-manager
data:
  issuerRef: |
    kind: ClusterIssuer
    name: knative-selfsigned-issuer
  clusterLocalIssuerRef: |
    kind: ClusterIssuer
    name: knative-selfsigned-issuer
  systemInternalIssuerRef: |
    kind: ClusterIssuer
    name: knative-selfsigned-issuer

For the Operator route, this illustrates the values to merge, not a ConfigMap to apply directly. After the chosen workflow has propagated the change, read the active configuration:

kubectl --context "$CONTEXT" get configmap config-certmanager -n "$SERVING_NAMESPACE" -o yaml

Confirm that active data contains the intended issuer references. For the Operator route, also check custom-resource status using the generic apply/verify workflow. These checks establish propagation, not certificate issuance or client trust. Return to the external-domain TLS guide to configure and verify that feature, or the corresponding encryption guide for another feature.

To recover, inspect the current winning entry again, then restore the recorded values through the same authoritative workflow and repeat the propagation check. Do not restore Operator-managed issuer settings by directly editing the generated ConfigMap.

Managing trust and rotation without downtime

As pointed out above, each client that calls a Knative Service using HTTPS needs to trust the CA and/or intermediate chain. If you take a look at the encryption overview, you can see that there are multiple places where trust needs to be distributed:

  • Cluster external client (Browser and/or other application): this is considered out of scope of Knative.
  • Cluster internal client (e.g. Knative or Vanilla K8s workload): see below.
  • Knative system components (e.g. Activator, Queue-Proxy, Ingress-Controller): see below.

Trusting for cluster internal clients (e.g. Knative or Vanilla K8s workload)

As Knative does not control all workload and the settings are highly dependent on your runtime and/or language, this is out of scope of Knative. But here a few points to consider, as there are several ways on how to provide CAs to your application:

  • Adding the CA bundle to a Container image on build-time (be aware that this complicates CA rotation, you basically need to rebuild every application)
  • Mounting a CA bundle to the filesystem (e.g. from a Secret or ConfigMap)
  • Reading it from environment variable
  • Accessing it from a Secret/ConfigMap via K8s API

If reloading certificates without downtime is important for your client, the workload must either watch changes on the K8s resource (Secret/ConfigMap) or watch the filesystem. If the workload is watching the filesystem, it is important to note that using ionotify to catch changing Secrets/ConfigMaps is not very reliable on K8s. Tests have shown that it is more reliable to regularly poll and check the certificate on the filesystem for changes.

Here are a few examples for golang:

Trusting for Knative system components

Knative system components can be configured to trust one or many CA bundles from ConfigMaps. The cluster operator needs to ensure, to configure them accordingly to avoid any downtimes during a rotation. Knative components look for a ConfigMap in the namespace where the component runs, e.g:

  • knative-serving
  • istio-system (when using net-istio)
  • kourier-system (when using net-kourier)
  • Each namespace where a Knative Service runs

Knative looks for a ConfigMap with the label networking.knative.dev/trust-bundle: "true" and will read all data keys (regardless of the name). One key can contain one or multiple CAs/Intermediates. If they are valid, they will be added to the trust store of the Knative components.

Here is an example of how ConfigMap could look like:

apiVersion: v1
data:
  cacerts.pem: |
    -----BEGIN CERTIFICATE-----
    MIIDDTCCAfWgAwIBAgIQMQuip05h7NLQq2TB+j9ZmTANBgkqhkiG9w0BAQsFADAW
    MRQwEgYDVQQDEwtrbmF0aXZlLmRldjAeFw0yMzExMjIwOTAwNDhaFw0yNDAyMjAw
    OTAwNDhaMBYxFDASBgNVBAMTC2tuYXRpdmUuZGV2MIIBIjANBgkqhkiG9w0BAQEF
    AAOCAQ8AMIIBCgKCAQEA3clC3CV7sy0TpUKNuTku6QmP9z8JUCbLCPCLACCUc1zG
    FEokqOva6TakgvAntXLkB3TEsbdCJlNm6qFbbko6DBfX6rEggqZs40x3/T+KH66u
    4PvMT3fzEtaMJDK/KQOBIvVHrKmPkvccUYK/qWY7rgBjVjjLVSJrCn4dKaEZ2JNr
    Fd0KNnaaW/dP9/FvviLqVJvHnTMHH5qyRRr1kUGTrc8njRKwpHcnUdauiDoWRKxo
    Zlyy+MhQfdbbyapX984WsDjCvrDXzkdGgbRNAf+erl6yUm6pHpQhyFFo/zndx6Uq
    QXA7jYvM2M3qCnXmaFowidoLDsDyhwoxD7WT8zur/QIDAQABo1cwVTAOBgNVHQ8B
    Af8EBAMCAgQwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDwYDVR0TAQH/BAUwAwEB/zAd
    BgNVHQ4EFgQU7p4VuECNOcnrP9ulOjc4J37Q2VUwDQYJKoZIhvcNAQELBQADggEB
    AAv26Vnk+ptQrppouF7yHV8fZbfnehpm07HIZkmnXO2vAP+MZJDNrHjy8JAVzXjt
    +OlzqAL0cRQLsUptB0btoJuw23eq8RXgJo05OLOPQ2iGNbAATQh2kLwBWd/CMg+V
    KJ4EIEpF4dmwOohsNR6xa/JoArIYH0D7gh2CwjrdGZr/tq1eMSL+uZcuX5OiE44A
    2oXF9/jsqerOcH7QUMejSnB8N7X0LmUvH4jAesQgr7jo1JTOBs7GF6wb+U76NzFa
    8ms2iAWhoplQ+EHR52wffWb0k6trXspq4O6v/J+nq9Ky3vC36so+G1ZFkMhCdTVJ
    ZmrBsSMWeT2l07qeei2UFRU=
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  labels:
    networking.knative.dev/trust-bundle: "true"
  name: knative-bundle
  namespace: knative-serving

Using trust-manager to distribute the bundle

As it can be a cumbersome task to distribute the CA bundle to all the namespaces, you can use trust-manager to automatically distribute the CA bundles. Please refer to their documentation for more information on how to do this.

Trust during rotation

During a rotation of a CA and/or Intermediate certificates your clients will need to trust the old and the new CA/chain until the rotation is done. Using the trust approach from above, you can do a full chain rotation without downtime:

  1. Make sure your existing setup is up and running.
  2. Make sure all Knative Services have the relevant certificates and are not expired.
  3. Make sure your CA (and full chain) is not expired.
  4. Add the existing and the new CA (and the full chain) to the trust bundle (either manually or via trust-manager).
  5. Reconfigure your cert-manager ClusterIssuers or Issuers to use the new CA.
  6. Wait until all certificates are expired and are renewed by cert-manager.
  7. All certificates are now signed by the new CA.
  8. Add some grace period to make sure all components did pick up all the changes.
  9. Remove the old CA from the trust bundle.

We use analytics and cookies to understand site traffic. Information about your use of our site is shared with Google for that purpose. Learn more.