This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Setup

Setting up your environment for Capsule

1 - Installation

Installing the Capsule Controller

Requirements

  • Helm 3 is required when installing the Capsule Operator chart. Follow Helm’s official documentation for installing Helm on your operating system.
  • A Kubernetes cluster (v1.16+) with the following Admission Controllers enabled:
    • PodNodeSelector
    • LimitRanger
    • ResourceQuota
    • MutatingAdmissionWebhook
    • ValidatingAdmissionWebhook
  • A Kubeconfig file accessing the Kubernetes cluster with cluster admin permissions.
  • Cert-Manager is required by default but can be disabled. It is used to manage the TLS certificates for the Capsule Admission Webhooks.

Installation

We officially only support the installation of Capsule using the Helm chart. The chart itself handles the installation/upgrade of the required CustomResourceDefinitions. The following Artifact Hub repositories are official:

Perform the following steps to install the Capsule operator:

  1. Add repository:

     helm repo add projectcapsule https://projectcapsule.github.io/charts
    
  2. Install Capsule:

     helm install capsule projectcapsule/capsule --version 0.15.2 -n capsule-system --create-namespace
    

    or (OCI)

     helm install capsule oci://ghcr.io/projectcapsule/charts/capsule --version 0.15.2 -n capsule-system --create-namespace
    
  3. Show the status:

     helm status capsule -n capsule-system
    
  4. Upgrade the Chart

     helm upgrade capsule projectcapsule/capsule -n capsule-system
    

    or (OCI)

     helm upgrade capsule oci://ghcr.io/projectcapsule/charts/capsule --version 0.15.2
    
  5. Uninstall the Chart

     helm uninstall capsule -n capsule-system
    

Production

Here are some key considerations to keep in mind when installing Capsule. Also check out the Best Practices for more information.

Scalability

For large clusters you might need to consider adjusting values for the Capsule controller.

QPS/Burst

In order to handle a large number of tenants and resources, you may need to increase the QPS and Burst values for the Capsule controller. This avoids the controller being throttled by the Kubernetes API server (Client Rate limited). You can set the following values in the Helm chart:

manager:
  options:
    clientConnectionQPS: 400
    clientConnectionBurst: 200

Workers

Define the number of workers for the Capsule controller, which translates into the number of concurrent reconciles:

manager:
  options:
    workers: 4

Cache Synchronisation

The more resources you have in your cluster, the longer it will take for the Capsule controller to sync its cache. You can adjust the cache sync period to a higher value to reduce the load on the API server:

manager:
  options:
    cacheSyncTimeout: "10m"

Leader Election Timeout

In high pressure environments leader election may fail due to the default timeout values. You can adjust the leader election timeout values to avoid this issue:

E0707 08:38:18.319041       1 leaderelection.go:452] "Error retrieving lease lock"
  err="Get \"https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/capsule-
  system/leases/42c733ea.clastix.capsule.io?timeout=5s\": net/http: request canceled
  (Client.Timeout exceeded while awaiting headers)" lock="capsule-
  system/42c733ea.clastix.capsule.io"
  I0707 08:38:18.442700       1 leaderelection.go:299] "Failed to renew lease"

Tune leader election with manager.options.leaderElection.leaseDuration, manager.options.leaderElection.renewDeadline, and manager.options.leaderElection.retryPeriod. Increasing these values makes Capsule more tolerant of slow or overloaded Kubernetes API servers; for example, raising renewDeadline also raises the leader-election client request timeout because controller-runtime uses roughly half of that value. The tradeoff is slower failover: if the active controller really dies, standby replicas will wait longer before taking leadership. Keep the ordering valid: leaseDuration should be greater than renewDeadline, and renewDeadline should be greater than retryPeriod.

manager:
  options:
    leaderElection:
      leaseDuration: "60s"
      renewDeadline: "40s"
      retryPeriod: "5s"

Worst-case leader failover is slower, around 60s, if the active pod really dies. Keep manager.options.leaderElection.leaseDuration > manager.options.leaderElection.renewDeadline > manager.options.leaderElection.retryPeriod.

API Priority and Fairness (APF)

With APF enabled, the Capsule controller will be subject to the APF configuration of the cluster. If you are running a large cluster with many tenants, you may need to adjust the APF configuration to ensure that the Capsule controller has sufficient resources to operate effectively. For more information on APF, see Kubernetes API Priority and Fairness.

We provide a built-in APF configuration for the Capsule controller, which provides API priority for all resources managed by Capsule. This configuration is applied automatically when you install Capsule. To enable the built-in APF configuration, set the following value in the Helm chart:

# Manager Options
manager:
  apiPriorityAndFairness:
    # -- Change to `true` if you want to insulate the API calls made by Capsule admission controller activities.
    # This will help ensure Capsule stability in busy clusters.
    # Ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/
    enabled: true

    # -- Only the first matching FlowSchema for a given request matters. If multiple FlowSchemas match a single inbound request, it will be assigned based on the one with the highest matchingPrecedence.
    # Ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/#flowschema
    matchingPrecedence: 900

    # -- Priority level configuration.
    # The block is directly forwarded into the priorityLevelConfiguration, so you can use whatever specification you want.
    # ref: https://kubernetes.io/docs/concepts/cluster-administration/flow-control/#prioritylevelconfiguration
    priorityLevelConfigurationSpec:
      type: Limited
      limited:
          nominalConcurrencyShares: 100
          limitResponse:
            type: Queue
            queuing:
              queues: 64
              handSize: 6
              queueLengthLimit: 100

Strict RBAC

By default, the Capsule controller runs with the ClusterRole cluster-admin, which provides full access to the cluster. This is because the controller must be able to grant RoleBindings on a per-namespace basis: Kubernetes prevents privilege escalation through RBAC, so a RoleBinding referencing a ClusterRole can only be created by a subject that either already holds all permissions of that ClusterRole or holds the bind verb on it. For production environments we recommend configuring stricter RBAC permissions for the Capsule controller by setting the following value in the Helm chart:

manager:
  rbac:
    strict: true

This replaces the cluster-admin binding with a ClusterRole that grants the controller only the permissions required for its own operation (manager.rbac.minimal is a deprecated alias for this option; both flags enable the same behavior). To satisfy the escalation prevention described above, this ClusterRole contains the narrow bind verb on the ClusterRoles listed in manager.rbac.bindableClusterRoles (default: admin) plus the ClusterRoles managed by Capsule itself (the manager.options.rbac provisioner, deleter, administrationClusterRoles and promotionClusterRoles), which are always bindable. With default values this renders as:

- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["clusterroles"]
  resourceNames: ["admin", "capsule-namespace-deleter", "capsule-namespace-provisioner"]
  verbs: ["bind"]

Tenant owner RoleBindings for the default ClusterRoles therefore work out of the box, without the controller inheriting all of admin’s permissions.

Any further ClusterRole that is assigned to Tenant owners or used for additional RoleBindings needs the same treatment. This applies only to ClusterRoles that are not managed by Capsule (see Configuration). The narrowest option is adding it to manager.rbac.bindableClusterRoles — note that the list replaces the default, so keep admin in it:

manager:
  rbac:
    strict: true
    bindableClusterRoles:
      - admin
      - prometheus-servicemonitors-viewer

Alternatively, you can aggregate the permissions of further ClusterRoles into the controller. The ClusterRole for the controller aggregates all ClusterRoles carrying one of the following labels:

  • projectcapsule.dev/aggregate-to-controller: "true"
  • projectcapsule.dev/aggregate-to-controller-instance: {{ .Release.Name }}
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: prometheus-servicemonitors-viewer
  labels:
    projectcapsule.dev/aggregate-to-controller: "true"
rules:
...

This is broader than bind — the controller then actually holds these permissions. It is required whenever the controller must exercise the permissions itself, for example when replicating resources via GlobalTenantResources without impersonation.

You can also directly grant arbitrary additional permissions via Helm values:

manager:
  rbac:
    strict: true
    clusterRole:
      extraResources:
        - apiGroups: ["storage.k8s.io"]
          resources: ["storageclasses"]
          verbs: ["get", "list", "watch", "update", "patch"]

If you are missing permissions you will see an error status for the respective tenants reflecting

kubectl get tnt
NAME    STATE    NAMESPACE QUOTA   NAMESPACE COUNT   NODE SELECTOR   READY   STATUS                                                                                                                                                                                                                                                                                                                                          AGE
green   Active                     2                                 False   cannot sync rolebindings items: rolebindings.rbac.authorization.k8s.io "capsule:managed:658936e7f2a30e35" is forbidden: user "system:serviceaccount:capsule-system:capsule" (groups=["system:serviceaccounts" "system:serviceaccounts:capsule-system" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:...   5s

Before you enable strict mode, you must implement the required permissions for your use case. Depending on which features you are using, you may need to take manual action, for example:

Admission Policies

While Capsule provides a robust framework for managing multi-tenancy in Kubernetes, it does not include built-in admission policies for enforcing specific security or operational standards for all possible aspects of a Kubernetes cluster. We provide additional policy recommendations here.

Certificate Management

By default, Capsule delegates its certificate management to cert-manager. This is the recommended way to manage the TLS certificates for Capsule. However, you can also use Capsule’s built-in TLS reconciler to manage the certificates. This is not recommended for production environments. To enable the TLS reconciler, use the following values:

certManager:
  generateCertificates: false
tls:
  enableController: true
  create: true

Webhooks

Capsule uses admission webhooks to enforce tenant boundaries and apply namespace rules. If a matching webhook has failurePolicy: Fail, requests can be rejected while Capsule is unavailable. Excluding the operations needed to recover your cluster is especially important in single-node clusters and when admission controllers depend on each other.

Many of Capsule’s namespaced resource webhooks already select tenant namespaces. Review the webhook values for the enabled hooks, including hooks for cluster-scoped resources. The following conditions provide explicit exemptions for selected system workloads and trusted controllers.

System-critical Components

Two different kinds of requests need consideration: operations on resources inside system namespaces, and operations performed by trusted system service accounts. Use both conditions below when you want to keep system workloads running and allow those controllers to manage resources elsewhere in the cluster while Capsule is unavailable. They address different requests; a service-account exemption alone does not cover every operation involved in creating and scheduling a Pod.

webhooks:
  matchConditions:
    # Skip namespaced resources in these namespaces, regardless of the caller.
    # Keep the Namespace objects themselves subject to ownership checks.
    - name: exclude-system-namespaced-resources
      expression: >-
        !has(request.namespace) ||
        !(request.namespace in ["kube-system", "kube-public", "kyverno-system"]) ||
        (request.resource.group == "" &&
         request.resource.resource == "namespaces")        

    # Skip requests from service accounts in these trusted namespaces,
    # including requests for cluster-scoped resources or other namespaces.
    - name: exclude-trusted-system-serviceaccounts
      expression: >-
        !has(request.userInfo.username) ||
        !["kube-system", "kube-public", "kyverno-system"].exists(ns,
          request.userInfo.username.startsWith("system:serviceaccount:" + ns + ":")
        )        

These values apply to all enabled Capsule mutating and validating webhooks. Choose the namespaces for your installation and update both lists. Adding another namespace only requires another list entry, rather than a separate expression for every namespace.

Kubernetes evaluates match conditions before calling the webhook. All conditions must be true for a request to match; any condition returning false skips that webhook. The two exemptions therefore work independently: a request can be skipped because of its target namespace or its authenticated caller. A skipped request does not depend on the webhook server being available.

Why the target namespace condition is needed

The first condition excludes namespaced resources such as Pods, Services, ConfigMaps, and requests to namespaced subresources such as pods/binding. It applies regardless of who submits the request. This covers a human creating a system Pod, a controller creating a replacement Pod, the scheduler binding it to a node, and a kubelet updating its status, wherever the resource and operation rules for that webhook would otherwise match.

Those requests do not necessarily come from a service account in the Pod’s namespace. For example, the scheduler commonly authenticates as system:kube-scheduler, and a human creating a Pod authenticates as that human. The Pod’s spec.serviceAccountName describes the identity its containers will use; it does not change the caller submitting the Pod or scheduling request. An exemption for service accounts in kube-system therefore cannot, by itself, keep all Pod operations in kube-system independent of Capsule.

This condition prevents the configured Capsule webhooks from blocking those namespaced operations. Kubernetes RBAC, other admission controllers, node capacity, taints, and scheduling constraints still apply. It also means Capsule’s validation and mutation are skipped for every caller acting on those namespaced resources, so reserve these exclusions for namespaces where that policy exemption is intended.

Why the service-account condition is needed

The second condition checks the authenticated API caller. Kubernetes service-account usernames have the form system:serviceaccount:<namespace>:<name>. The prefix includes the colon after the namespace, so kube-system-other does not accidentally match kube-system.

A trusted controller may create or update a Namespace, ClusterRole, or another cluster-scoped resource. Such a resource has no target namespace, even when the controller runs in kube-system. A controller can also manage resources in application namespaces. The target namespace condition keeps those requests in scope; the caller condition provides the additional exemption needed for that controller’s work to continue without Capsule.

This is a broad trust decision: every service account in a listed namespace bypasses the configured webhooks for all otherwise matching requests, including requests outside its own namespace. Only include namespaces whose service accounts and credentials are controlled by trusted administrators. Anyone able to run a Pod as one of those service accounts may obtain its identity. For tighter control, replace the prefix test with a list of exact service-account usernames. This condition does not exempt certificate identities such as system:kube-scheduler or system:kube-controller-manager; the target namespace condition already covers their namespaced operations in the selected namespaces.

If your only goal is to exclude Pods and other namespaced resources in these namespaces, the first condition is sufficient. Add the caller condition when trusted controllers also need the wider exemption for cluster-scoped resources or resources in other namespaces.

Preserve Namespace ownership checks

A Namespace object such as Namespace/kube-system is cluster-scoped. Updating it is different from updating a Pod inside kube-system. The final clause in the first condition deliberately returns true for core Namespace resources, keeping them eligible for admission. The has(...) guards handle admission requests where optional namespace or username fields are absent.

Do not add an exemption based on a Namespace object’s name. A blanket namespaceSelector exclusion is also unsuitable for the Namespace ownership webhook: Kubernetes evaluates that selector against a Namespace object’s own labels. Excluding the system Namespace objects would let callers with Kubernetes RBAC permission to update them bypass Capsule’s ownership checks. Capsule installations can rely on those checks to constrain tenant owners who have broad Namespace RBAC permissions.

No additional namespaceSelector exclusion is needed for the CEL conditions above. The intended behavior is:

RequestEffect of these conditions
An ordinary user creates a Pod in kube-systemThe target namespace condition skips the webhook; Kubernetes RBAC must still permit the operation.
The scheduler binds a Pod in kyverno-systemThe target namespace condition skips the webhook, even though the caller is not a service account from that namespace.
An ordinary tenant owner updates Namespace/kube-systemNeither exemption applies; Capsule’s Namespace ownership checks remain in scope.
A trusted service account from kube-system creates a Namespace or updates a resource in an application namespaceThe caller condition skips the webhook.
An ordinary tenant owner updates a resource in a tenant application namespaceNeither exemption applies; the normal webhook selection and admission checks apply.

Global, mutating/validating, and per-hook match conditions are combined. Existing selectors and resource rules also continue to determine which requests can reach a webhook. If you use the per-hook opts.capsuleUsers or opts.administrators settings, review their interaction with explicit conditions: Capsule uses explicitly configured matchConditions instead of generating the identity condition from those options.

Kyverno Helm values

When Kyverno also needs these exemptions, configure its chart separately under config.matchConditions. Capsule’s webhooks.matchConditions values do not configure Kyverno. The example below uses the same three namespaces and the same two conditions.

Kyverno’s chart also has default namespace exclusions. To keep Namespace objects subject to policy for ordinary callers, remove the overlapping blanket exclusions and express the intended exemptions through CEL. In the Kyverno chart values, excludeKyvernoNamespace: false disables the automatic exclusion of Kyverno’s installation namespace. Setting matchExpressions: [] replaces the default namespace selector expressions; an empty map can leave defaults in place when Helm merges values. Despite its name, resourceFiltersExcludeNamespaces removes those namespaces from the chart’s default resource filters; it does not add a new namespace exclusion.

config:
  # Remove blanket exclusions that would also skip Namespace objects.
  excludeKyvernoNamespace: false
  webhooks:
    namespaceSelector:
      matchExpressions: []
  resourceFiltersExcludeNamespaces:
    - kube-system
    - kube-public
    - kyverno-system

  matchConditions:
    # Exempt system workloads without excluding their Namespace objects.
    - name: exclude-system-namespaced-resources
      expression: >-
        !has(request.namespace) ||
        !(request.namespace in ["kube-system", "kube-public", "kyverno-system"]) ||
        (request.resource.group == "" &&
         request.resource.resource == "namespaces")        

    # Exempt trusted controllers, including their cluster-scoped operations.
    - name: exclude-trusted-system-serviceaccounts
      expression: >-
        !has(request.userInfo.username) ||
        !["kube-system", "kube-public", "kyverno-system"].exists(ns,
          request.userInfo.username.startsWith("system:serviceaccount:" + ns + ":")
        )        

The CEL conditions prevent matching requests from being sent to Kyverno. Resource filters alone are insufficient for this availability goal because they are evaluated inside Kyverno after a webhook request reaches it. Other default resource filters remain in place. Review any additional selectors, filters, and caller exclusions in your installed Kyverno version; these values do not remove unrelated exclusions or configure background policy scans.

If Capsule runs in capsule-system and Kyverno must also exempt its workloads and service accounts, add capsule-system to both CEL namespace lists and to resourceFiltersExcludeNamespaces in the Kyverno values. Apply the same trust assessment before exempting that namespace.

Nodes

The Node webhook is relevant when you use Node metadata. It is disabled by default. Enable it only when you need that enforcement:

webhooks:
  hooks:
    nodes:
      enabled: true

To allow matching Node operations to continue when the webhook cannot be reached, you can set its failure policy to Ignore. This also means those operations proceed without Capsule’s Node validation during an outage:

webhooks:
  hooks:
    nodes:
      failurePolicy: Ignore

Nodes are cluster-scoped, so the target namespace condition above does not exempt them. The global service-account condition does exempt requests made by trusted service accounts, but kubelets commonly authenticate in the system:nodes group. If you intentionally want to bypass Capsule’s Node checks for kubelets, use a caller condition on this hook:

webhooks:
  hooks:
    nodes:
      matchConditions:
        - name: exclude-kubelet-requests
          expression: >-
            !has(request.userInfo.groups) ||
            !("system:nodes" in request.userInfo.groups)            

This exempts kubelet callers from this Node webhook, regardless of which Node they target. It does not exclude a namespace, and it bypasses the Node metadata enforcement that would otherwise apply to those callers. Prefer this scoped configuration over a global exemption for all node identities.

Namespaces

The Namespace webhook enforces Capsule Tenant boundaries. It is enabled by default and should remain enabled. The global conditions above keep Namespace objects in scope for ordinary callers while exempting trusted service accounts. Exempting resources inside a system namespace must not become an exemption for an ordinary tenant owner’s updates to the Namespace object itself.

Protected

By default resources with the following values are protected by a webhook to be changed by [Capsule Users]:

webhooks:
  hooks:
    managed:
      objectSelector:
        matchExpressions:
          - key: "projectcapsule.dev/created-by"
            operator: In
            values:
            - "controller"
            - "resources"
          - key: "projectcapsule.dev/managed-by"
            operator: In
            values:
            - "controller"

GitOps

There are no specific requirements for using Capsule with GitOps tools like ArgoCD or FluxCD. You can manage Capsule resources as you would with any other Kubernetes resource.

ArgoCD

Visit the ArgoCD Integration for more options to integrate Capsule with ArgoCD.

Manifests to get you started with ArgoCD. For ArgoCD you might need to skip the validation of the CapsuleConfiguration resources, otherwise there might be errors on the first install:

manager:
  options:
    annotations:
      argocd.argoproj.io/sync-options: "Validate=false,SkipDryRunOnMissingResource=true"
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: capsule
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: system
  source:
    repoURL: ghcr.io/projectcapsule/charts
    targetRevision: 0.15.2
    chart: capsule
    helm:
      valuesObject:
        crds:
          install: true
        manager:
          options:
            annotations:
              argocd.argoproj.io/sync-options: "Validate=false,SkipDryRunOnMissingResource=true"
            capsuleConfiguration: default
            ignoreUserGroups:
              - oidc:administators
            users:
              - kind: Group
                name: oidc:kubernetes-users
              - kind: Group
                name: system:serviceaccounts:tenants-system
        monitoring:
          dashboards:
            enabled: true
          serviceMonitor:
            enabled: true
            annotations:
              argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

  destination:
    server: https://kubernetes.default.svc
    namespace: capsule-system

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - ServerSideApply=true
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - PruneLast=true
    - RespectIgnoreDifferences=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
---
apiVersion: v1
kind: Secret
metadata:
  name: capsule-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  url: ghcr.io/projectcapsule/charts
  name: capsule
  project: system
  type: helm
  enableOCI: "true"

FluxCD

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: capsule
  namespace: flux-system
spec:
  serviceAccountName: kustomize-controller
  targetNamespace: "capsule-system"
  interval: 10m
  releaseName: "capsule"
  chart:
    spec:
      chart: capsule
      version: "0.15.2"
      sourceRef:
        kind: HelmRepository
        name: capsule
      interval: 24h
  install:
    createNamespace: true
  upgrade:
    remediation:
      remediateLastFailure: true
  driftDetection:
    mode: enabled
  values:
    crds:
      install: true
    manager:
      options:
        capsuleConfiguration: default
        ignoreUserGroups:
          - oidc:administators
        users:
          - kind: Group
            name: oidc:kubernetes-users
          - kind: Group
            name: system:serviceaccounts:tenants-system
    monitoring:
      dashboards:
        enabled: true
      serviceMonitor:
        enabled: true
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: capsule
  namespace: flux-system
spec:
  type: "oci"
  interval: 12h0m0s
  url: oci://ghcr.io/projectcapsule/charts

Security

See all available Artifacts

Signature

To verify artifacts you need to have cosign installed. This guide assumes you are using v2.x of cosign. All of the signatures are created using keyless signing. You can set the environment variable COSIGN_REPOSITORY to point to this repository. For example:

# Docker Image
export COSIGN_REPOSITORY=ghcr.io/projectcapsule/capsule

# Helm Chart
export COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule

To verify the signature of the docker image, run the following command.

COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign verify ghcr.io/projectcapsule/capsule:<release_tag> \
  --certificate-identity-regexp="https://github.com/projectcapsule/capsule/.github/workflows/docker-publish.yml@refs/tags/*" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" | jq

To verify the signature of the helm image, run the following command.

COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign verify ghcr.io/projectcapsule/charts/capsule:<release_tag> \
  --certificate-identity-regexp="https://github.com/projectcapsule/capsule/.github/workflows/helm-publish.yml@refs/tags/*" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" | jq

Provenance

Capsule creates and attests to the provenance of its builds using the SLSA standard and meets the SLSA Level 3 specification. The attested provenance may be verified using the cosign tool.

Verify the provenance of the docker image.

cosign verify-attestation --type slsaprovenance \
  --certificate-identity-regexp="https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/*" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  ghcr.io/projectcapsule/capsule:0.15.2 | jq .payload -r | base64 --decode | jq
cosign verify-attestation --type slsaprovenance \
  --certificate-identity-regexp="https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/*" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  ghcr.io/projectcapsule/charts/capsule:0.15.2 | jq .payload -r | base64 --decode | jq

Software Bill of Materials (SBOM)

An SBOM (Software Bill of Materials) in CycloneDX JSON format is published for each release, including pre-releases. You can set the environment variable COSIGN_REPOSITORY to point to this repository. For example:

# Docker Image
export COSIGN_REPOSITORY=ghcr.io/projectcapsule/capsule

# Helm Chart
export COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule

To inspect the SBOM of the docker image, run the following command.

COSIGN_REPOSITORY=ghcr.io/projectcapsule/capsule cosign download sbom ghcr.io/projectcapsule/capsule:0.15.2

To inspect the SBOM of the helm image, run the following command.

COSIGN_REPOSITORY=ghcr.io/projectcapsule/charts/capsule cosign download sbom ghcr.io/projectcapsule/charts/capsule:0.15.2

Compatibility

The Kubernetes compatibility is announced for each Release. Generally we are up to date with the latest upstream Kubernetes Version. Note that the Capsule project offers support only for the latest minor version of Kubernetes. Backwards compatibility with older versions of Kubernetes and OpenShift is offered by vendors.

2 - Configuration

Understand the Capsule configuration options and how to use them.

The configuration for the capsule controller is done via its dedicated configuration Custom Resource. You can explain the configuration options and how to use them:

CapsuleConfiguration

The configuration for Capsule is done via its dedicated configuration Custom Resource. You can explain the configuration options and how to use them:

kubectl explain capsuleConfiguration.spec

administrators

These entities are automatically owners for all existing tenants. Meaning they can add namespaces to any tenant. However they must be specific by using the capsule label for interacting with namespaces. Because if that label is not defined, it’s assumed that namespace interaction was not targeted towards a tenant and will therefore be ignored by capsule. May also be handy in GitOps scenarios where certain service accounts need to be able to manage namespaces for all tenants.

Read More

manager:
  options:
    administrators:
      - kind: User
        name: admin-user

users

These entities are automatically owners for all existing tenants. Meaning they can add namespaces to any tenant. However they must be specific by using the capsule label for interacting with namespaces. Because if that label is not defined, it’s assumed that namespace interaction was not targeted towards a tenant and will therefore be ignored by capsule. May also be handy in GitOps scenarios where certain service accounts need to be able to manage namespaces for all tenants.

Read More

manager:
  options:
    users:
      - kind: User
        name: owner-user
      - kind: Group
        name: projectcapsule.dev

ignoreUserWithGroups

Define groups which when found in the request of a user will be ignored by the Capsule. This might be useful if you have one group where all the users are in, but you want to separate administrators from normal users with additional groups.

manager:
  options:
    ignoreUserWithGroups:
      - company:org:administrators

enableTLSReconciler

Toggles the TLS reconciler, the controller that is able to generate CA and certificates for the webhooks when not using an already provided CA and certificate, or when these are managed externally with Vault, or cert-manager.

tls:
  enableController: true

forceTenantPrefix

Enforces the Tenant owner, during Namespace creation, to name it using the selected Tenant name as prefix, separated by a dash. This is useful to avoid Namespace name collision in a public CaaS environment.

manager:
  options:
    forceTenantPrefix: true

nodeMetadata

Allows to set the forbidden metadata for the worker nodes that could be patched by a Tenant. This applies only if the Tenant has an active NodeSelector, and the Owner have right to patch their nodes.

manager:
  options:
    nodeMetadata:
      forbiddenLabels:
        denied:
          - "node-role.kubernetes.io/*"
        deniedRegex: ""
      forbiddenAnnotations:
        denied:
          - "node.alpha.kubernetes.io/*"
        deniedRegex: ""

Read More

overrides

Allows to set different name rather than the canonical one for the Capsule configuration objects, such as webhook secret or configurations.

protectedNamespaceRegex

Disallow creation of namespaces, whose name matches this regexp

manager:
  options:
    protectedNamespaceRegex: "^(kube|default|capsule|admin|system|com|org|local|localhost|io)$"

allowServiceAccountPromotion

ServiceAccounts within tenant namespaces can be promoted to owners of the given tenant this can be achieved by labeling the serviceaccount and then they are considered owners. This can only be done by other owners of the tenant. However ServiceAccounts which have been promoted to owner can not promote further serviceAccounts.

Read More

manager:
  options:
    allowServiceAccountPromotion: true

cacheInvalidation

The reconcile periode caches are invalidated. Invalidation is already attempted when resources change, however in certain scenarios it might be necessary to do out of order cache invalidations to ensure proper garbage collection of resources.

manager:
  options:
    cacheInvalidation: 0h30m0s

rbac

Define configurations for the RBAC which is being managed and applied by Capsule.

manager:
  options:
    rbac:
      # -- The ClusterRoles applied for Administrators
      administrationClusterRoles:
        - capsule-namespace-deleter

      # -- The ClusterRoles applied for ServiceAccounts which had owner Promotion
      promotionClusterRoles:
        - capsule-namespace-provisioner
        - capsule-namespace-deleter

      # -- Name for the ClusterRole required to grant Namespace Deletion permissions.
      deleter: capsule-namespace-deleter

      # -- Name for the ClusterRole required to grant Namespace Provision permissions.
      provisioner: capsule-namespace-provisioner

impersonation

Replications impersonate a ServiceAccount, falling back to the controller ServiceAccount when none is resolved. tenantDefaultServiceAccount is set to default for TenantResource; GlobalTenantResource has no default and requires both globalDefaultServiceAccount and globalDefaultServiceAccountNamespace. Within this configuration you can define properties such as the endpoint of the kube-apiserver and if service account promotion should be allowed for this client. By default the https://kubernetes.default.svc endpoint is used.

manager:
  options:
    impersonation:
      # Kubernetes API Endpoint to use for the operations
      endpoint: "https://capsule-proxy.capsule-system.svc:8081"

      # Toggles if TLS verification for the endpoint is performed or not
      skipTlsVerify: false

      # Key in the secret that holds the CA certificate (e.g., "ca.crt")
      caSecretKey: "ca.crt"

      # Name of the secret containing the CA certificate
      caSecretName: "capsule-proxy-tls"

      # Namespace where the CA certificate secret is located
      caSecretNamespace: "capsule-system"

      # Default ServiceAccount for global resources (GlobalTenantResource) [Cluster Scope]
      # When defined, users are required to use this ServiceAccount anywhere in the cluster
      # unless they explicitly provide their own. Once this is set, Capsule will add this ServiceAccount
      # for all GlobalTenantResources, if they don't already have a ServiceAccount defined.
      globalDefaultServiceAccount: "capsule-global-sa"

      # Namespace of the for the ServiceAccount provided by the globalDefaultServiceAccount property
      globalDefaultServiceAccountNamespace: "tenant-system"

      # Default ServiceAccount for tenant resources (TenantResource) [Namespaced Scope]
      # Applied to every TenantResource which doesn't define its own, in that resource's
      # own Namespace.
      tenantDefaultServiceAccount: "default"

admission

Configuration for the dynamic admission webhooks used by Capsule for mutating and validating requests. The settings are used from the static webhook configurations created during installation of Capsule and abstracted by the helm chart

manager:
  options:
    admission:
      mutating:
        client:
          caBundle: cert
          url: https://172.24.52.212:9443
        name: capsule-dynamic
      validating:
        client:
          caBundle: cert
          url: https://172.24.52.212:9443
        name: capsule-dynamic

Controller Options

Depending on the version of the Capsule Controller, the configuration options may vary. You can view the options for the latest version of the Capsule Controller or by executing the controller locally:

$ go run ./cmd/controller/ --zap-log-level 7 -h
      --cache-sync-timeout duration       The timeout used when waiting for controller cache synchronization. If unset or 0, the controller-runtime default is used.
      --client-connection-burst int32     Burst to use for interacting with kubernetes apiserver. (default 30)
      --client-connection-qps float32     QPS to use for interacting with kubernetes apiserver. (default 20)
      --configuration-name string         The CapsuleConfiguration resource name to use (default "default")
      --enable-http2                      If set, HTTP/2 will be enabled for the metrics and webhook servers
      --enable-leader-election            Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
      --enable-pprof                      Enables Pprof endpoint for profiling (not recommend in production)
      --metrics-addr string               The address the metric endpoint binds to. (default ":8080")
      --metrics-cert-key string           The name of the metrics server key file. (default "tls.key")
      --metrics-cert-name string          The name of the metrics server certificate file. (default "tls.crt")
      --metrics-cert-path string          The directory that contains the metrics server certificate.
      --metrics-secure                    If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.
      --version                           Print the Capsule version and exit
      --webhook-cert-key string           The name of the webhook key file. (default "tls.key")
      --webhook-cert-name string          The name of the webhook certificate file. (default "tls.crt")
      --webhook-cert-path string          The directory that contains the webhook certificate. (default "/tmp/k8s-webhook-server/serving-certs")
      --webhook-port int                  The port the webhook server binds to. (default 9443)
      --workers int                       MaxConcurrentReconciles is the maximum number of concurrent Reconciles which can be run. (default 1)
      --zap-devel                         Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
      --zap-encoder encoder               Zap log encoding (one of 'json' or 'console')
      --zap-log-level level               Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', 'panic'or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
      --zap-stacktrace-level level        Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
      --zap-time-encoding time-encoding   Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.

Define additional options in the values.yaml when installing via Helm:

manager:
  extraArgs:
  - "--enable-leader-election=true"

3 - Authentication

Integrate Capsule with Authentication of your Kubernetes cluster

Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by userGroups option in the CapsuleConfiguration, which defaults to projectcapsule.dev.

OIDC

In the following guide, we’ll use Keycloak an Open Source Identity and Access Management server capable to authenticate users via OIDC and release JWT tokens as proof of authentication.

Configuring OIDC Server

Configure Keycloak as OIDC server:

  • Add a realm called caas, or use any existing realm instead
  • Add a group projectcapsule.dev
  • Add a user alice assigned to group projectcapsule.dev
  • Add an OIDC client called kubernetes (Public)
  • Add an OIDC client called kubernetes-auth (Confidential (Client Secret))

For the kubernetes client, create protocol mappers called groups and audience If everything is done correctly, now you should be able to authenticate in Keycloak and see user groups in JWT tokens. Use the following snippet to authenticate in Keycloak as alice user:

$ KEYCLOAK=sso.clastix.io
$ REALM=kubernetes-auth
$ OIDC_ISSUER=${KEYCLOAK}/realms/${REALM}

$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/token \
     -d grant_type=password \
     -d response_type=id_token \
     -d scope=openid \
     -d client_id=${OIDC_CLIENT_ID} \
     -d client_secret=${OIDC_CLIENT_SECRET} \
     -d username=${USERNAME} \
     -d password=${PASSWORD} | jq

The result will include an ACCESS_TOKEN, a REFRESH_TOKEN, and an ID_TOKEN. The access-token can generally be disregarded for Kubernetes. It would be used if the identity provider was managing roles and permissions for the users but that is done in Kubernetes itself with RBAC. The id-token is short lived while the refresh-token has longer expiration. The refresh-token is used to fetch a new id-token when the id-token expires.

{
   "access_token":"ACCESS_TOKEN",
   "refresh_token":"REFRESH_TOKEN",
   "id_token": "ID_TOKEN",
   "token_type":"bearer",
   "scope": "openid groups profile email"
}

To introspect the ID_TOKEN token run:

$ curl -k -s https://${OIDC_ISSUER}/protocol/openid-connect/introspect \
     -d token=${ID_TOKEN} \
     --user kubernetes-auth:${OIDC_CLIENT_SECRET} | jq

The result will be like the following:

{
  "exp": 1601323086,
  "iat": 1601322186,
  "aud": "kubernetes",
  "typ": "ID",
  "azp": "kubernetes",
  "preferred_username": "alice",
  "email_verified": false,
  "acr": "1",
  "groups": [
    "capsule.clastix.io"
  ],
  "client_id": "kubernetes",
  "username": "alice",
  "active": true
}

Configuring Kubernetes API Server

Configuring Kubernetes for OIDC Authentication requires adding several parameters to the API Server. Please, refer to the documentation for details and examples. Most likely, your kube-apiserver.yaml manifest will looks like the following:

The configuration file approach allows you to configure multiple JWT authenticators, each with a unique issuer.url and issuer.discoveryURL. The configuration file even allows you to specify CEL expressions to map claims to user attributes, and to validate claims and user information.

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
    url: https://${OIDC_ISSUER}
    audiences:
    - kubernetes
    - kubernetes-auth
    audienceMatchPolicy: MatchAny
  claimMappings:
    username:
      claim: 'email'
      prefix: ""
    groups:
      claim: 'groups'
      prefix: ""
  certificateAuthority: <PEM encoded CA certificates>

This file must be present and consistent across all kube-apiserver instances in the cluster. Add the following flag to the kube-apiserver manifest:

spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --authentication-configuration-file=/etc/kubernetes/authentication/authentication.yaml

Read More

OIDC Flags (Legacy)

spec:
  containers:
  - command:
    - kube-apiserver
    ...
    - --oidc-issuer-url=https://${OIDC_ISSUER}
    - --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
    - --oidc-client-id=kubernetes
    - --oidc-username-claim=preferred_username
    - --oidc-groups-claim=groups
    - --oidc-username-prefix=-

KinD

As reference, here is an example of a KinD configuration for OIDC Authentication, which can be useful for local testing:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
    kubeadmConfigPatches:
     - |
       kind: ClusterConfiguration
       apiServer:
           extraArgs:
             oidc-issuer-url: https://${OIDC_ISSUER}
             oidc-username-claim: preferred_username
             oidc-client-id: kubernetes
             oidc-username-prefix: "keycloak:"
             oidc-groups-claim: groups
             oidc-groups-prefix: "keycloak:"
             enable-admission-plugins: PodNodeSelector       

Configuring kubectl

There are two options to use kubectl with OIDC:

  • OIDC Authenticator
  • Use the –token option

Plugin

One way to use OIDC authentication is the use of a kubectl plugin. The Kubelogin Plugin for kubectl simplifies the process of obtaining an OIDC token and configuring kubectl to use it. Follow the link to obtain installation instructions.

kubectl oidc-login setup \
	--oidc-issuer-url=https://${OIDC_ISSUER} \
	--oidc-client-id=kubernetes-auth \
	--oidc-client-secret=${OIDC_CLIENT_SECRET}

Manual

To use the OIDC Authenticator, add an oidc user entry to your kubeconfig file:

$ kubectl config set-credentials oidc \
    --auth-provider=oidc \
    --auth-provider-arg=idp-issuer-url=https://${OIDC_ISSUER} \
    --auth-provider-arg=idp-certificate-authority=/path/to/ca.crt \
    --auth-provider-arg=client-id=kubernetes-auth \
    --auth-provider-arg=client-secret=${OIDC_CLIENT_SECRET} \
    --auth-provider-arg=refresh-token=${REFRESH_TOKEN} \
    --auth-provider-arg=id-token=${ID_TOKEN} \
    --auth-provider-arg=extra-scopes=groups

To use the --token option:

$ kubectl config set-credentials oidc --token=${ID_TOKEN}

Point the kubectl to the URL where the Kubernetes APIs Server is reachable:

$ kubectl config set-cluster mycluster \
    --server=https://kube.projectcapsule.io:6443 \
    --certificate-authority=~/.kube/ca.crt

If your APIs Server is reachable through the capsule-proxy, make sure to use the URL of the capsule-proxy.

Create a new context for the OIDC authenticated users:

$ kubectl config set-context alice-oidc@mycluster \
    --cluster=mycluster \
    --user=oidc

As user alice, you should be able to use kubectl to create some namespaces:

$ kubectl --context alice-oidc@mycluster create namespace wind-production
$ kubectl --context alice-oidc@mycluster create namespace wind-development
$ kubectl --context alice-oidc@mycluster create namespace water-marketing

Warning: once your ID_TOKEN expires, the kubectl OIDC Authenticator will attempt to refresh automatically your ID_TOKEN using the REFRESH_TOKEN. In case the OIDC uses a self signed CA certificate, make sure to specify it with the idp-certificate-authority option in your kubeconfig file, otherwise you’ll not able to refresh the tokens.

4 - OpenShift

How to install Capsule and the Capsule Proxy on OpenShift

Introduction

This guide shows how to deploy Capsule and the Capsule Proxy on OpenShift with a single Helm chart and one OpenShift toggle, so workloads run under the OpenShift SCCs. It also covers the recommended tenant settings to keep OpenShift-managed metadata protected, and optional customizations of the OpenShift Console.

Why Capsule on OpenShift

Capsule and OpenShift complement each other. Capsule brings the tenancy model, with rules and policy enforcement for tenants. OpenShift provides the native security boundaries through SecurityContextConstraint (SCC).

  • Tenancy model (Capsule): tenant owners can create as many namespaces as they need, and Capsule manages the RBAC for them. Resource quotas are spread across the whole tenant instead of being set per namespace, and rules enforce which metadata, resources and settings tenants may use.
  • Security boundaries (OpenShift): SCCs control what the workloads of a tenant are allowed to do, such as the user they run as and the privileges they can request. Tenant owners only get the SCCs you grant them, so they stay within OpenShift’s security boundaries.

The Capsule Proxy extends this model. It solves RBAC issues when listing cluster-wide resources, and it allows some operators to be installed inside a tenant: add the service account as a tenant owner and set the KUBERNETES_SERVICE_HOST environment variable of the operator deployment to the Capsule Proxy URL. The operator then behaves as if it has cluster-admin access, while remaining fully confined to the tenant.

Limitations

The following limitations are known when using OpenShift with Capsule:

  • A tenant owner cannot create a namespace/project in the OpenShift GUI. This must be done with kubectl.

  • When copying the login token from the OpenShift GUI, the server address will always point to the Kubernetes API instead of the Capsule Proxy, so the --server flag has to be adjusted manually. Browser-based oc login --web against the Capsule Proxy works without this workaround.

Prerequisites

Before starting, ensure you have:

  • OpenShift cluster with cluster-admin privileges
  • kubectl CLI configured
  • Helm 3.x installed
  • cert-manager installed (for example the cert-manager Operator for Red Hat OpenShift)

Capsule Installation

Remove the self-provisioners ClusterRoleBinding

By default, OpenShift includes a self-provisioner role and ClusterRoleBinding that allows all users to create namespaces. This is recommended to be disabled. For more information about this rolebinding, see the Red Hat documentation.

To disable the rolebinding, follow the following steps: Remove the subjects from the ClusterRoleBinding:

kubectl patch clusterrolebinding.rbac self-provisioners -p '{"subjects": null}'

Also set autoupdate to false so the ClusterRoleBinding is not reverted by OpenShift.

kubectl patch clusterrolebinding.rbac self-provisioners -p '{ "metadata": { "annotations": { "rbac.authorization.kubernetes.io/autoupdate": "false" } } }'

Extend the admin role

This example extends the default Kubernetes admin role so tenant owners gain admin privileges on all namespaces within their tenant. The extension adds:

  • The finalizers required to create/edit resources managed by Capsule
  • The SCCs that tenant owners can use (in this example, restricted-v3, restricted-v2 and nonroot-v2).
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: extend-admin-role
  labels:
    rbac.authorization.k8s.io/aggregate-to-admin: 'true'
rules:
  - verbs:
      - update
    apiGroups:
      - capsule.clastix.io
    resources:
      - '*/finalizers'
  - apiGroups:
      - security.openshift.io
    resources:
      - securitycontextconstraints
    resourceNames:
      - restricted-v3
      - restricted-v2
      - nonroot-v2
    verbs:
      - 'use'

Helm Chart Values

A single set of values configures Capsule, the OpenShift specific settings and the Capsule Proxy. The proxy is a dependency of the Capsule chart, so it is configured under the proxy key. Each part is explained in the sections below.

# values.yaml
global:
  openshift:
    enabled: true # Default: false. See OpenShift Toggle
    requiredScc: restricted-v2 # Default: restricted-v2. Recommended are restricted-v2 and restricted-v3

openshift:
  etcdEncryption:
    enabled: true # Default: false. Enable if etcd encryption is enabled. See etcd Encryption chapter

proxy:
  enabled: true # Default: false. See Capsule Proxy chapter
  options:
    extraArgs:
      - "--feature-gates=ProxyClusterScoped=true" # Default: false. See ProxyClusterScoped and OpenShift Projects
  ingress:
    enabled: true # Default: false
    annotations:
      route.openshift.io/termination: "reencrypt"
      route.openshift.io/destination-ca-certificate-secret: capsule-proxy-root-secret # Depends on release name. If something else is used then "capsule", rename this.
    hosts:
      - host: "capsule-proxy.example.com" # Set the desired domain for the Capsule Proxy.
        paths: ["/"]

Install the chart with these values:

helm upgrade --install capsule oci://ghcr.io/projectcapsule/charts/capsule --create-namespace -n capsule-system --version 0.15.2 -f values.yaml

OpenShift Toggle

The Capsule chart has a global OpenShift toggle, global.openshift.enabled, which configures everything OpenShift needs in one go. When enabled, it:

  • Omits the securityContext and podSecurityContext of the Capsule controller and all jobs, so the SCC assigns the UID, fsGroup and SELinux level.

  • Adds the openshift.io/required-scc annotation to all pods. The value is restricted-v2 by default and can be changed with global.openshift.requiredScc.

  • Sets hostUsers: false on all pods when global.openshift.requiredScc is restricted-v3. This SCC enforces user namespaces and is available since OpenShift 4.20.

The option is global, so the Capsule Proxy dependency inherits it (see Capsule Proxy).

The security contexts are only omitted when they are not configured explicitly. To keep a security context on OpenShift, set its enabled value explicitly (for example securityContext.enabled: true for the controller, or jobs.securityContext.enabled: true for the jobs).

etcd Encryption

If etcd encryption is enabled on your OpenShift cluster and you are using TenantResource or GlobalTenantResource to replicate ConfigMap or Secret resources, the Capsule replication webhook must be configured to skip requests from OpenShift’s storage version migrator. Without this exclusion, the migrator, which iterates over all Secrets and ConfigMaps to rotate the encryption key, will be blocked or cause reconciliation errors, because the webhook intercepts its requests.

Enable the exclusion with openshift.etcdEncryption.enabled in the Helm Chart Values. It adds an exclusion for the system:serviceaccount:openshift-kube-storage-version-migrator:kube-storage-version-migrator-sa service account to the replication webhook.

Note in advanced configurations: The exclusion is rendered as an explicit matchCondition. When explicit match conditions are set on a webhook, the generated capsule-user-gate is not created. If you clear webhooks.hooks.replications.matchConditions and enable webhooks.hooks.replications.opts.capsuleUsers or webhooks.hooks.replications.opts.administrators, the chart refuses to render with openshift.etcdEncryption.enabled. Set explicit match conditions for the replication webhook in that case.

Example Tenant and TenantOwners

A minimal example tenant looks like the following:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  permissions:
    matchOwners:
      - matchLabels:
          team: devops
  priorityClasses:
    allowed:
      - openshift-user-critical
  rules:
    - audience:
        - kind: Custom
          name: "CapsuleUser"
      enforce:
        action: deny
        metadata:
          - kinds:
              - "Namespace"
            labels:
              ".*openshift.io/.*":
                required: false
              "pod-security.kubernetes.io/enforce":
                required: false
            annotations:
              ".*openshift.io/.*":
                required: false
              "pod-security.kubernetes.io/enforce":
                required: false

Combined with a TenantOwner resource to grant access to the tenant:

apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
  labels:
    team: devops
  name: devops
spec:
  kind: Group
  name: "oidc:org:devops:a"

This tenant is only a starting point. For further configuration of a tenant, see the Quickstart and the Best Practices, which include a complete Tenant baseline. More information about tenants and tenant owners can be found in the chapter Tenants.

Capsule Proxy

The Capsule Proxy is installed as a dependency of the Capsule chart, under the proxy key. Because global.openshift.enabled is a global value, the proxy inherits it and no OpenShift specific proxy configuration is needed. When enabled, the proxy:

  • Omits its (pod)SecurityContexts and those of its jobs, so the configured SCC applies.
  • Gets the openshift.io/required-scc annotation and, with restricted-v3, hostUsers: false, using the same global.openshift.requiredScc value as Capsule.
  • Gets the option --public-paths=/.well-known/oauth-authorization-server added to its arguments.

The public path allows oc login --web to be used against the Capsule Proxy. The oc login --web flow retrieves the OAuth server metadata from /.well-known/oauth-authorization-server on the API endpoint before any credentials exist. Since Capsule Proxy requires authentication on every request, this lookup is rejected by default and the login aborts with unsupported protocol scheme "". Public paths solve this: requests to a public path are proxied to the upstream API server without authentication, while everything else keeps requiring one.

The proxy values in the Helm Chart Values enable the proxy and expose it through an OpenShift Route with reencrypt termination. The certificates are generated by cert-manager, which is the default.

ProxyClusterScoped and OpenShift Projects

The ProxyClusterScoped feature gate is enabled in the Helm Chart Values through proxy.options.extraArgs. It allows tenant users to list cluster-scoped resources that are granted through a GlobalProxySettings. OpenShift Projects are the equivalent of Kubernetes Namespaces, and it’s recommended to create a GlobalProxySettings for Projects so users can list those as well.

apiVersion: capsule.clastix.io/v1beta1
kind: GlobalProxySettings
metadata:
  name: openshift-projects
spec:
  rules:
  - subjects:
    - kind: Group
      name: oidc:org:devops:a
    clusterResources:
    - apiGroups:
      - "project.openshift.io"
      resources:
      - "projects"
      operations:
      - List
      selector:
        matchLabels:
            capsule.clastix.io/tenant: solar

Console Customization

The OpenShift console can be customized. For example, the capsule-proxy can be added as a shortcut on the top right application menu with the ConsoleLink CR:

apiVersion: console.openshift.io/v1
kind: ConsoleLink
metadata:
  name: capsule-proxy-consolelink
spec:
  applicationMenu:
    imageURL: 'https://github.com/projectcapsule/capsule/raw/main/assets/logo/capsule.svg'
    section: 'Capsule'
  href: 'https://capsule-proxy.example.com'
  location: ApplicationMenu
  text: 'Capsule Proxy Kubernetes API'

It’s also possible to add links specific for certain namespaces, which are shown on the Namespace/Project overview. These can also be tenant specific by adding a NamespaceSelector:

apiVersion: console.openshift.io/v1
kind: ConsoleLink
metadata:
  name: namespaced-consolelink-solar
spec:
  text: "solar Docs"
  href: "https://linktothesolardocs.com"
  location: "NamespaceDashboard"
  namespaceDashboard:
    namespaceSelector:
      matchExpressions:
        - key: capsule.clastix.io/tenant
          operator: In
          values:
            - solar

Also a custom logo can be provided, for example by adding the Capsule logo.

Add these config lines to the existing cluster CR Console.

kubectl create configmap console-capsule-logo --from-file capsule-logo.png -n openshift-config
apiVersion: operator.openshift.io/v1
kind: Console
metadata:
  name: cluster
spec:
  customization:
    customLogoFile:
      key: capsule-logo.png
      name: console-capsule-logo
    customProductName: Capsule OpenShift Cluster

5 - Rancher

How to install Capsule and the Capsule Proxy on Rancher

The integration between Rancher and Capsule, aims to provide a multi-tenant Kubernetes service to users, enabling:

  • a self-service approach
  • access to cluster-wide resources

to end-users.

Tenant users will have the ability to access Kubernetes resources through:

  • Rancher UI
  • Rancher Shell
  • Kubernetes CLI

On the other side, administrators need to manage the Kubernetes clusters through Rancher.

Rancher provides a feature called Projects to segregate resources inside a common domain. At the same time Projects doesn’t provide way to segregate Kubernetes cluster-scope resources.

Capsule as a project born for creating a framework for multi-tenant platforms, integrates with Rancher Projects enhancing the experience with Tenants.

Capsule allows tenants isolation and resources control in a declarative way, while enabling a self-service experience to tenants. With Capsule Proxy users can also access cluster-wide resources, as configured by administrators at Tenant custom resource-level.

You can read in detail how the integration works and how to configure it, in the following guides.

capsule rancher addon

Tenants and Projects

This guide explains how to setup the integration between Capsule and Rancher Projects.

It then explains how for the tenant user, the access to Kubernetes resources is transparent.

Pre-requisites

  • An authentication provider in Rancher, e.g. an OIDC identity provider
  • A Tenant Member Cluster Role in Rancher

Configure an identity provider for Kubernetes

You can follow this general guide to configure an OIDC authentication for Kubernetes.

For a Keycloak specific setup you can check this resources list.

Known issues

Keycloak new URLs without /auth makes Rancher crash

Create the Tenant Member Cluster Role

A custom Rancher Cluster Role is needed to allow Tenant users, to read cluster-scope resources and Rancher doesn’t provide e built-in Cluster Role with this tailored set of privileges.

When logged-in to the Rancher UI as administrator, from the Users & Authentication page, create a Cluster Role named Tenant Member with the following privileges:

  • get, list, watch operations over IngressClasses resources.
  • get, list, watch operations over StorageClasses resources.
  • get, list, watch operations over PriorityClasses resources.
  • get, list, watch operations over Nodes resources.
  • get, list, watch operations over RuntimeClasses resources.

Configuration (administration)

Tenant onboarding

When onboarding tenants, the administrator needs to create the following, in order to bind the Project with the Tenant:

  • In Rancher, create a Project.

  • In the target Kubernetes cluster, create a Tenant, with the following specification:

    kind: Tenant
    ...
    spec:
      namespaceOptions:
        additionalMetadata:
          annotations:
            field.cattle.io/projectId: ${CLUSTER_ID}:${PROJECT_ID}
          labels:
            field.cattle.io/projectId: ${PROJECT_ID}
    

    where $CLUSTER_ID and $PROEJCT_ID can be retrieved, assuming a valid $CLUSTER_NAME, as:

    CLUSTER_NAME=foo
    CLUSTER_ID=$(kubectl get cluster -n fleet-default ${CLUSTER_NAME} -o jsonpath='{.status.clusterName}')
    PROJECT_IDS=$(kubectl get projects -n $CLUSTER_ID -o jsonpath="{.items[*].metadata.name}")
    for project_id in $PROJECT_IDS; do echo "${project_id}"; done
    

    More on declarative Projects here.

  • In the identity provider, create a user with correct OIDC claim of the Tenant.

  • In Rancher, add the new user to the Project with the Read-only Role.

  • In Rancher, add the new user to the Cluster with the Tenant Member Cluster Role.

Create the Tenant Member Project Role

A custom Project Role is needed to allow Tenant users, with minimum set of privileges and create and delete Namespaces.

Create a Project Role named Tenant Member that inherits the privileges from the following Roles:

  • read-only
  • create-ns

Usage

When the configuration administrative tasks have been completed, the tenant users are ready to use the Kubernetes cluster transparently.

For example can create Namespaces in a self-service mode, that would be otherwise impossible with the sole use of Rancher Projects.

Namespace creation

From the tenant user perspective both CLI and the UI are valid interfaces to communicate with.

From CLI

  • Tenants kubectl-logs in to the OIDC provider
  • Tenant creates a Namespace, as a valid OIDC-discoverable user.

the Namespace is now part of both the Tenant and the Project.

As administrator, you can verify with:

kubectl get tenant ${TENANT_NAME} -o jsonpath='{.status}'
kubectl get namespace -l field.cattle.io/projectId=${PROJECT_ID}

From UI

  • Tenants logs in to Rancher, with a valid OIDC-discoverable user (in a valid Tenant group).
  • Tenant user create a valid Namespace

the Namespace is now part of both the Tenant and the Project.

As administrator, you can verify with:

kubectl get tenant ${TENANT_NAME} -o jsonpath='{.status}'
kubectl get namespace -l field.cattle.io/projectId=${PROJECT_ID}

Additional administration

Project monitoring

Before proceeding is recommended to read the official Rancher documentation about Project Monitors.

In summary, the setup is composed by a cluster-level Prometheus, Prometheus Federator via which single Project-level Prometheus federate to.

Network isolation

Before proceeding is recommended to read the official Capsule documentation about NetworkPolicy at Tenant-level`.

Network isolation and Project Monitor

As Rancher’s Project Monitor deploys the Prometheus stack in a Namespace that is not part of neither the Project nor the Tenant Namespaces, is important to apply the label selectors in the NetworkPolicy ingress rules to the Namespace created by Project Monitor.

That Project monitoring Namespace will be named as cattle-project-<PROJECT_ID>-monitoring.

For example, if the NetworkPolicy is configured to allow all ingress traffic from Namespace with label capsule.clastix.io/tenant=foo, this label is to be applied to the Project monitoring Namespace too.

Then, a NetworkPolicy can be applied at Tenant-level with Capsule GlobalTenantResources. For example it can be applied a minimal policy for the wind Tenant:

apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
  name: wind-networkpolicies
spec:
  tenantSelector:
    matchLabels:
      capsule.clastix.io/tenant: wind
  resyncPeriod: 360s
  pruningOnDelete: true
  resources:
    - namespaceSelector:
        matchLabels:
          capsule.clastix.io/tenant: wind
      rawItems:
      - apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: wind-minimal
        spec:
          podSelector: {}
          policyTypes:
            - Ingress
            - Egress
          ingress:
            # Intra-Tenant
            - from:
              - namespaceSelector:
                  matchLabels:
                    capsule.clastix.io/tenant: wind
            # Rancher Project Monitor stack
            - from:
              - namespaceSelector:
                  matchLabels:
                    role: monitoring
            # Kubernetes nodes
            - from:
              - ipBlock:
                  cidr: 192.168.1.0/24
          egress:
            # Kubernetes DNS server
            - to:
              - namespaceSelector: {}
                podSelector:
                  matchLabels:
                    k8s-app: kube-dns
                ports:
                  - port: 53
                    protocol: UDP
            # Intra-Tenant
            - to:
              - namespaceSelector:
                  matchLabels:
                    capsule.clastix.io/tenant: wind
            # Kubernetes API server
            - to:
              - ipBlock:
                  cidr: 10.43.0.1/32
                ports:
                  - port: 443

Capsule Proxy and Rancher Projects

This guide explains how to setup the integration between Capsule Proxy and Rancher Projects.

It then explains how for the tenant user, the access to Kubernetes cluster-wide resources is transparent.

Rancher Shell and Capsule

In order to integrate the Rancher Shell with Capsule it’s needed to route the Kubernetes API requests made from the shell, via Capsule Proxy.

The capsule-rancher-addon allows the integration transparently.

Install the Capsule addon

Add the Clastix Helm repository https://clastix.github.io/charts.

By updating the cache with Clastix’s Helm repository a Helm chart named capsule-rancher-addon is available.

Install keeping attention to the following Helm values:

  • proxy.caSecretKey: the Secret key that contains the CA certificate used to sign the Capsule Proxy TLS certificate (it should be"ca.crt" when Capsule Proxy has been configured with certificates generated with Cert Manager).
  • proxy.servicePort: the port configured for the Capsule Proxy Kubernetes Service (443 in this setup).
  • proxy.serviceURL: the name of the Capsule Proxy Service (by default "capsule-proxy.capsule-system.svc" hen installed in the capsule-system Namespace).

Rancher Cluster Agent

In both CLI and dashboard use cases, the Cluster Agent is responsible for the two-way communication between Rancher and the downstream cluster.

In a standard setup, the Cluster Agents communicates to the API server. In this setup it will communicate with Capsule Proxy to ensure filtering of cluster-scope resources, for Tenants.

Cluster Agents accepts as arguments:

  • KUBERNETES_SERVICE_HOST environment variable
  • KUBERNETES_SERVICE_PORT environment variable

which will be set, at cluster import-time, to the values of the Capsule Proxy Service. For example:

  • KUBERNETES_SERVICE_HOST=capsule-proxy.capsule-system.svc
  • (optional) KUBERNETES_SERVICE_PORT=9001. You can skip it by installing Capsule Proxy with Helm value service.port=443.

The expected CA is the one for which the certificate is inside the kube-root-ca ConfigMap in the same Namespace of the Cluster Agent (cattle-system).

Capsule Proxy

Capsule Proxy needs to provide a x509 certificate for which the root CA is trusted by the Cluster Agent. The goal can be achieved by, either using the Kubernetes CA to sign its certificate, or by using a dedicated root CA.

With the Kubernetes root CA

Note: this can be achieved when the Kubernetes root CA keypair is accessible. For example is likely to be possibile with on-premise setup, but not with managed Kubernetes services.

With this approach Cert Manager will sign certificates with the Kubernetes root CA for which it’s needed to be provided a Secret.

kubectl create secret tls -n capsule-system kubernetes-ca-key-pair --cert=/path/to/ca.crt --key=/path/to/ca.key

When installing Capsule Proxy with Helm chart, it’s needed to specify to generate Capsule Proxy Certificates with Cert Manager with an external ClusterIssuer:

  • certManager.externalCA.enabled=true
  • certManager.externalCA.secretName=kubernetes-ca-key-pair
  • certManager.generateCertificates=true

and disable the job for generating the certificates without Cert Manager:

  • options.generateCertificates=false

Enable tenant users access cluster resources

In order to allow tenant users to list cluster-scope resources, like Nodes, Tenants need to be configured with proper proxySettings, for example:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: wind
spec:
  owners:
  - kind: User
    name: alice
    proxySettings:
    - kind: Nodes
      operations:
      - List
[...]

Also, in order to assign or filter nodes per Tenant, it’s needed labels on node in order to be selected:

kubectl label node worker-01 capsule.clastix.io/tenant=wind

and a node selector at Tenant level:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: wind
spec:
  nodeSelector:
    capsule.clastix.io/tenant: wind
[...]

The final manifest is:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: wind
spec:
  owners:
  - kind: User
    name: alice
    proxySettings:
    - kind: Nodes
      operations:
      - List
  nodeSelector:
    capsule.clastix.io/tenant: wind

The same appplies for:

  • Nodes
  • StorageClasses
  • IngressClasses
  • PriorityClasses

More on this in the official documentation.

Configure OIDC authentication with Keycloak

Pre-requisites

  • Keycloak realm for Rancher
  • Rancher OIDC authentication provider

Keycloak realm for Rancher

These instructions is specific to a setup made with Keycloak as an OIDC identity provider.

Mappers

  • Add to userinfo Group Membership type, claim name groups
  • Add to userinfo Audience type, claim name client audience
  • Add to userinfo, full group path, Group Membership type, claim name full_group_path

More on this on the official guide.

Rancher OIDC authentication provider

Configure an OIDC authentication provider, with Client with issuer, return URLs specific to the Keycloak setup.

Use old and Rancher-standard paths with /auth subpath (see issues below).

Add custom paths, remove /auth subpath in return and issuer URLs.

Configuration

Configure Tenant users

  1. In Rancher, configure OIDC authentication with Keycloak to use with Rancher.
  2. In Keycloak, Create a Group in the rancher Realm: capsule.clastix.io.
  3. In Keycloak, Create a User in the rancher Realm, member of capsule.clastix.io Group.
  4. In the Kubernetes target cluster, update the CapsuleConfiguration by adding the "keycloakoidc_group://capsule.clastix.io" Kubernetes Group.
  5. Login to Rancher with Keycloak with the new user.
  6. In Rancher as an administrator, set the user custom role with get of Cluster.
  7. In Rancher as an administrator, add the Rancher user ID of the just-logged in user as Owner of a Tenant.
  8. (optional) configure proxySettings for the Tenant to enable tenant users to access cluster-wide resources.

6 - Managed Kubernetes

Capsule on managed Kubernetes offerings

Capsule Operator can be easily installed on a Managed Kubernetes Service. Since you do not have access to the Kubernetes APIs Server, you should check with the provider of the service:

the default cluster-admin ClusterRole is accessible the following Admission Webhooks are enabled on the APIs Server:

  • PodNodeSelector
  • LimitRanger
  • ResourceQuota
  • MutatingAdmissionWebhook
  • ValidatingAdmissionWebhook

AWS EKS

This is an example of how to install AWS EKS cluster and one user manged by Capsule. It is based on Using IAM Groups to manage Kubernetes access

Create EKS cluster:

export AWS_DEFAULT_REGION="eu-west-1"
export AWS_ACCESS_KEY_ID="xxxxx"
export AWS_SECRET_ACCESS_KEY="xxxxx"

eksctl create cluster \
--name=test-k8s \
--managed \
--node-type=t3.small \
--node-volume-size=20 \
--kubeconfig=kubeconfig.conf

Create AWS User alice using CloudFormation, create AWS access files and kubeconfig for such user:

cat > cf.yml << EOF
Parameters:
  ClusterName:
    Type: String
Resources:
  UserAlice:
    Type: AWS::IAM::User
    Properties:
      UserName: !Sub "alice-${ClusterName}"
      Policies:
      - PolicyName: !Sub "alice-${ClusterName}-policy"
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
          - Sid: AllowAssumeOrganizationAccountRole
            Effect: Allow
            Action: sts:AssumeRole
            Resource: !GetAtt RoleAlice.Arn
  AccessKeyAlice:
    Type: AWS::IAM::AccessKey
    Properties:
      UserName: !Ref UserAlice
  RoleAlice:
    Type: AWS::IAM::Role
    Properties:
      Description: !Sub "IAM role for the alice-${ClusterName} user"
      RoleName: !Sub "alice-${ClusterName}"
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
          Action: sts:AssumeRole
Outputs:
  RoleAliceArn:
    Description: The ARN of the Alice IAM Role
    Value: !GetAtt RoleAlice.Arn
    Export:
      Name:
        Fn::Sub: "${AWS::StackName}-RoleAliceArn"
  AccessKeyAlice:
    Description: The AccessKey for Alice user
    Value: !Ref AccessKeyAlice
    Export:
      Name:
        Fn::Sub: "${AWS::StackName}-AccessKeyAlice"
  SecretAccessKeyAlice:
    Description: The SecretAccessKey for Alice user
    Value: !GetAtt AccessKeyAlice.SecretAccessKey
    Export:
      Name:
        Fn::Sub: "${AWS::StackName}-SecretAccessKeyAlice"
EOF

eval aws cloudformation deploy --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=test-k8s" \
  --stack-name "test-k8s-users" --template-file cf.yml

AWS_CLOUDFORMATION_DETAILS=$(aws cloudformation describe-stacks --stack-name "test-k8s-users")
ALICE_ROLE_ARN=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"RoleAliceArn\") .OutputValue")
ALICE_USER_ACCESSKEY=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"AccessKeyAlice\") .OutputValue")
ALICE_USER_SECRETACCESSKEY=$(echo "${AWS_CLOUDFORMATION_DETAILS}" | jq -r ".Stacks[0].Outputs[] | select(.OutputKey==\"SecretAccessKeyAlice\") .OutputValue")

eksctl create iamidentitymapping --cluster="test-k8s" --arn="${ALICE_ROLE_ARN}" --username alice --group capsule.clastix.io

cat > aws_config << EOF
[profile alice]
role_arn=${ALICE_ROLE_ARN}
source_profile=alice
EOF

cat > aws_credentials << EOF
[alice]
aws_access_key_id=${ALICE_USER_ACCESSKEY}
aws_secret_access_key=${ALICE_USER_SECRETACCESSKEY}
EOF

eksctl utils write-kubeconfig --cluster=test-k8s --kubeconfig="kubeconfig-alice.conf"
cat >> kubeconfig-alice.conf << EOF
      - name: AWS_PROFILE
        value: alice
      - name: AWS_CONFIG_FILE
        value: aws_config
      - name: AWS_SHARED_CREDENTIALS_FILE
        value: aws_credentials
EOF

Export “admin” kubeconfig to be able to install Capsule:

export KUBECONFIG=kubeconfig.conf

Install Capsule and create a tenant where alice has ownership. Use the default Tenant example:

kubectl apply -f https://raw.githubusercontent.com/clastix/capsule/master/config/samples/capsule_v1beta1_tenant.yaml

Based on the tenant configuration above the user alice should be able to create namespace. Switch to a new terminal and try to create a namespace as user alice:

# Unset AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY if defined
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY
kubectl create namespace test --kubeconfig="kubeconfig-alice.conf"

Azure AKS

This reference implementation introduces the recommended starting (baseline) infrastructure architecture for implementing a multi-tenancy Azure AKS cluster using Capsule. See CoAKS.

Charmed Kubernetes

Canonical Charmed Kubernetes is a Kubernetes distribution coming with out-of-the-box tools that support deployments and operational management and make microservice development easier. Combined with Capsule, Charmed Kubernetes allows users to further reduce the operational overhead of Kubernetes setup and management.

The Charm package for Capsule is available to Charmed Kubernetes users via Charmhub.io.