Using #GitLab #OIDC to authenticate against #Kubernetes

OpenID Connect (OIDC) and workload identity have been hot topics for a couple of years. This post demonstrates how to use GitLab as an OIDC provider to authenticate against a Kubernetes cluster - covering interactive access by users as well as automated access from pipeline jobs. The challenge is to combine both use cases in a single configuration.

Traditionally, communication between two services has been secured using shared secrets. This approach has been proven to be insecure because secrets can be leaked. OIDC is a modern approach to secure communication between services by using auto-generated short-lived tokens. The tokens are issued by an identity provider and can be verified by the service receiving the token. See the official site for an explanation of OIDC.

When a token is issued by an OIDC provider, it contains a set of claims which can be used to identify the user or service. Often these claims are mapped to roles or groups. Claims can be used by the consumer to authorize access to resources.

For example, a user authenticates against GitLab and receives a token containing claims representing the group memberships. The Kubernetes API server can be configured to accept tokens issued by GitLab and map the group memberships to Kubernetes roles using RBAC.

Other use cases include using Kubernetes services accounts to authenticate and authorize against cloud services or signing container images and artifacts using sigstore.

GitLab ships with an integrated OIDC provider which can be used to authenticate users and pipeline jobs. The issued tokens can be used to authenticate against Kubernetes and authorize access to resources using RBAC.

Part 1: Authenticating users

Following this guide, a GitLab application is configured to act as an OIDC provider. The application is configured to issue tokens containing claims representing the group memberships of the user.

When authenticating user MyUserName with ID 2 against GitLab at https://gitlab.example.com, the following represents a token issued by GitLab:

{
  "iss": "https://gitlab.example.com",
  "sub": "2",
  "aud": "REDACTED",
  "exp": 0,
  "iat": 0,
  "nonce": "REDACTED",
  "auth_time": 0,
  "sub_legacy": "REDACTED",
  "name": "MyUserName",
  "nickname": "MyUserName",
  "preferred_username": "MyUserName",
  "profile": "https://gitlab.example.com/MyUserName",
  "picture": "https://gitlab.example.com/uploads/-/system/user/avatar/2/avatar.png",
  "groups_direct": [
    "foo"
  ]
}

Note that the claim groups_direct contains the direct group memberships of the user. Based on the contents, the consumer is able to authorize access for the members of specific groups.

Part 2: Authenticating pipeline jobs

Since GitLab comes with the integrated OIDC provider, a pipeline job can easily request an ID token:

my_job:
  id_tokens:
    FIRST_ID_TOKEN:
      aud: some_string
    SECOND_ID_TOKEN:
      aud: another_string
  script: printenv

The typical use case for this feature is to authenticate against a cloud service, like AWS or Azure.

When requesting an ID token a job with ID 234 in pipeline 123 triggered by user MyUserName with ID 2 from project ` foo/bar` with ID 13, the following represents a token issued by GitLab:

{
  "namespace_id": "2",
  "namespace_path": "foo",
  "project_id": "13",
  "project_path": "foo/bar",
  "user_id": "2",
  "user_login": "MyUserName",
  "user_email": "username@example.com",
  "user_access_level": "owner",
  "pipeline_id": "123",
  "pipeline_source": "push",
  "job_id": "234",
  "ref": "main",
  "ref_type": "branch",
  "ref_path": "refs/heads/main",
  "ref_protected": "true",
  "runner_id": 1,
  "runner_environment": "self-hosted",
  "sha": "0fffdd6505a3f5f573e34338cb9eac1211bace14",
  "project_visibility": "internal",
  "ci_config_ref_uri": "gitlab.example.com/foo/bar//.gitlab-ci.yml@refs/heads/main",
  "ci_config_sha": "0fffdd6505a3f5f573e34338cb9eac1211bace14",
  "jti": "979f6254-c7ce-4632-be39-1b65163792b8",
  "iat": 0,
  "nbf": 0,
  "exp": 0,
  "iss": "https://gitlab.example.com",
  "sub": "project_path:foo/bar:ref_type:branch:ref:main",
  "aud": "REDACTED"
}

Notice that the claim sub contains a unique identifier for the project. The pipeline job is identified by the fields namespace_path, project_path, ref_type, and ref as well as pipeline_id and job_id. Based on these fields, the consumer is able to identify the project, the pipeline or even the job.

Part 3: Using Kubernetes as OIDC consumer

In order for Kubernetes to accept tokens issued by GitLab, the API server must be configured to trust the OIDC provider. The following command line options configure the API server to trust GitLab as an OIDC provider. This is also explained in the guide mentioned above.

kube-apiserver \
    --oidc-client-id=$AUDIENCE \
    --oidc-groups-claim=groups_direct \
    --oidc-groups-prefix='gitlab:' \
    --oidc-issuer-url=$ISSUER_URL \
    --oidc-username-claim=preferred_username \
    --oidc-username-prefix='oidc:'

When populating the above options note the following:

The options --oidc-groups-prefix and --oidc-username-prefix are used to prefix the values of the claims to make them easier to read and understand in RBAC role bindings.

Considering the two example tokens above, it becomes obvious that the API server cannot be configured to accept both token because of the difference in the fields: The primary use case for identifying a user is the list of group memberships. This field is missing from the ID token generated for a pipeline job. Pipelines are using fields that are not present in the ID token generated for a user.

Familiarize yourself with the command line options and thoroughly test them in a development cluster because syntax errors can cause the API server to fail during startup.

Fortunately, if authentication of ID token against the OIDC provider fails, the API server still recognizes and accepts all traditional ways of authentication including service account tokens.

Part 4: Using structured authentication Configuration

In Kubernetes 1.30, structured authentication configuration moved to beta and provides an alternative to the command line options. The structured configuration allows for more complex configurations and is easier to maintain. Unfortunately, it is mutually exclusive with the command line options presented above:

kube-apiserver \
    --authentication-config=/etc/kubernetes/auth/auth-config.yaml

With the structured authentication configuration it is possible to cover both use cases - authenticating a user as well as a pipeline job. Note that it only supports one item per issuer. Considering that the two use cases produce so different ID tokens, the claim mappings must be configued using the Common Expression Language (CEL).

The following structured authentication configuration uses conditionals to map the claims of the ID tokens to the fields used by Kubernetes:

---
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
    url: https://gitlab.example.com
    audiences:
    - group_application_id
    - id_token_audience
    audienceMatchPolicy: MatchAny
  claimMappings:
    username:
      expression: 'has(claims.preferred_username) ? "gitlab:" + claims.preferred_username : "gitlab-ci:" + claims.sub'
    groups:
      claim: grouops_direct
      prefix: "gitlab:"
    uid:
      expression: 'has(claims.preferred_username) ? "gitlab:" + claims.preferred_username : "gitlab-ci:" + claims.sub'

Part 5: Authorizing access using RBAC

Whenever an ID token was successfully authenticated against the OIDC provider by the API server, the claims are read from the token and Kubernetes objects are created to allow authorization:

The following ClusterRoleBindings demonstrate how to authorize the group foo to become cluster administration and the pipeline job as well as the user MyUserName to become a cluster viewer:

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: oidc-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- kind: User
  name: gitlab-ci:project_path:foo/bar:ref_type:branch:ref:main
- kind: User
  name: gitlab:MyUserName
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: oidc-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: Group
  name: gitlab:foo

Depending on your use case, you may have to adjust the claim mapping to produce User and Group resources with the correct names.

Part 6: Authentication using kubectl

When using OIDC to authenticate, ID tokens have a limited lifetime requiring to refresh the token. As a consequence, a process to obtain and refresh the ID token must be implemented. For kubectl the plugin called kubelogin is responsible for this. The binary must be installed into kubectl-oidc_login and placed in the path.

The plugin comes with a subcommand to setup the OIDC consumer. It will display the ID token in the terminal for your reference and display commands to update your kubeconfig:

kubectl oidc-login setup \
  --oidc-issuer-url=ISSUER_URL \
  --oidc-client-id=YOUR_CLIENT_ID

The following excerpt from the kubeconfig demonstrates how the oidc user is configured to use the kubelogin plugin:

# REDACTED
users:
- name: oidc
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: kubectl
      args:
        - oidc-login
        - get-token
        - --oidc-issuer-url=ISSUER_URL
        - --oidc-client-id=YOUR_CLIENT_ID
# REDACTED

By referencing the user oidc in a context, kubectl will use the plugin to obtain an ID token from the OIDC provider, present it to the Kubernetes API server and execute the requested subcommand like kubectl get pods. It will also take care of refreshing the token when it expires.

Part 7: Authenticating using id_tokens from GitLab CI

Any pipeline job can easily request an ID token from the integrated OIDC provider using the id_tokens keyword. The audience must be set using the sub-keyword aud.

When presenting such an ID token to Kubernetes (configured as dedscribed above), the pipeline job is effectively authenticated by the Kubernetes API server and User as well as Group resources are created based on the claims contained in the token.

The following pipeline requests an ID token with the audience kubernetes and creates a kubeconfig to Kubernetes using the environment variables CONTROL_PLANE_ENDPOINT and CONTROL_PLANE_CA containing the hostname and the CA certificate of the control plane, respectively.

demo:
  id_tokens:
    K8S_TOKEN:
      aud: kubernetes
  image: alpine
  before_script: |
    apk add --update-cache \
        kubectl
  script: |
    echo -n "${CONTROL_PLANE_CA}" | base64 -d >ca.pem
    kubectl config set-cluster demo --server=${CONTROL_PLANE_ENDPOINT} --certificate-authority=./ca.pem
    kubectl config set-credentials oidc-token --token=${K8S_TOKEN}
    kubectl config set-context demo --cluster=demo --user=oidc-token --namespace=default
    kubectl config use-context demo
    kubectl get pods --all-namespaces

Based on the authentication configuration presented above, Kubernetes creates a User resource for the sub claim in the ID token which uniquely identifies the pipeline (project path and branch). The User can be used for authorizing access to the cluster. The following command creates a new ClusterRoleBinding to grant viewer permissions to a specific pipeline:

kubectl create clusterrolebinding pipeline-foo-bar-main --clusterrole view --user=gitlab-ci:project_path:foo/bar:ref_type:branch:ref:main

Happy authenticating!

Sidenote: Rolling out the Authentication Configuration File using Cluster API

The authentication configuration file can also be rolled out using Cluster API:

1. Add a new item under files to create /etc/kubernetes/auth/auth-config.yaml on the node
2. Add extraVolumes to mount the file into the kube-apiserver pods
3. Add authentication-config under extraArgs

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: my-cluster
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraVolumes:
        - name: auth-config
          mountPath: /etc/kubernetes/auth
          hostPath: /etc/kubernetes/auth
          pathType: Directory
          readOnly: true
        extraArgs:
          cloud-provider: external
          authentication-config: /etc/kubernetes/auth/auth-config.yaml
      controllerManager:
        extraArgs:
          cloud-provider: external
    files:
    # REDACTED
    - content: |
        apiVersion: apiserver.config.k8s.io/v1beta1
        kind: AuthenticationConfiguration
        jwt:
        - issuer:
            url: https://gitlab.example.com
            audiences:
            - group_application_id
            - id_token_audience
            audienceMatchPolicy: MatchAny
          claimMappings:
            username:
              expression: 'has(claims.preferred_username) ? "gitlab:" + claims.preferred_username : "gitlab-ci:" + claims.sub'
            groups:
              claims: groups_direct
              prefix: "gitlab:"
            uid:
              expression: 'has(claims.preferred_username) ? "gitlab:" + claims.preferred_username : "gitlab-ci:" + claims.sub'
      owner: root:root
      path: /etc/kubernetes/auth/auth-config.yaml
      permissions: "0600"
# REDACTED