Nicholas Dille

Talk about mastering #kubectl #Kubernetes @ #DevOps #Meetup #Freiburg

2025-12-15T14:00:00-06:00

I gave a talk at DevOps Meetup Freiburg about mastering kubectl with a focus on little known facts and tricks.

Find my slides here.

The slides are a result of my slide and demo build system. Take a look at the release for this talk.

Two day workshop about #GitLab CI (German)

2025-11-27T14:00:00-06:00

In addition to operating GitLab for our development teams, we are also using GitLab ourselves to automate the deployment and update of our services. Based on this experience I held a two day workshop for heise Academy to demonstrate the wide range of features provided by GitLab CI.

I covered the following topics:

Fundamentals
- Introduction to jobs and stages
- Using variables
- Adding before and after scripts
- Using images
- Adding defaults
- Storing and retrieving artifacts
- Job dependencies
- Adding schedules
- CI/CD configuration
- Adding unit tests
- Manually starting pipelines with forms
Advanced topics
- Using environment for deployments
- Triggering pipelines
- Child pipelines
- Job templates
- (Workflow) Rules
- Merge requests
- Matrix jobs
- Variable precedence
- GitLab feature deprecations
Expert level
- Roles and permissions in GitLab
- Understanding job tokens
- Working with Git submodules
- Adding services
- Build container images
- Using the GitLab container registry
- Creating releases
- Using branch protection
- Troubleshooting pipelines
- Using runners
- Caching intermediate results
- Renovating dependencies
- Exploring security features in GitLab community edition
- CI/CD components
- CI/CD steps
- Using pipeline inputs
- Using secure files

Find my slides here as well as the exercises.

The slides and exercises are a result of my slide and demo build system. Take a look at the release for this event.

Talk about Shell Code @ CLC Conference #CLC_Conf

2025-11-20T14:00:00-06:00

I just attended this year’s ContainerConf in Mannheim. It was awesome meeting old and new friends and talking all things containers. I had the pleasure to contribute a full-day workshop about CI/CD with GitLab as well as a talk about shell code.

Find my slides and demos for the talk here.

Workshop about #GitLab CI @ CLC Conference #CLC_Conf

2025-11-18T14:00:00-06:00

I just attended this year’s CLC Conference in Mannheim. It was awesome meeting old and new friends and talking all things containers and automation. I had the pleasure to contribute a full-day workshop about CI/CD with GitLab as well as a talk about shell code.

Find my slides and demos for the workshop here.

Workshop about operating #GitLab (German)

2025-11-13T14:00:00-06:00

Part of my daily work is operating a GitLab instance for our development teams. Based on this experience I am offering a workshop for heise Academy to share my knowledge.

I covered the following topics:

Handling the web UI
- Projects
- Users
- Authorization
- User Profiles
- Server settings
Customizing the deployment
- Using a reverse proxy for routing requests
- Understanding the directory layout
- Authenicating against LDAP
- Outgoing mail through SMTP
- Using the GitLab container registry
- Adding integrations
- Troubleshooting
- Updating GitLab
- Attaching runners for GitLab CI
- Using GitLab Pages
- Monitoring GitLab

Find my slides here.

The slides are a result of my slide and demo build system. Take a look at the release for this event.

Fun with envsubst

2025-05-25T15:02:00-05:00

When using enbsubst to substitute environment variables, empty variables will be replaced with an empty string. This may not be the desired result. Even though, envsubst supports explcitly naming the variables to substitute, it is uncomfortable to use. This post demonstrated additional options to preserve environment variables that have no value set.

Prerequisites

Let’s consider the following file:

This line contains a variable that exists: $varexists
This line contains a variable that is not set: $varnotset

Well-known solution

The well-known solution is to explicitly name the variables that should be substituted. This will only substitute the variable that exists and leave the other variable untouched:

cat file | envsubst '${varexists}'

Obfuscating the dollar sign

The first solution is to obfuscate the dollar sign in front of the variable that is not set. This will prevent envsubst from substituting the variable:

This line contains a variable that exists: $varexists
This line contains a variable that is not set: §varnotset

The following command will process the above example:

cat file | envsubst | sed 's/§/$/g'

Inserting the dollar sign

This solution uses a dummy variable to insert the dollar sign in front of variables that are not set:

This line contains a variable that exists: $varexists
This line contains a variable that is not set: ${dollar}varnotset

The following command will process the above example:

cat file | dollar='$' envsubst

Restoring the dollar sign

This solution will work on the original file without any modifications:

This line contains a variable that exists: $varexists
This line contains a variable that is not set: $varnotset

The following command will process the above example:

cat file | varnotset='$varnotset' envsubst

Collecting existing variables

The last solution is based on an imaginary tool called envsubst-helper:

Collect all variables used in the file
Check which variables are set
Output all variables that are set

cat file | envsubst "$(envsubst-helper)"

Alternative tools

There are alternative tools that can be used to substitute existing environment variables but envsubst is readily available on most systems.

Two day workshop about #GitLab CI (German)

2025-05-14T15:00:00-05:00

I covered the following topics:

Fundamentals
- Introduction to jobs and stages
- Using variables
- Adding before and after scripts
- Using images
- Adding defaults
- Storing and retrieving artifacts
- Job dependencies
- Adding schedules
- CI/CD configuration
- Adding unit tests
- Manually starting pipelines with forms
Advanced topics
- Using environment for deployments
- Triggering pipelines
- Child pipelines
- Job templates
- (Workflow) Rules
- Merge requests
- Matrix jobs
- Variable precedence
- GitLab feature deprecations
Expert level
- Roles and permissions in GitLab
- Understanding job tokens
- Working with Git submodules
- Adding services
- Build container images
- Using the GitLab container registry
- Creating releases
- Using branch protection
- Troubleshooting pipelines
- Using runners
- Caching intermediate results
- Renovating dependencies
- Exploring security features in GitLab community edition
- CI/CD components
- CI/CD steps
- Using pipeline inputs
- Using secure files

Find my slides here as well as the exercises.

The slides and exercises are a result of my slide and demo build system. Take a look at the release for this event.

Using #GitLab #OIDC to authenticate against #Kubernetes

2025-02-24T14:02:00-06:00

OpenID Connect (OIDC) and workload identity have been hot topics for a couple of years. This post demonstrates how to use GitLab as an OIDC provider to authenticate against a Kubernetes cluster - covering interactive access by users as well as automated access from pipeline jobs. The challenge is to combine both use cases in a single configuration.

Traditionally, communication between two services has been secured using shared secrets. This approach has been proven to be insecure because secrets can be leaked. OIDC is a modern approach to secure communication between services by using auto-generated short-lived tokens. The tokens are issued by an identity provider and can be verified by the service receiving the token. See the official site for an explanation of OIDC.

When a token is issued by an OIDC provider, it contains a set of claims which can be used to identify the user or service. Often these claims are mapped to roles or groups. Claims can be used by the consumer to authorize access to resources.

For example, a user authenticates against GitLab and receives a token containing claims representing the group memberships. The Kubernetes API server can be configured to accept tokens issued by GitLab and map the group memberships to Kubernetes roles using RBAC.

Other use cases include using Kubernetes services accounts to authenticate and authorize against cloud services or signing container images and artifacts using sigstore.

GitLab ships with an integrated OIDC provider which can be used to authenticate users and pipeline jobs. The issued tokens can be used to authenticate against Kubernetes and authorize access to resources using RBAC.

Part 1: Authenticating users

Following this guide, a GitLab application is configured to act as an OIDC provider. The application is configured to issue tokens containing claims representing the group memberships of the user.

When authenticating user MyUserName with ID 2 against GitLab at https://gitlab.example.com, the following represents a token issued by GitLab:

{
  "iss": "https://gitlab.example.com",
  "sub": "2",
  "aud": "REDACTED",
  "exp": 0,
  "iat": 0,
  "nonce": "REDACTED",
  "auth_time": 0,
  "sub_legacy": "REDACTED",
  "name": "MyUserName",
  "nickname": "MyUserName",
  "preferred_username": "MyUserName",
  "profile": "https://gitlab.example.com/MyUserName",
  "picture": "https://gitlab.example.com/uploads/-/system/user/avatar/2/avatar.png",
  "groups_direct": [
    "foo"
  ]
}

Note that the claim groups_direct contains the direct group memberships of the user. Based on the contents, the consumer is able to authorize access for the members of specific groups.

Part 2: Authenticating pipeline jobs

Since GitLab comes with the integrated OIDC provider, a pipeline job can easily request an ID token:

my_job:
  id_tokens:
    FIRST_ID_TOKEN:
      aud: some_string
    SECOND_ID_TOKEN:
      aud: another_string
  script: printenv

The typical use case for this feature is to authenticate against a cloud service, like AWS or Azure.

When requesting an ID token a job with ID 234 in pipeline 123 triggered by user MyUserName with ID 2 from project ` foo/bar` with ID 13, the following represents a token issued by GitLab:

{
  "namespace_id": "2",
  "namespace_path": "foo",
  "project_id": "13",
  "project_path": "foo/bar",
  "user_id": "2",
  "user_login": "MyUserName",
  "user_email": "username@example.com",
  "user_access_level": "owner",
  "pipeline_id": "123",
  "pipeline_source": "push",
  "job_id": "234",
  "ref": "main",
  "ref_type": "branch",
  "ref_path": "refs/heads/main",
  "ref_protected": "true",
  "runner_id": 1,
  "runner_environment": "self-hosted",
  "sha": "0fffdd6505a3f5f573e34338cb9eac1211bace14",
  "project_visibility": "internal",
  "ci_config_ref_uri": "gitlab.example.com/foo/bar//.gitlab-ci.yml@refs/heads/main",
  "ci_config_sha": "0fffdd6505a3f5f573e34338cb9eac1211bace14",
  "jti": "979f6254-c7ce-4632-be39-1b65163792b8",
  "iat": 0,
  "nbf": 0,
  "exp": 0,
  "iss": "https://gitlab.example.com",
  "sub": "project_path:foo/bar:ref_type:branch:ref:main",
  "aud": "REDACTED"
}

Notice that the claim sub contains a unique identifier for the project. The pipeline job is identified by the fields namespace_path, project_path, ref_type, and ref as well as pipeline_id and job_id. Based on these fields, the consumer is able to identify the project, the pipeline or even the job.

Part 3: Using Kubernetes as OIDC consumer

In order for Kubernetes to accept tokens issued by GitLab, the API server must be configured to trust the OIDC provider. The following command line options configure the API server to trust GitLab as an OIDC provider. This is also explained in the guide mentioned above.

kube-apiserver \
    --oidc-client-id=$AUDIENCE \
    --oidc-groups-claim=groups_direct \
    --oidc-groups-prefix='gitlab:' \
    --oidc-issuer-url=$ISSUER_URL \
    --oidc-username-claim=preferred_username \
    --oidc-username-prefix='oidc:'

When populating the above options note the following:

Option	Description
`--oidc-client-id`	The audience of the token issued by GitLab (matched the client or application ID)
`--oidc-groups-claim`	Referenced a list in the ID token with group memberships
`--oidc-issuer-url`	The URL of the OIDC provider (GitLab)
`--oidc-username-claim`	References a field to identify the user or the pipeline job

The options --oidc-groups-prefix and --oidc-username-prefix are used to prefix the values of the claims to make them easier to read and understand in RBAC role bindings.

Considering the two example tokens above, it becomes obvious that the API server cannot be configured to accept both token because of the difference in the fields: The primary use case for identifying a user is the list of group memberships. This field is missing from the ID token generated for a pipeline job. Pipelines are using fields that are not present in the ID token generated for a user.

Familiarize yourself with the command line options and thoroughly test them in a development cluster because syntax errors can cause the API server to fail during startup.

Fortunately, if authentication of ID token against the OIDC provider fails, the API server still recognizes and accepts all traditional ways of authentication including service account tokens.

Part 4: Using structured authentication Configuration

In Kubernetes 1.30, structured authentication configuration moved to beta and provides an alternative to the command line options. The structured configuration allows for more complex configurations and is easier to maintain. Unfortunately, it is mutually exclusive with the command line options presented above:

kube-apiserver \
    --authentication-config=/etc/kubernetes/auth/auth-config.yaml

With the structured authentication configuration it is possible to cover both use cases - authenticating a user as well as a pipeline job. Note that it only supports one item per issuer. Considering that the two use cases produce so different ID tokens, the claim mappings must be configued using the Common Expression Language (CEL).

The following structured authentication configuration uses conditionals to map the claims of the ID tokens to the fields used by Kubernetes:

---
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
    url: https://gitlab.example.com
    audiences:
    - group_application_id
    - id_token_audience
    audienceMatchPolicy: MatchAny
  claimMappings:
    username:
      expression: 'has(claims.preferred_username) ? "gitlab:" + claims.preferred_username : "gitlab-ci:" + claims.sub'
    groups:
      claim: grouops_direct
      prefix: "gitlab:"
    uid:
      expression: 'has(claims.preferred_username) ? "gitlab:" + claims.preferred_username : "gitlab-ci:" + claims.sub'

Part 5: Authorizing access using RBAC

Whenever an ID token was successfully authenticated against the OIDC provider by the API server, the claims are read from the token and Kubernetes objects are created to allow authorization:

Use Case	Claim	Resource type	Resource name
User authentication	`preferred_username`	User	`MyUserName`
User Authentication	`groups_direct`	Group	`foo`
Job authentication	`sub`	User	`project_path:foo/bar:ref_type:branch:ref:main`
Job authentication	`namespace_path`	Group	`foo`

The following ClusterRoleBindings demonstrate how to authorize the group foo to become cluster administration and the pipeline job as well as the user MyUserName to become a cluster viewer:

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: oidc-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
- kind: User
  name: gitlab-ci:project_path:foo/bar:ref_type:branch:ref:main
- kind: User
  name: gitlab:MyUserName
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: oidc-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: Group
  name: gitlab:foo

Depending on your use case, you may have to adjust the claim mapping to produce User and Group resources with the correct names.

Part 6: Authentication using `kubectl`

When using OIDC to authenticate, ID tokens have a limited lifetime requiring to refresh the token. As a consequence, a process to obtain and refresh the ID token must be implemented. For kubectl the plugin called kubelogin is responsible for this. The binary must be installed into kubectl-oidc_login and placed in the path.

The plugin comes with a subcommand to setup the OIDC consumer. It will display the ID token in the terminal for your reference and display commands to update your kubeconfig:

kubectl oidc-login setup \
  --oidc-issuer-url=ISSUER_URL \
  --oidc-client-id=YOUR_CLIENT_ID

The following excerpt from the kubeconfig demonstrates how the oidc user is configured to use the kubelogin plugin:

# REDACTED
users:
- name: oidc
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: kubectl
      args:
        - oidc-login
        - get-token
        - --oidc-issuer-url=ISSUER_URL
        - --oidc-client-id=YOUR_CLIENT_ID
# REDACTED

By referencing the user oidc in a context, kubectl will use the plugin to obtain an ID token from the OIDC provider, present it to the Kubernetes API server and execute the requested subcommand like kubectl get pods. It will also take care of refreshing the token when it expires.

Part 7: Authenticating using `id_tokens` from GitLab CI

Any pipeline job can easily request an ID token from the integrated OIDC provider using the id_tokens keyword. The audience must be set using the sub-keyword aud.

When presenting such an ID token to Kubernetes (configured as dedscribed above), the pipeline job is effectively authenticated by the Kubernetes API server and User as well as Group resources are created based on the claims contained in the token.

The following pipeline requests an ID token with the audience kubernetes and creates a kubeconfig to Kubernetes using the environment variables CONTROL_PLANE_ENDPOINT and CONTROL_PLANE_CA containing the hostname and the CA certificate of the control plane, respectively.

demo:
  id_tokens:
    K8S_TOKEN:
      aud: kubernetes
  image: alpine
  before_script: |
    apk add --update-cache \
        kubectl
  script: |
    echo -n "${CONTROL_PLANE_CA}" | base64 -d >ca.pem
    kubectl config set-cluster demo --server=${CONTROL_PLANE_ENDPOINT} --certificate-authority=./ca.pem
    kubectl config set-credentials oidc-token --token=${K8S_TOKEN}
    kubectl config set-context demo --cluster=demo --user=oidc-token --namespace=default
    kubectl config use-context demo
    kubectl get pods --all-namespaces

Based on the authentication configuration presented above, Kubernetes creates a User resource for the sub claim in the ID token which uniquely identifies the pipeline (project path and branch). The User can be used for authorizing access to the cluster. The following command creates a new ClusterRoleBinding to grant viewer permissions to a specific pipeline:

kubectl create clusterrolebinding pipeline-foo-bar-main --clusterrole view --user=gitlab-ci:project_path:foo/bar:ref_type:branch:ref:main

Happy authenticating!

Sidenote: Rolling out the Authentication Configuration File using Cluster API

The authentication configuration file can also be rolled out using Cluster API:

Add a new item under files to create /etc/kubernetes/auth/auth-config.yaml on the node
Add extraVolumes to mount the file into the kube-apiserver pods
Add authentication-config under extraArgs

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: my-cluster
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraVolumes:
        - name: auth-config
          mountPath: /etc/kubernetes/auth
          hostPath: /etc/kubernetes/auth
          pathType: Directory
          readOnly: true
        extraArgs:
          cloud-provider: external
          authentication-config: /etc/kubernetes/auth/auth-config.yaml
      controllerManager:
        extraArgs:
          cloud-provider: external
    files:
    # REDACTED
    - content: |
        apiVersion: apiserver.config.k8s.io/v1beta1
        kind: AuthenticationConfiguration
        jwt:
        - issuer:
            url: https://gitlab.example.com
            audiences:
            - group_application_id
            - id_token_audience
            audienceMatchPolicy: MatchAny
          claimMappings:
            username:
              expression: 'has(claims.preferred_username) ? "gitlab:" + claims.preferred_username : "gitlab-ci:" + claims.sub'
            groups:
              claims: groups_direct
              prefix: "gitlab:"
            uid:
              expression: 'has(claims.preferred_username) ? "gitlab:" + claims.preferred_username : "gitlab-ci:" + claims.sub'
      owner: root:root
      path: /etc/kubernetes/auth/auth-config.yaml
      permissions: "0600"
# REDACTED

My contributions @ ContainerConf 2024 #CLC_Conf #GitLab #kubernetes #RBAC

2024-11-12T16:08:00-06:00

Find my slides and demos for the workshop here.

Find my slides and demos for the talk here.

Workshop about operating #GitLab (German)

2024-11-07T14:00:00-06:00

Part of my daily work is operating a GitLab instance for our development teams. Based on this experience I held a workshop for heise Academy to share my knowledge.

I covered the following topics:

Handling the web UI
- Projects
- Users
- Authorization
- User Profiles
- Server settings
Customizing the deployment
- Using a reverse proxy for routing requests
- Understanding the directory layout
- Authenicating against LDAP
- Outgoing mail through SMTP
- Using the GitLab container registry
- Adding integrations
- Troubleshooting
- Updating GitLab
- Attaching runners for GitLab CI
- Using GitLab Pages
- Monitoring GitLab

Find my slides here.

The slides are a result of my slide and demo build system. Take a look at the release for this event.

Nicholas Dille

Talk about mastering #kubectl #Kubernetes @ #DevOps #Meetup #Freiburg

Two day workshop about #GitLab CI (German)

Talk about Shell Code @ CLC Conference #CLC_Conf

Workshop about #GitLab CI @ CLC Conference #CLC_Conf

Workshop about operating #GitLab (German)

Fun with envsubst

Prerequisites

Well-known solution

Obfuscating the dollar sign

Inserting the dollar sign

Restoring the dollar sign

Collecting existing variables

Alternative tools

Two day workshop about #GitLab CI (German)

Using #GitLab #OIDC to authenticate against #Kubernetes

Part 1: Authenticating users

Part 2: Authenticating pipeline jobs

Part 3: Using Kubernetes as OIDC consumer

Part 4: Using structured authentication Configuration

Part 5: Authorizing access using RBAC

Part 6: Authentication using kubectl

Part 7: Authenticating using id_tokens from GitLab CI

Sidenote: Rolling out the Authentication Configuration File using Cluster API

My contributions @ ContainerConf 2024 #CLC_Conf #GitLab #kubernetes #RBAC

Workshop about operating #GitLab (German)

Part 6: Authentication using `kubectl`

Part 7: Authenticating using `id_tokens` from GitLab CI