How to Automate the Merging of Layers from #Docker Images in #PowerShell

In my last post, I explained how to create Docker images from the layers of other images. As this is a rather complex process, I have published a PowerShell module to automate this.

Table of Contents

1. How to Reduce the Build Time of a Monolithic Docker Image by merging Layers from Docker Images (published)

2. How to Automate the Merging of Layers from Docker Images in PowerShell (this post)

3. How to Fix Package Manager Databases when Merging Layers from Docker Images (coming soon)

Why Automate this?

The process of creating an image from layers of other images is rather complex. It involves the following steps:

1. Split the monolithic Dockerfile into a base image and for every section of independent commands, create a new derived image

2. Build and push the base image as well as all derived images

3. Download the image manifest and the image configuration of all images (base image and derived images)

4. Start the new image from the image manifest and the image configuration of the base image

5. Add the layers from the derived images to the image manifest of the base image

6. Add the layers and commands from the derived images to the image configuration of the base image

7. Mount layers from the base image and the derived images to the new image repository

8. Upload the new image configuration

9. Upload the new image manifest

I have done this by hand several times to test the process. But after a successful proof of concept I started automating this in Powershell.

Why PowerShell?

I decided to use PowerShell because it is a scripting language I happen to prefer. It is also a cross-platform lanugage.

(Sneak peak: I will probably also implement this in bash. But do not expect this to be published soon.)

Usage

The PowerShell code expects that you have already build and pushed the base image as well as the derived images. It will only work against the Docker registry used to store all the images.

The module called DockerRegistry contains functions to interact with a Docker registry:

• Get-DockerImageManifest - download the image manifest
• Get-DockerImageBlob - download the image configuration
• Add-DockerImageLayer - mount a layer from an existing image
• New-DockerImageBlob - upload an image configuration
• New-DockerImageManifest - upload an image manifest

These functions perform rather basic tasks necessary for Merge-DockerImageLayer which is used to merge the layers of the base image and the derived images. The following example assumes that you have followed the example in the first post about merging layers.

Install-Module -Name DockerRegistry
$Params = @{
    Registry           = 'http://10.0.0.100:5000'
    Name               = 'target'
    BaseRepository     = 'base'
    ParallelRepository = 'maven', 'golang'
}
Merge-DockerImageLayer @Params

The above commands will result in the follow rudimentary output to show you the progress of the process:

Patching layers
  from maven
  from golang
Patching history
  base has 8 entries
  from maven
    with 9 entries
    appending 1 entries
  from golang
    with 10 entries
    appending 2 entries
Patching rootfs
  from maven
    with 7 entries
    appending 1 entries
  from golang
    with 7 entries
    appending 1 entries
Mounting layers
  from maven
  from golang
Uploading config
Uploading manifest

Authentication

The above example assumes that you are working with a private registry without authentication. If you are using a proper Docker registry, you have the following options to define authentication data before calling Merge-DockerImageLayer:

1. Use basic authentication:

    $Credential = Get-Credential
    $PSDefaultParameters = @{
        'Get-DockerImageManifest:Credential' = $Credential
        'Get-DockerImageBlob:Credential' = $Credential
        'Add-DockerImageLayer:Credential' = $Credential
        'New-DockerImageBlob:Credential' = $Credential
        'New-DockerImageManifest:Credential' = $Credential
    }
   

2. Use token authentication:

    $PSDefaultParameters = @{
        'Get-DockerImageManifest:Token' = $Token
        'Get-DockerImageBlob:Token' = $Token
        'Add-DockerImageLayer:Token' = $Token
        'New-DockerImageBlob:Token' = $Token
        'New-DockerImageManifest:Token' = $Token
    }
   

3. Use an API key in the HTTP header:

    $PSDefaultParameters = @{
        # Header key
        'Get-DockerImageManifest:HeaderKey' = 'X-JFrog-Art-Api'
        'Get-DockerImageBlob:HeaderKey' = 'X-JFrog-Art-Api'
        'Add-DockerImageLayer:HeaderKey' = 'X-JFrog-Art-Api'
        'New-DockerImageBlob:HeaderKey' = 'X-JFrog-Art-Api'
        'New-DockerImageManifest:HeaderKey' = 'X-JFrog-Art-Api'
        # Header value
        'Get-DockerImageManifest:HeaderValue' = $Token
        'Get-DockerImageBlob:HeaderValue' = $Token
        'Add-DockerImageLayer:HeaderValue' = $Token
        'New-DockerImageBlob:HeaderValue' = $Token
        'New-DockerImageManifest:HeaderValue' = $Token
    }
   

Feedback

If you have feedback concerning the PowerShell code, please open an issue on GitHub.