Docker BuildKit is an opt-in image building engine which offers substantial improvements over the traditional process. BuildKit creates images layers in parallel, accelerating the overall build process.
What is BuildKit?
BuildKit was developed as part of the Moby project, a Docker effort to “assemble specialized container systems without reinventing the wheel.” It was announced in 2017 and began shipping with Docker Engine in 2018’s version 18.09.
BuildKit focuses on improving build performance, storage management, and extensibility. Its headline claims are parallel processing, more advanced caching, a pluggable architecture, and automatic garbage collection. These combine into a build system that’s more efficient and more extensible than the original engine.
Layers which don’t impact on each other can be built simultaneously, reducing waiting times for stages to complete. BuildKit also optimizes access to the local files you reference with COPY instructions. It tracks the changes you make and only copies files that have been modified since the last build, instead of transferring the entire build context.
BuildKit simplifies multi-platform builds too. You can supply the –platform flag to specify targets to build for. BuildKit will automatically assemble an appropriate image manifest to cover all specified architectures.
How Does BuildKit Work?
BuildKit’s performance enhancements are facilitated by the use of a low-level build definition format, dubbed LLB. It’s a graph-based binary format that ties together complex build definitions.
As layers are directly linked, BuildKit facilitates quicker comparison of build graphs and the content they include. The default Dockerfile builder needs to rely on imprecise heuristics to determine whether two images are comparable.
The LLB is completely separate from the BuildKit “frontend.” The frontend takes a human-readable representation of an image, such as a Dockerfile, and converts it into an LLB graph. Once an LLB’s been generated, it can be exported and moved between environments. Exporting to a registry lets frontend clients acquire an existing LLB to further improve first-time build performance.
The LLB technical details are fairly complex. The underlying technology is based on graph theory and checksum comparison. You don’t need to understand it to benefit from its power: as an end user, builds work in the same way as ever, with the docker build command.
BuildKit still creates OCI-compliant images that are portable across different container environments. You can use BuildKit to create any Docker image with a Linux base. Windows images aren’t currently supported.
Activating BuildKit Support
There are two ways to enable BuildKit. If you want to build a single image with the feature, set the DOCKER_BUILDKIT environment variable in your shell:
For long-term use, configure the Docker daemon to use BuildKit by default. Create or edit the /etc/docker/daemon.json file and add the following contents to the top-level config object:
Reload the daemon configuration to apply the change:
BuildKit will now be used instead of the default build engine when you run the docker build command.
You can tell when BuildKit is active because it produces different CLI output to the regular engine. BuildKit’s progress display offers improved readability and clear visualization of when each stage starts and completes. The information includes a breakdown of the time taken to build each layer.
“docker buildx”
You can also interact with BuildKit through docker buildx commands. These will always use BuildKit. The dockerx command group exposes advanced BuildKit functionality including the ability to build on a remote host.
A single BuildKit client can interact with several distinct image builder instances. This facilitates multi-platform builds by letting you add a builder for each architecture you’re targeting.
Here’s an example of adding a remote host as a BuildKit target. This assumes the target machine has a Docker socket exposed on TCP port 2375. You can reference the host using any Docker endpoint identifier, or the name of a Docker context (obtained from docker context ls). The latter lets you build on compatible cloud environments by adding them as a context.
You can select the builder to use with the docker buildx use command:
Then you can use the build command to build your image on the selected builder instance:
You can remove builder instances using docker buildx rm, passing in the builder’s name. Builders are listed using docker buildx ls; you can use docker buildx inspect to get more detailed information about a specific builder.
If you want to check BuildKit’s disk usage, run the docker buildx du command. You can clear the build cache to free up storage with docker buildx prune. This may reduce performance next time you rebuild your image, as previously cached layers will be reconstructed.
Build Features
BuildKit adds a couple of extra build-time features to simplify your Dockerfile steps.
You can pass in secret data using the –secret flag. This lets your Dockerfile access sensitive values without storing them inside the image. The value’s only available at build time.
This Dockerfile references a secret called demo-secret. Its value is read from the demo-secret.txt file when you run docker build. The RUN instruction with the –mount flag provides access to the secret. The file is temporarily mounted into the /run/secrets directory.
BuildKit also lets you forward your host’s SSH agent so your build instructions can interact with existing remote connections. Pass the –ssh flag to docker build and add –mount=type=ssh to RUN instructions in your Dockerfile:
These features make image building more convenient without affecting overall security. SSH agent forwarding and secret mounts are only available in BuildKit; there’s no counterpart in the default build engine.
Conclusion
BuildKit is the next-generation Docker image builder which uses a graphed binary format to drastically accelerate builds. Although performance will vary considerably for each Dockerfile, you can expect to see substantial speed-ups in cases where parallel processing of image layers is possible.
BuildKit’s architecture optimizes some of the newest Dockerfile features. Multi-stage builds benefit from skipping of unused stages which makes the process more efficient than the standard builder.
Although BuildKit is now stable, Docker still doesn’t ship with it on by default. Make sure to enable it in your Docker client if you want to use its features. There’s an active proposal to make BuildKit the standard build engine but there are still unresolved issues preventing the switch.
