Optimizing Dockerfile for Phoenix mixed with Node, Rust & Tailwind

Ok, so with Phoenix 1.6 we finally get an out-of-the-box setup based on just a single stack — Elixir. Dockerfile for such a setup may as well be a copy-paste of an example from Phoenix documentation. But, as I’ve stated in my article about assets in Phoenix 1.6, I don’t believe we could get far with a full-stack Phoenix project absent Node.

Also, I’ve loved & cherished development powered by Elixir and Phoenix for many years, but now — thanks to Rustler — it’s better and more future-proof than ever with the ease of adding Rust implants featuring crazy raw performance, baked in safety, great momentum and growing community = growing number of open source packages.

But, as you grow the stack it’ll naturally start taking more time to build everything, which makes proceeding smart very important. Heck, you could just apt-get install node rust in the builder stage in that stock Dockerfile, but (aside from other problems such as versioning) you’re letting go of all the parallelization & caching opportunities provided by Docker, ending up with build/rebuild times well over 10 minutes.

With Node-backed assets out of the Phoenix’s scope and with Elixir + Rust combo still getting there docs-wise, it’s up to us members of the community to fill the void. Thankfully, I’ve had to figure out such Phoenix + Node + Rust setup for Heyplay and below I’m gonna share the end result along with the thought process and benchmarks. And while Heyplay doesn’t use Tailwind or umbrella, I’ll also show how to adjust for it as a bonus.

The Dockerfile

Let’s start from the end. Below is the Dockerfile from Heyplay that features all the optimizations that I’ve applied. Regardless of your Dockerfile proficiency, please read it along with the comments to get the high-level picture of all the included stages. I’ll explain it all in a minute.

# Control all versions in one place
ARG ALPINE_VERSION=3.15.0
ARG ELIXIR_VERSION=1.13.1
ARG ERLANG_VERSION=24.2
ARG NODE_VERSION=14.18.1
ARG RUST_VERSION=1.57.0

# 1. Fetch Elixir deps
# - Elixir packages with package.json are brought together into deps-assets for asset compilation
FROM hexpm/elixir:$ELIXIR_VERSION-erlang-$ERLANG_VERSION-alpine-$ALPINE_VERSION as fetch-elixir-deps
RUN apk add --update --no-cache git build-base
RUN mkdir /app
WORKDIR /app
RUN mix do local.hex --force, local.rebar --force
ENV MIX_ENV=prod
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir deps-assets && find deps -name package.json -mindepth 2 -maxdepth 2 | while read i; do cp -R $(dirname $i) deps-assets; done

# 2a. Fetch Node modules -> compile Node assets
FROM node:$NODE_VERSION-alpine as build-node-assets
RUN mkdir /app
WORKDIR /app
RUN mkdir /app/assets
COPY assets/package.json assets/yarn.lock ./assets/
COPY --from=fetch-elixir-deps /app/deps-assets ./deps
RUN yarn --cwd assets install --immutable
COPY assets ./assets
RUN yarn --cwd assets deploy

# 2b. Fetch & compile Rust deps -> compile the Rust lib
# - the target-feature=-crt-static flag is needed for our lib to work on Alpine
# - Rustler's skip_compilation? flag is set to true in prod.exs to compile separately 
FROM rust:$RUST_VERSION-alpine as build-rust
RUN apk add --update --no-cache git build-base protoc
WORKDIR /
RUN cargo new app --lib
WORKDIR /app
COPY native/heyplay_engine/Cargo.toml native/heyplay_engine/Cargo.lock ./
ENV RUSTFLAGS="-C target-feature=-crt-static"
RUN cargo rustc --release && rm -rf src
COPY native/heyplay_engine/src src
COPY native/heyplay_engine/build.rs .
RUN cargo clean --package heyplay_engine --release && cargo rustc --release

# 2c. Compile Elixir deps -> compile Elixir app -> digest assets -> assemble the release
# - only the compile-time config is needed for compilation (runtime.exs follows later)
FROM fetch-elixir-deps as build
RUN mkdir config
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile
COPY lib lib
COPY priv priv
RUN mix compile
COPY --from=build-node-assets /app/priv/static/assets priv/static/assets
RUN mix phx.digest
COPY rel rel
COPY config/runtime.exs config/
RUN mkdir -p priv/native
COPY --from=build-rust /app/target/release/libheyplay_engine.so priv/native/libheyplay_engine.so
RUN mix release

# 3. Prepare non-root user -> copy the release
# - the SERVER env var is used in runtime.exs to enable Phoenix endpoint, Oban queues etc
FROM alpine:$ALPINE_VERSION AS app
RUN apk add --update --no-cache bash openssl postgresql-client libstdc++
ENV MIX_ENV=prod
RUN adduser -D -h /app app
WORKDIR /app
USER app
COPY --from=build --chown=app /app/_build/prod/rel/heyplay .
CMD env SERVER=1 bin/heyplay start

Formatting

As you can see, my Dockerfile heavily relies on multi-staging by defining a whooping number of 5 stages. Each stage is kept compact by bringing it together with no empty lines and by putting the comments for complexities that need explanation on top of each stage, while avoiding obvious comments and redundant line breaks before every single line — like the Phoenix’s stock Dockerfile does.

Also, all technologies in the stack that require building and versioning on the Dockerfile level have their versions defined on top via ARG. This helps to highlight the pieces in the stack and to avoid inconsistencies e.g. between Alpine versions in hexpm:elixir and alpine images.

The end result is a file that’s only 70 lines of code, making it shorter and IMO easier on the eye than the stock one. I’m not saying it’s less complex — it obviously takes more understanding of Docker to get a complete grasp of it, but at least the presentation doesn’t make it worse.

Optimizations

Now, let’s discuss the thought process behind this beast.

Parallelization

The main idea is to execute in parallel commands that are time consuming and independent of each other. And the tool for achieving that is multi-staging. For our case, there are three heavyweight, independent stages:

• stage 2a where we fetch Node modules and compile assets
• stage 2b where we fetch & compile Rust crates and compile our lib
• stage 2c where we compile Elixir deps and our Phoenix app

For small apps, each of these may span roughly from 1 to 3 minutes (depending on the hardware of course) while raising to 5 or more minutes for larger projects with more dependencies or fiercer asset optimizers.

Note that for this to work, stage 1 has to setup the, well… stage, in order for stage 2c to have the Elixir stack + deps in place and for stage 2a to pick up those Elixir deps that also serve as Node modules for assets.

Also note that the part of stage 2c after mix compile actually depends on stage 2a and stage 2b as it merges together Elixir + Node + Rust artifacts into the final release with fingerprinted assets.

This optimization affects both projects that use caching and those that don’t, as long as parallel building a.k.a. Docker buildkit is enabled.

Caching

Docker layer caching comes down to copying the right files into the container at the right times. Once some files are copied in, the commands executed afterwards are cached along with their results for as long as the copied files are not changed. So it comes down to copying as few files as possible, especially before commands that are time consuming.

You may see this strategy applied at various points across all stages, like:

• stage 1 followed by stage 2c when we copy Mix manifest to fetch deps, copy config to compile then and copy lib to compile the app
• stage 2a when we copy Node/Yarn manifest to fetch deps and copy all assets to build the production-ready asset bundle
• stage 2b when we copy Cargo manifest to fetch & compile deps and copy the library source to compile our Rust implants

The great twist to this technique is that we may copy between stages and the caching will work just like when copying from the source context. It’s extremely powerful considering that the source files may be filtered, rearranged, postprocessed or generated right within the Dockerfile.

This may not be that much of a deal in stage 2c and stage 3 where commands that follow the cross-stage copy only take seconds to complete, but it plays a crucial role in stage 2a where changes in Elixir deps don’t bust the cached output from Yarn commands.

This optimization requires caching to be enabled, which may require more effort than with parallel builds due to the need for persistence. I was able to cover this when deploying Heyplay to Fly.io by using deploy --remote-only to build on a dedicated, persistent remote machine. I’ve previously used DLC on Codeship and I sincerely hope for GitHub Actions to have a first-class support for it too.

Other cases

Now that we have the Heyplay case covered through and through, let’s see how to approach other common Phoenix scenarios.

Tailwind with JIT

Heyplay doesn’t use Tailwind for styling but looks like every other web app on the planet does these days, so let’s see how to adjust the above Dockerfile for Tailwind’s JIT feature that must take a look at Elixir sources in order to generate an optimized CSS bundle.

It’s simple if we need the entire directory like lib/myapp_web — just copy it before yarn deploy and you’re good to go.

If, however, you’d like to select only some of the files, you may take the “caching via cross-stage copy” technique to the next level by adding an extra stage whose sole purpose is to select only the needed files, e.g.:

FROM alpine:$ALPINE_VERSION as extract-assets-from-lib
RUN mkdir -p /app/lib
WORKDIR /app
COPY lib/myapp_web lib/myapp_web
RUN find lib/myapp_web -type f ! \( -name "*ex" \) -print | xargs rm -f
RUN find lib/myapp_web -type d -empty | xargs -r rmdir -p --ignore-fail-on-non-empty

This leaves just the Elixir sources including Heex templates, but removes things like Protobuffs, CSS or whatever else. The last line also removes empty directories that no longer hold any files to strengthen the cache.

Umbrella

In umbrella projects, there’s a concern of extracting mix.exs files from all the sub-projects in order to get the deps. This may be achieved in a similar way to the Tailwind JIT example — by recursively handpicking just these files from the apps directory. No need to hardcode each nested mix.exs into the Dockerfile and risk polluting it or hitting its size limit.

No Rust / no Node

What if you want to take advantage of optimizing Elixir + Node, but you’re not into Rust? Well, good news! Just throw away the entire Rust stage (stage 2b) and relevant lines from stage 2c that copy artifacts from it.

Same for the other way around — if you’re not into Node, either because native ESBuild fits your bill or your app is an API project, just remove the entire Node stage (stage 2a) and relevant copy from stage 2c.

Benchmark

I’ve run the benchmarks for the Heyplay project, checking the most common image building scenarios on my 16-inch MacBook Pro 2019 with 8-core Intel Core i9 CPU and 16 GB of RAM — a hardware that should be considered rather powerful by CI standards, with GitHub Actions taking roughly twice as long to build the same images.

The stock Dockerfile is basically the optimized one with multi-staging cut out, with Node and Rust installed via apk and with Rust compiled during mix compile like Rustler does by default.

Here are the results:

Here’s what we can basically read here:

• initial build time (simulated by passing --no-cache to docker build) goes down by 35% from 4:07 to 2:27, so even without caching involved (or when we completely miss it) the gain is quite considerable especially if we account for limited CI hardware resources
• cached rebuilds for changes in Elixir + assets + Rust, which I’d call the „worst case daily scenario” (as the only thing worse is updating Elixir deps or stack versions), go down by 70% from 2:23 to 0:43, which is a game-changing difference, especially for slower CIs
• cached rebuilds for changes just in Elixir, which I’d put my bets on as „the most fequent best case scenario” in a Phoenix project, go down by 88% from 2:18 to 0:16 and the only viable comment to this is 😵🔥

As pointed out above, the actual CI times that I got on GitHub Actions were much worse, often hitting 8-9 minutes for Heyplay at its early stages, making this optimization a necessity not even close to “premature”.

I’d like to close this section by looking at the specifics of the Heyplay project to consider how these results may change for other kinds of apps:

• the use of ESBuild (even if Node-backed) allows Heyplay to build production-ready assets in just around 2s while projects based on Webpack (a default before Phoenix 1.6) will have this number in the 0:20-4:00 range — this means that projects with just ESBuild like Heyplay will “only” parallelize module fetching in the Node stage (still nice as fetching Node modules is known not to be a trivial & fast task)
• on the other hand, the use of Rust heavily increases the efficiency of this optimization for Heyplay as Rust really takes its time to compile both deps and the app — it’s 1:22 and 0:34 on my Mac respectively
• Heyplay doesn’t, but for projects that use Tailwind I’d consider its impact on the above results negligible, because from my experience the Tailwind JIT only takes a couple of seconds to do its magic given a moderate Phoenix source codebase

Results for the cached rebuilds with changes in just a single part of the stack could be optimized for the stock Dockerfile by changing the order, so e.g. if we’d compile Rust before Elixir then Elixir-only changes would get cheaper, but then Rust-only changes would become more expensive. With multi-staging we don’t have to pick one or the other.

Summary

Taking advantage of caching and parallelization allows to design a blazing-fast CI and CD for Phoenix stacks extended with Node and Rust, ensuring lightning fast feedback loop and production updates — traits useful both in daily use and during emergencies.

Another great thing is that, as much as I’ve put my bets on this particular stack combo for Heyplay, the same techniques may be easily applied to accomodate other stacks — including Python, Go or C++.

Happy container-izing!