Speed up brew 3.x updates in an ephemeral macOS environment

I recently optimized a pipeline responsible for building Python wheels for a variety of platforms and CPU architectures. One of the targets was macOS 12.x. The third-party CI provider offered ephemeral macOS virtual machines.

The machines I was dealing with came pre-installed with Brew 3.2 and a very outdated formula list. I needed up-to-date formulae for my use-case. I configured the pipeline to run brew update prior to installing them. Unfortunately, brew update took over 3 minutes to complete.

Taps are slowing us down

Most the of time executing the brew update command was spent around the following logs:

Checking if we need to fetch /usr/local/Homebrew/Library/Taps/adoptopenjdk/homebrew-openjdk...
Checking if we need to fetch /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask-versions...
Checking if we need to fetch /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core...
Checking if we need to fetch /usr/local/Homebrew/Library/Taps/rbenv/homebrew-tap...
Fetching /usr/local/Homebrew/Library/Taps/adoptopenjdk/homebrew-openjdk...
Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask-versions...
Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core...
Fetching /usr/local/Homebrew...
Fetching /usr/local/Homebrew/Library/Taps/rbenv/homebrew-tap...
Updating /usr/local/Homebrew...
Branch 'master' set up to track remote branch 'master' from 'origin'.
Switched to and reset branch 'master'
Your branch is up to date with 'origin/master'.
Switched to and reset branch 'stable'
Current branch stable is up to date.

Updating /usr/local/Homebrew/Library/Taps/adoptopenjdk/homebrew-openjdk...
Successfully rebased and updated refs/heads/master.                             

Updating /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask-versions...
Successfully rebased and updated refs/heads/master.                             

Updating /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core...
Successfully rebased and updated refs/heads/master.                             

Updating /usr/local/Homebrew/Library/Taps/rbenv/homebrew-tap...
Successfully rebased and updated refs/heads/master.

This is updating the taps, which are Git repositories containing all the formulae. Our local clones of these repositories are very old. The git fetch and subsequent git checkout are taking a very long time.

Brew 4.0.0 replaced fetching the formulas from Git with fetching them from an HTTP API. This resulted in a significant speed boost for most users. Although Brew updates itself prior to fetching the taps, it does not perform the taps update with the updated version of brew. In other words, we’re stuck with the slower git-based fetch approach to updating the taps till we update.

I couldn’t find a documented way of forcing Brew to update itself first before updating the formulas. If we had at least Brew 3.3.0, we could’ve already opted into the new API-based fetch by setting the HOMEBREW_INSTALL_FROM_API=1 environment variable. Since I was on Brew 3.2.0, this wasn’t really an option.

Updating brew without updating taps

Brew itself is installed by cloning/downloading the Brew Git repository. We can fetch and check out a newer version outside the Brew CLI and then update the formulas with Brew 4.x:

Taps are slowing us down

Updating brew without updating taps

Impact