Measuring Gatsby Projects Build Time Using Paid Plans of Popular Static Website Hosting Platforms
Jamstack is growing at full speed. New tools and frameworks pop up almost every month, giving static site generation a second life.
Our go-to framework for building websites has always been Gatsby. Three years ago, when we started using it, there weren't many options for deploying a site: Netlify or Vercel. Today the landscape looks very different: you also have Gatsby Cloud, Cloudflare Pages, AWS Amplify, and you can even set up similar build tooling on your own infrastructure with projects like Waypoint.
A statically generated site has one major drawback: build time. If you have a small blog or a one-page site, any hosting provider or framework will give you very similar results and you won't notice the difference. But things get complicated once you go beyond 100 pages. At that scale, the platform can make a huge difference in build performance for developers, designers, and marketing teams, because nobody wants to wait hours to see their changes published.
So I decided to run tests on the most popular platforms using their paid plans.
Table of Contents
Platforms
Netlify
Pro plan: $20/month per member.
Vercel
Pro plan: $20/month per member.
Gatsby Cloud
Pro plan: $50/month with 2 seats included and $20 for each additional member.
Cloudflare Pages
Pro plan: $20/month, unlimited seats.
AWS Amplify
The first 1000 minutes are free; after that, $0.01 per build minute.
Self-hosted
$50/month: your own server running Drone CI with these specs: AMD Ryzen 5 3600 Hexa-Core "Matisse" (Zen2), 64 GB of RAM, and a 512 GB NVMe SSD in RAID. Unlimited seats. Deployment to Netlify with netlify-cli.
Test Cases
I started by measuring build time with an empty cache. While you'll normally have a cache that works well and reduces build time, there are cases where you need to clear it by hand, or Gatsby clears it for you when you change gatsby-node.js, gatsby-config.js, or package.json.
I ran five executions per test and noted the average. I also used a stopwatch to check whether there was a significant difference between the actual completion notice and the figure shown by the interface. The stopwatch wasn't meant to be precise, just to find out whether the platforms tell the truth and don't leave out build phases like environment initialization or publishing from their reported time.
Markdown-based Website
Build time with an empty cache
The first test was on a Markdown-based website. The results surprised me quite a bit. Gatsby Cloud finished in two minutes, eight times faster than Netlify, which took 16. Gatsby Cloud's times are very close to what I get running builds locally on Apple M1 chips, which is crazy. I'm sure they're doing some magic behind the scenes, at the very least parallel image optimization.
Vercel was somewhat faster than Cloudflare Pages: 10:30 versus 11:23. I clearly underestimated AWS Amplify: I thought they'd run builds on free-tier EC2 instances, but they use a machine with 4 vCPUs and 7 GB of memory (nice to see that transparency in the interface), so it finished in 9:47.
The self-hosted solution was quite fast and came very close to Gatsby Cloud in build time, but uploading the results to Netlify takes roughly half that time, so it ended up in second place.
The funny thing is that Gatsby Cloud managed to complete the entire build in the time it took Cloudflare just to initialize the environment.
Build time for a Deploy Preview
Here I started to suspect this was going to be a dead heat, because the competitors began showing their weak spots and how differently they handle each scenario. For the Deploy Preview test I created a branch off master and made two changes: one in a jsx file and one in a md file.
Netlify (16:16), Cloudflare (11:16), and AWS Amplify (9:14) gave the worst results, and here's why. Netlify has a very handy plugin that preserves the cache between builds, BUT it doesn't work if the build takes longer than 15 minutes.
Cloudflare Pages doesn't offer any option to preserve the cache between builds, so it always gives slow results.
AWS Amplify does let you preserve the cache, but none of its official guides explains how to make it work. Simply adding the necessary folders to its build yaml file doesn't do the trick. After spending a reasonable amount of time investigating it, I gave up, because I'm fairly sure its users won't want to get into that mess either. Without that setup, the only thing preserved between builds is node_modules.
Vercel did a great job and finished in 3:32. I was impressed, because all the caching magic happens behind the scenes and you don't need to install any extra Gatsby plugins, either on the platform or in the code, for it to work correctly.
Gatsby Cloud won again with 0:49, and the self-hosted version was only 25 seconds behind.
Build time with a warm cache
By build time with a warm cache I mean a subsequent build triggered from the same branch that already had a build run on it before. This is what happens every time you redeploy a site after a content change or push more commits to your Pull Request.
Vercel (1:08) and the self-hosted version (1:06) were very close, but Gatsby Cloud was faster at 0:27.
Cloudflare Pages, Netlify, and AWS Amplify, on the other hand, definitely have room for improvement.
Headless WordPress Website
Next I tested Gatsby v4 connected to a WordPress site. The site has two languages, a good number of common WP plugins, and about 1000 pages in total. This is where things get tough.
Build time with an empty cache
Netlify didn't even finish the build: it stopped at 22 minutes. Vercel and Cloudflare Pages completed it in 24 minutes. Gatsby Cloud was fast, but even for it this was hard: 17:47. AWS Amplify was the biggest surprise, with consistent times around 15 minutes. The self-hosted CI got the best result: 11:30. The server used was probably closer to the WP hosting and took less time to download the initial assets.
Build time for a Deploy Preview
I think we can consider Cloudflare, Netlify, and AWS Amplify eliminated at this round, so let's go straight to Vercel. It took 2:39 on the Deploy Preview, a very good result. Gatsby Cloud did the same in 1:45, and the self-hosted version handled it in 1:23.
Build time with a warm cache
On Vercel there was no difference compared to the Deploy Preview. On Gatsby Cloud, on the other hand, it took 20 seconds less. The self-hosted version kept up its excellent results. That said, I have to mention that on one occasion uploading files via Netlify CLI from the self-hosted version took more than 6 minutes. I decided not to include it since it only happened once.
Extra
The analysis wouldn't be complete without showing you one more thing. All the tests above were done by changing a jsx file and a blog post. But look at what Gatsby Cloud can do when you make a content change (from a CMS like WP, Contentful, Prismic, etc.), which is the most common scenario.
From an architecture standpoint, and judging by the build logs, it looks like Gatsby Cloud keeps the latest instance of your production version idling, so it can wake it up quickly, skip some build steps, and, knowing the content diff, generate a new build in very little time.
Only 20 seconds passed between the change in the CMS and seeing it live in production! It's really impressive. A few years ago, Gatsby's promise of rebuilding a site in a matter of seconds sounded hard to believe, but now it seems like it might actually be true. Good job!
Note: this result only happens if you change a post or a page. If you change shared data, like menus, it will have to rebuild every page and will end up taking around 50 seconds.
Winners
Gatsby Cloud won the race with the best build time in its category. I'm sure it uses the most powerful machines of all the platforms and does some magic behind the scenes. Vercel proved to be a solid competitor: it didn't shine on cold runs, but caching changes everything. In practice, nobody worries about a build that takes less than 10 minutes, but once you cross that threshold the questions about how to speed it up start piling up. With that in mind, I can comfortably recommend both Gatsby Cloud and Vercel.
Despite how much I like Cloudflare's products, Pages seems like the weakest one in their catalog. Development has been very slow and I haven't noticed any significant changes between the first beta and the current version. In my opinion, Cloudflare has the most affordable and reasonable pricing plan, with no limit on seats or bandwidth. So it's worth keeping an eye on.
AWS Amplify... what can I say? It's AWS. If you love their services and their UX doesn't seem like the worst in the world to you, it can still be an option. Build configuration in YAML files, separate settings to enable Deploy Previews and builds for other branches, no Cache-Control policies out of the box... none of this works in its favor.
Netlify is, I suppose, the most popular and very affordable solution on the market. Great interface and an impressive feature set, but build time can become a serious bottleneck if your site grows quickly.
A self-hosted alternative is fast, has no limits, and costs only $50/month for a very powerful machine; just keep in mind that you'll probably need to spend hours on setup every now and then.
