Experimental Page Build Optimizations for Incremental Data Changes
Building sites with large amounts of content (10,000s nodes upwards) is relatively fast with Gatsby. However, some projects might start to experience issues when adopting CI/CD principles - continuously building and deploying. Gatsby rebuilds the complete app on each
gatsby build which means the complete app also needs to be deployed. Doing this each time a small data change occurs unnecessarily increases demand on CPU, memory, and bandwidth.
One solution to these problems might be to use Gatsby Cloud’s Build features.
For projects that require self-hosted environments, where Gatsby Cloud would not be an option, deploying only the content that has changed or is new (incremental data changes, you might say) can help reduce build times, deployment times and demand on resources.
For more info on the standard build process please see overview of the gatsby build process
To enable this enhancement, use the environment variable
GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true in your
gatsby build command, for example:
GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
This will run the Gatsby build process, but only build pages that have data changes since your last build. If there are any changes to code (JS, CSS) the bundling process returns a new webpack compilation hash which causes all pages to be rebuilt.
You may want to retrieve a list of the pages that were built. For example, if you want to perform a sync action in your CI/CD pipeline.
To list the paths in the build assets (
public) folder, you can use one (or both) of the following arguments in your
--log-pagesparameter will output all the file paths that were updated or deleted at the end of the build stage.
--write-to-filecreates two files in the
.cachefolder, with lists of the changed paths in the build assets (
newPages.txtwill contain a list of new or changed paths
deletedPages.txtwill contain a list of deleted paths
If there are no changed or deleted paths, then the relevant files will not be created in the
This enhancement works by comparing the page data from the previous build to the new page data. This creates a list of page directories that are passed to the static build process.
To enable this build option you will need to set an environment variable, which requires access to do so in your build environment.
This feature is not available with
You will need to persist the
publicdirectories between builds. This allows for comparisons and reuse of previously built files. If
.cachedirectory was not persisted then a full build will be triggered. If
publicdirectory was not persisted then you might experience failing builds or builds that are missing certain assets.
Any code or static query changes (templates, components, source handling, new plugins etc) will prompt the creation of a new webpack compilation hash and trigger a full build.
Note: When using the
GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES flag it is important to do so consistently when building your project. Otherwise, the cache will be cleared and the necessary data for comparison will no longer be available, removing the ability to check for incremental data changes.