In this guide, you will learn how to leverage Neon’s branching feature to create a database for every preview environment. We will use Vercel as an example deployment provider and GitHub actions as the  CI/CD tool.

Staging environments and why they are not ideal

Teams often set up a production-like environment, known as “staging”, where they test and validate new features before releasing them. The goal of this pattern is to ensure quality; however, it introduces challenges when you have several developers collaborating on the same project:

1. If the staging environment goes down, the entire team cannot preview their changes before deploying to production.
2. If a developer wants to preview their changes in isolation, they will prevent other team members from deploying to the staging environment.
3. You likely end up with a queue of changes that need to be reviewed, forcing you into doing a big release that includes several changes. The problem with doing a big release is that if something goes wrong unexpectedly, and you’re unsure of the cause, you must undo all of your changes.

Thankfully, a new pattern is becoming increasingly popular:  preview environments. 

Preview environments: A better way to build software

Rather than having a shared and fixed staging environment where all developers collaborate, you automatically provision a production-like environment for every new code change a developer wants to introduce. In other words, every pull request will have its own isolated, production-like environment.

This enables developers to build new features in parallel without affecting each other. It also makes it possible to do frequent small releases, making it easier to revert changes if something goes wrong. 

Many deployment platforms are starting to support this flow out of the box by enabling you to set up an automatic Git integration. This way, you do not have to worry about maintaining and managing the necessary infrastructure for handling this flow.

Working with databases in preview environments

When teams want to set up preview environments for an app that is backed by a database, they typically pick one of the following approaches:

• Setting up a shared database for all previews
• Creating a database for every pull request and seeding it with random data
• Creating a database for every pull request and seeding it with production data

All of these options are not ideal and present their own drawbacks. 

Creating a database for every pull request and seeding it with random data means that preview environments will not accurately represent production. So you will not be able to make changes to your production database confidently. 

The alternative would be to create a database for every pull request and import a backup of the production database. The problem with this flow is that depending on the size of your database, it can take a long time to deploy a preview environment.

That is why teams often skip the step of creating a database for every preview and just use a shared database. The drawback is that any changes made to that shared database will impact all active previews. So, if a developer’s changes include evolving the database schema, they must notify their team members about this change and potentially prevent them from deploying. Furthermore, if the shared database goes down, no one will be able to create preview environments. This flow is somewhat similar to having a staging environment, but now the database is the bottleneck.

The great news is that you can use Neon’s branching feature to create a production-like database for every preview. This database will contain production data and can be created in seconds.

What is Neon?

Neon is fully managed serverless Postgres. This means you don’t have to pick a size for your database upfront, and it will automatically allocate resources to meet your database’s workload. 

To get started, go ahead and create an account. Next, you will want to create a new project. Choose 15 as the Postgres version, pick the region that’s closest to where you want to deploy your app and pick a size for your compute endpoint (you can change this later). After you create the project, you will get a connection string that you can use to connect to your database.

Neon’s architecture separates storage and compute. This makes a Neon Postgres instance stateless, which makes it possible to automatically scale compute resources up or down based on demand.  To learn more, check out Neon’s architecture.

Neon’s object hierarchy

When you create a project in Neon, a default Postgres cluster is created with a default database called neondb.  Within a project’s Postgres cluster, you can have many databases. 

This is important to know because it will help you understand how Neon’s branching feature works.

Neon branching

Neon enables you to create copies of your project’s Postgres cluster, where each copy is completely isolated from the other. We refer to this copying process as “branching” and call each copy a “branch”. Branches are created using copy-on-write, making it fast and cost-effective.

The default Postgres cluster that gets created along with a new project is represented by a branch called main.

Creating a branch in the Neon console

To see branching in action, you will first need to create a new table and add some data to it.  Navigate to the SQL Editor and add the following SQL query.

This query creates a new table called elements. The table has three columns:element_name, atomic_number, and symbol. You are then inserting ten elements into that table.

You can check that the data was added by going to the “Tables” page. You will find that you have a table called elements with ten items.

To create a branch, go to the “Branches” page, and click “New Branch”. You will then be redirected to the branch creation page.

Here, you will first need to specify the parent branch that will be copied. Right now, you only have the main branch, but you can create a branch from other branches as well. 

You will then need to specify the data you want to include. For that, you have several options:

• Head: this will include all of the data up to the current point in time
• Time: this enables you to include all data up to a certain point in time. This is useful if you want to restore your branch to a previous state. You can create a branch from any previous point in time as long as it falls within the default seven-day history window.
• LSN: this stands for Log Sequence Number, which is a pointer in Postgres’s write-ahead log. This option is more specific than the “time” option, and you can use it to revert a branch to a previous state.

Finally, you have the option to configure a compute endpoint. If you want to connect to a branch from a client or application, you will need to configure one. If you decide not to, you can do it later. Branches that don’t have a compute endpoint associated with them act as a backup or a snapshot.

Choose the main branch as the parent, pick the “Head” option, and click “Create a branch”. You will get a new connection string that is different than the main branch. This connection string will be associated with the newly created branch. 

If you go to the tables page, you can select the newly created branch, and you will find that it contains all of the main branch’s data. 

Now, to test that branches are isolated from one another, go back to the SQL Editor, select the newly created branch from the branch selector drop-down in the top-right of the console, and run the following query:

This query adds ten more items to the elements table. If you go back to the tables page, you will be able to compare the difference in data between the newly created branch and the main branch. 

Creating a branch for every Preview environment

Neon offers an API that you can use to manage resources programmatically. You can use it along with a CI/CD tool to achieve the following flow:

1. A developer will work on changes locally, which may or may not include database migrations.
2. Once they are happy with their changes, they open a pull request which triggers a CI/CD workflow that does the following:
   1. Creates a Neon branch
   2. Applies database migrations if there are any
   3. Creates a preview deployment and uses the newly created Neon branch for the database
3. If everything looks good in the preview environment, the developer will merge their changes. This will trigger another CI/CD workflow that deploys the app to production. This workflow does the following:
   1. Applies the database migrations to the main Neon branch (this is your production database)
   2. Creates a production deployment
   3. Deletes the old Neon preview branch, which is no longer needed

Seeing the flow in action

Setting up the demo project locally

To see the flow in action, we created a demo app that displays the data coming from the elements table that we created previously. The app is built using Next.js, Prisma, and Neon and is deployed to Vercel. 

For our CI/CD pipeline for this project, we’ll use GitHub Actions, which are automated workflows that you define as code. These workflows are written in YAML, and they live in a .github folder inside your project’s repository. 

To clone the demo app locally, run the following command:

You will then need to copy the .env.example file to a new file called .env. You can do that by running the following command:

Next, add the following environment variables which you can get from the Neon console. Add the database credentials:

Finally, run the setup script, which creates the tables and runs a seed script:

You can now start a development server by running the following command:

This command starts a development server which will run at http://localhost:3000. You should see the following UI:

A look at the GitHub actions

If you open the project in your text editor of choice, you will find that there are two files in the .github folder located at the root of your project:

• deploy-preview.yml: this workflow is responsible for creating and deploying to a preview environment. It runs whenever a pull request is opened.
• deploy-production.yml: this workflow is responsible for deploying the app to production. It runs whenever a commit is pushed to the main git branch.

Before diving into the code for these two workflows, you must add a few secrets to your GitHub project’s repository. To do that, navigate to your repository in GitHub, and go to  Settings > Security > Secrets and variables > Actions. You will then need to add the following secrets:

• VERCEL_TOKEN – this is the API key for Vercel. It will be used to create preview environments (also known as Preview Deployments) and deploy the app to production. You will find it in Vercel, in Settings > Tokens > Create Token.
• NEON_API_KEY – this is the API key for the Neon user, which will be used to create the branches. You can find it in your Neon account, under Settings > Developer settings > Generate new API key.
• NEON_PROJECT_ID – this is the ID of the Neon project. You can find it in the Neon console, in the Settings page. 
• DATABASE_URL – this is the connection string of the production database. It uses connection pooling. You can find in it in the Connection Details widget on the Neon Dashboard.

You can learn more about connection pooling in the Neon documentation.

• DIRECT_DATABASE_URL – this is the direct connection string to the database. Prisma Migrate requires this to be able to run migrations reliably. You can find in it in the Connection Details widget on the Neon Dashboard.
• PG_USERNAME – the name of the database role that will be used when creating a branch. You can obtain the user name from your Neon connection string.
• PG_PASSWORD – password of the database role that will be used when creating a branch. The password can also be found in your Neon connection string.

These are all of the secrets you will need. Now, here is an overview of the GitHub actions that this project uses. When you open the deploy-preview.yml file, you will see the following:

This workflow runs every time we create a pull request by listening to the pull_request event. It checks out into the repository and gets the pull request’s SHA. This is a unique identifier for every Git commit and will be used as the branch name when creating a branch. This is done so that when we merge the pull request, we can find the Neon branch by its name and delete it. 

Next, a Neon branch is created by using the create-branch-action. This is a GitHub action created and maintained by Neon. The action takes the following arguments:

• project_id: the Neon project ID. This value is loaded from the repository secrets. 
• branch_name: we get this value from the previous step.
• api_key: your Neon API key.
• username:  the username of the database role that will be used when creating a branch. This value is loaded from the repository secrets. 
• password: the password of the database role that will be used when creating a branch. This value is loaded from the repository secrets. 

The next step is running Prisma Migrate, which is an imperative database schema migration tool,  against that newly created branch. We first generate the Prisma Client and then run the prisma migrate deploy command. This command makes Prisma Migrate compare the migrations folder that is located in prisma/migrations against the specified database. If there are any pending migrations, it will apply them. If the migration runs successfully, the workflow creates a preview deployment on Vercel using the Vercel CLI.

Finally, we comment on the pull request with the URL of the preview deployment, which we get from the deployment step, along with a link to the Neon console for the project branch.

It is a good idea to disable the automatic preview deployments that Vercel creates because they are now triggered using a GitHub action. To do that, you need to enable the “Ignored Build Step” field. Go to your Vercel project settings, select Git, and then add the following command: if [ "$VERCEL_ENV" == "production" ]; then exit 1; else exit 0; fi. This command causes Vercel to skip triggering a build if it’s a preview. You can learn more in this guide.

When you open the deploy-production.yml file, you will see the following:

This workflow runs every time we push a commit to the main branch on GitHub by listening to the push event.  It checks out into the repository and gets the pull request’s SHA. It then returns the Neon Preview branch ID by sending a request to the Neon API, which lists all project branches filtered by the pull request’s SHA. This will be used in the branch deletion step.

Next, the workflow runs Prisma Migrate, but this time it runs it against the production database. It then deploys the app to production. Finally, the old Neon Preview branch is deleted.

Conclusion

In this tutorial, you learned how to provision a database for every preview environment by leveraging Neon’s branching feature along with GitHub Actions and Vercel.

You can swap out GitHub actions with another CI/CD tool or swap out Vercel with another deployment provider, as long as they support the preview environment workflow. If there are other deployment providers or CI/CD tools you would like us to cover, feel free to reach out to us in our community forum. 

If you want to try out this flow on Neon, you can sign up today for free. No credit card required.