Running server tests inside a transaction

When our tests run, we often need to clear the database between tests so we can have a clean slate.

In our old coding pattern, this wasn't enforced so there were lots of dependencies between tests, which is not a good pattern. Tests should be isolated from each other.

If we did clear the database, we were using TruncateAll() to delete all records in the database, which is slow.

For those reasons, as of June 2021, we have a new coding pattern that will use transactions to clear the database between tests and manage database connections in a more optimized fashion.

Read more for background on this.

How to update a test to the new pattern

First let's look at the old pattern. The subtests often relied on db records that were created in the TEST SETUP section. Sometimes, a later subtest would even rely on a change from an earlier subtest.

Old pattern:

func (suite *MTOShipmentServiceSuite) TestMTOShipmentUpdater() {

    // TEST SETUP
    // Set up mocked objects and other local (non-db) variables
    // Prepare the database with testdatagen and suite.DB() calls

    suite.T().Run("Subtest 1", func(t *testing.T) {
        // Run Test
        // Check errors
    })

    suite.T().Run("Subtest 2", func(t *testing.T) {
        // Run Test
        // Check errors
    })  

In our new pattern, the database will be cleaned for each subtest. So we can't rely on any db records changed in the TEST SETUP section, or any db records changed in previous subtests. We need to put all that setup into a function, called setupTestData here, and call that in each subtest.

New Pattern:

func (suite *MTOShipmentServiceSuite) TestMTOShipmentUpdater() {

    // TEST SETUP
    // Set up mocked objects and other local (non-db) variables
    
    // MOVE DB SETUP TO A FUNCTION
    setupTestData := func() {  
        // Prepare the database with testdatagen and suite.DB() calls
    }

    suite.Run("Subtest 1", func() {
        setupTestData()    // <-- Call the DB setup function
        // Run Test
        // Check errors
    }

    suite.Run("Subtest 2", func() {
        setupTestData()  
        // Run Test
        // Check errors
    }

What's different

View in PR

You can see these changes in a PR if you prefer.

Follow these directions

1. The function declaration for the main test needs to be simplified.

   • suite.T().Run → suite.Run
   • func(t *testing.T) → func()

   From:

    suite.T().Run("Subtest 1", func(t *testing.T) {

   To:

    suite.Run("Subtest 1", func() {

2. All DB setup should be moved to a function that is called in each subtest. There are more than one ways of achieving this, the following is just one example.

    // MOVE DB SETUP TO A FUNCTION
    setupTestData := func() {  
        // Prepare the database with testdatagen and suite.DB() calls
    }
   
    suite.Run("Subtest 1", func() {
        setupTestData()    // <-- Call the DB setup function
        // Run Test
        // Check errors
    }

3. Remove any calls to suite.TruncateAll() in the tests.

4. Replace suite.T().Fatalf or suite.T().Fatal with suite.Fail.

5. Remove testing from the imports at the top of the file.

6. Run all tests in the package.

How to update the package to the new pattern

Hopefully the package you are working on has already been updated to use the new pattern.

Check if package has been updated

To check, navigate to the file that sets up testing for the package, usually it is located at <path_to_package>/api_test.go

For example, for the ghcapi package, the file is in pkg/handlers/ghcapi/api_test.go.

Search for the new option testingsuite.WithPerTestTransaction().

If it does not exist in the file, you should update the package.

Update the package to use the faster testing setup

You need to make two changes to this file.

1. Update the package testing suite to use the new option - which is called testingsuite.WithPerTestTransaction()

2. Remove calls to TruncateAll(). TruncateAll empties the database and we no longer have to do this since we use the magic of transactions.

   You can also remove the function SetupTest if all it did was call TruncateAll() and suite.FatalNoError(err)

3. Update the tests to remove TruncateAll()

   Now's a good time to remove all calls to TruncateAll() from the tests in this package too!

4. Run the tests and see if any fail. If none fail, then you are done! If tests fail, read below to fix them.

Done! The package is now setup for faster tests. Thank you.

Troubleshooting

My tests started failing after making these changes

If you use Suite.Run:

• This could be because each subtest is sharing DB setup. Check that you have extracted all the shared db setup into a separate function, and call that function at the beginning of each subtest. A tip is to look for suite.DB(). The call to suite.DB() performs some of the setup to handle the per test transactions, so any call to suite.DB() (even if you're creating a stubbed object) should be moved from the parent test.

  Look at the diff of pkg/handlers/ghcapi/orders_test.go in this example.

• If the shared setup was already in a separate function, it could just be a matter of calling the setup function at the beginning of each subtest. Here's an example.

• If you've properly extracted the DB setup, it's possible a subtest was depending on the previous subtest. This is not a good pattern, try to unwind that dependency. Each test should pass in isolation from each other.

• Remove any calls to suite.TruncateAll() in the tests. Here's an example of how the models tests were converted to use transactions.

• Replace suite.T().Fatalf with suite.Fail.

• If the issue is related to appContext, make sure to get the correct appContext inside the subtest using suite.AppContextForTest().

• If you were using t from the passed in t *testing.T, use suite.T() instead.

I can't extract the db setup into a function, should I undo everything?

Sometimes, it's too much of refactor to pull out all the setup, or the setup takes a very long time.

In these cases, you can use suite.PreloadData. This can only be called once per test and will set up data that will then be used by all subtests. Once it is called, it will set a savepoint from which all subtests will start. When a subtests ends, the db will rollback to the savepoint. This way, each subtest can reuse the preloaded data, but subtests cannot modify the data used by another test. In most situations, it is preferrable to extract db setup into a setup function.

PreloadData can be preferred in one case:

• Where there is a large amount of necessary setup, like populating the rate engine for payment calculations. In this case it can be valuable to have a preloaded setup.

So if you absolutely must use PreloadData, the process is easy. Create a suite.PreloadData function and do all your test setup inside. Here's an example

Background

In June 2021, we introduced the go-txdb tool to allow us to run tests within a transaction, and then roll back the transaction after the test. This allows each test to start with a clean DB state, and is much faster than truncating the DB, which is how we've been resetting the DB all this time. Here is the PR that introduced transactions in tests: https://github.com/transcom/mymove/pull/6650

The original PR was designed in a non-breaking way such that existing tests that still use truncation can continue to run. The idea was to make the transaction feature opt-in, and incrementally update each package to use transactions. The PR updated several packages to give examples of what it takes to start using transactions. Follow the steps below to convert more packages.

About running tests in code editor or IDE

Before you start running tests in Goland or other editor, make sure to run make server_test_setup first.

Why data setup has to be repeated within each subtest

In pkg/testingsuite/pop_suite.go, we created our own Run function that overrides the default testify suite.Run to ensure that the test DB is torn down for per-transaction tests.

It would be nice if subtests could start a new transaction inside the current connection so they could reuse db setup between subtests. Unfortunately, because database/sql and pop do not support nested transactions, this gets complicated and hairy quickly. When testing that approach, connections wouldn't get closed and cause other tests to hang or subtests would report incorrect errors about transactions already being closed.

And so, if per test transaction is enabled, each subtest gets a new connection. This means subtests are really just like main tests, but subtests are a helpful way to group tests together, which can be useful. Therefore, shared setup has to be moved to a function that can be run once per subtest. In testing this approach, it was still faster with per test transactions than the old way of cloning a DB per package.

If tests require a really large amount of data setup, you can use suite.PreloadData. If PreloadData is called, all subtests will start with the data that was set up in PreloadData, but subtests won't be able to modify the data used by another test.

Note that any call to suite.DB() needs to be in a setupTestData function, suite.PreloadData, or in a subtest.

Updating/adding to existing tests that use transactions

Make a note of how the tests are set up, and follow the existing patterns, such as using suite.Run, and calling the shared DB setup function within each new subtest you add, where applicable.