Skip to content

Integration tests in Node.js and Jest

How to do integration testing within a Node.js service with Jest

Published by Nikolas Rist on / Backend, Nodejs

In the previous article, we looked at how to write component tests within a NodeJS service. This helps to test functionality through the whole application, including an external system (e.g., the database). At some point, you might wonder how to test your Data Access Objects (DAOs), this is the time when integration tests come into play.

You can find a full implementation example for unit, component, and integration tests within this GitHub repository. All following code snippets come from this repository. Please don't hesitate to write comments or create pull-requests to improve the examples.

Why?

Today we can say, testing DAOs with unit tests is not a good idea because you test the implementation instead of functionality. This is error-prone and in addition introduces a lot of overhead work.

Back in time, we started with the idea of unit-testing the DAOs of our service. We are using Knex raw queries heavily, and we had many releases failing at runtime or sometimes during the component test. This is everything but not failing fast and releasing a reliable service.
Our strategy was to check the Knex raw function to receive the expected string. Somehow, naive to think this will prevent us from creating bugs and broken queries. Even beforehand tested queries within a playground did not guarantee a working query in the end. At this time, we decided to remove all unit tests for DAOs and replace them with proper integration tests, which really test the queries against a database container.

This article provides you with a brief overview of how you can add integration tests, in addition to unit and component tests, to expand your testing strategy.

Component vs. Integration test

Let's have a brief look at the difference between component and integration tests. As stated in the info deck on microservice testing by Toby Clemson, component tests always test a whole service. In contrast, the integration test, only tests one module of your service communicating with an external service.

File structure

Let's see which files are in place within the repository and what is new in contrast to the previous article.

In addition to our bar.component.test.ts and foo.unit.test.ts files, we now introduce the foo.dao.integration.test.ts file. Next to the new integration file, we extract the setup of our database with testcontainers into a setup-test.ts file, which allows re-using in the component and the integration tests. The integration file lies next to the DAO file to have a direct relation, similar to unit tests.

The new file structure looks as follows:

- src
  - domain
    - foo-handler.ts
    - foo-handler.unit.test.ts
    - foo.dao.ts
    - foo.dao.integration.test.ts
    - foo.models.ts
  - tests
    - bar.component.test.ts
    - setup-test.ts

How to test the DAO

The main question is: How can we ensure that the DAOs are working as expected and that refactoring hasn't introduced regressions?

We use the fooDAO as an example to answer this question. The fooDAO, provides an insert and a delete function.

The first code example is the unit test approach we came from and should be avoided, as said before. The second example is the integration test approach, which we recommend.

These code examples show the main difference between both approaches with a simple use case. At first glance, you might argue, the overhead is not that big, so let's do unit tests. In reality, we had hundreds of lines of code for mocking and assertions in our unit test files. This made the tests a mess and hard to maintain or understand what's going on. Not to mention that bugs were simply not found.

import { Knex } from 'knex';
import { Foo, FooUpdate } from './foo.models';

export type DAO = typeof fooDAO;

export const fooDAO = {
  insert: async (pg: Knex, item: FooUpdate): Promise<Foo> => {
    const items = await pg<Foo>('foo').insert(item).returning('*');
    return items[0];
  },
  delete: async (pg: Knex, itemId: string): Promise<number> => {
    return await pg<number>('foo').where('id', itemId).del();
  }
};

In the example implementation, each DAO function awaits the Knex client as the first parameter. Another possible implementation is, to use classes which have the Knex client as constructor parameter.

For integration testing, it is mandatory to have the Knex client as input and not as imported dependency. Either directly within the function or in the constructor.

But, why is it mandatory? For each integration test, the testcontainer is created on the fly with some random port defined by the testcontainers library. Therefore, you need to hand the created client into the DAO, and cannot create a static test config upfront.

Unit test

With the DAO in place, the according unit test may look like this:

import { Knex } from 'knex';
import { fooDAO } from './foo.dao';

const pgMock = {
  insert: jest.fn(),
  returning: jest.fn(),
  where: jest.fn(),
  del: jest.fn(),
  table: jest.fn()
};

describe('FooDAO', () => {
  beforeEach(() => {
    pgMock.insert.mockReset().mockReturnThis();
    pgMock.returning.mockReset();
    pgMock.where.mockReset().mockReturnThis();
    pgMock.del.mockReset();
    pgMock.table.mockReset().mockReturnThis();
  });

  it('inserts', async () => {
    pgMock.returning.mockReturnValue([]);
    const item = { name: 'TestFoo' };

    await fooDAO.insert(pgMock as unknown as Knex, item);

    expect(pgMock.table).toBeCalledWith('foo');
    expect(pgMock.insert).toHaveBeenCalledWith(item);
    expect(pgMock.returning).toHaveBeenCalledWith('*');
  });

  it('deletes', async () => {
    pgMock.returning.mockReturnValue([]);

    await fooDAO.delete(pgMock as unknown as Knex, 'testItem');

    expect(pgMock.table).toBeCalledWith('foo');
    expect(pgMock.where).toHaveBeenCalledWith('id', 'testItem');
    expect(pgMock.del).toHaveBeenCalled();
  });
});

You already see the overhead introducing the mock for the Knex client, implementing each used function as a jest.fn(). Assume you add another call in the DAO, you have to adapt the mock and the specific test, checking that the function is called. This results in a lot of boilerplate code and overhead work for simple and small changes.

In contrast to this, the integration test uses a real client and nothing needs to be mocked. Only some test data must be defined, if necessary. The integration test, should test all functionality of the DAO. In this example, both test cases are tested at once, this allows to re-use the created object. In reality, you might want to split the tests for each DAO function into different tests, depending on the complexity and data needed. This is totally up to you and the data you use.

Integration test

import { ContainerInstance, 
          startContainer, 
          stopContainer, 
          TEST_TIMEOUTS 
        } from '../test/test-setup';
import { fooDAO } from './foo.dao';

describe('FooDAO', () => {
  // Timeout is needed otherwise test fails with missing database connection
  jest.setTimeout(TEST_TIMEOUTS); 
  let containerInstance: ContainerInstance;

  beforeAll(async () => {
    containerInstance = await startContainer();
  });

  afterAll(async () => await stopContainer(containerInstance));

  it('inserts and deletes items', async () => {
    const result = await fooDAO.insert(containerInstance.pg, { name: 'testFoo' });
    expect(result).toMatchObject({
      name: 'testFoo'
    });
    expect(result.id).toBeTruthy();

    const deletionResult = await fooDAO.delete(containerInstance.pg, result.id);
    expect(deletionResult).toEqual(1);
  });
});

Let's walk through some details within the integration test.

Similar to the component test, there is a beforeAll function spinning up our ContainerInstance (holding the Knex client and the testcontainer reference), and the afterAll function stopping the container instance again. Within our test, we call all DAO functions we want to test and expect the result. In this case, the created Foo object and the count of deleted items. You can find a more detailed explanation to the setup file within the previous article about component tests.

Test separation

As said above, it is up to you, how to structure your tests. Nevertheless, for simple CRUD operations, you can use one test case to have all functions tested one after the other. This reduces the overhead of data preparation for each test case and makes the test more concise.

Different use cases should be separated into different test cases to reduce dependencies between DAO functions while testing. This has two major benefits. First you really test only the single functionality, the scope is narrow and precise. Second, you reduce the amount of work if something on a different function changes. Only the necessary test needs to be updated instead of all existing test cases using this function to prepare test data.

Another option to avoid dependencies within tests and used DAO functions is, to use plain database queries on the client to prepare your tables for each test.

A rule of thumb, on how to split your tests, might be: CRUD operations can be tested within a single test. All functions providing business logic should be tested separately, as they might change more often and are more complex.

Test data

Another challenge is, how to store and provide test data. Here again, finally, it is up to you how to do this. Our advice is, for complex logic using numerous parameters and including many tables, to have a prepared default data set in place. This might be stored somewhere in SQL or typescript files. Implement a loadTestdata function in your setup file to add all data upfront and at once.

Keep in mind to clear the database and re-fill it for each test scenario. Take care to not represent or include edge cases within the default data set. As soon as you test for edge cases, add or remove such data on the fly within each test, to have no impact on other tests by changing the default data set in general.

Small functions like crud operations can be tested with test data defined directly within the test file. Here as well, it might come to the point where you need so much test data, which clutters your tests. At this point, you can think about if it is worth moving all test data out into separate files to import from. This has the drawback, that you lose the direct view, of how your test data looks like, but you get more overview within the test file on the other hand.

Run integration tests

Similar to the component tests, we have to introduce a new script within the package.json file to call jest with the specific pattern matching for our integration tests.

  "test:integration": "jest integration",

Performance

In general, Jest can run seven integration tests in parallel. The biggest impact on the duration for integration tests is the starting of the different docker containers. As this happens in parallel, it does not make a huge difference having two or seven integration tests running.

Spinning up a single container for all integration tests is currently not possible, due to the implementation of Jest. You cannot hand in the connection information for the database. As the parallel starting of all seven containers at once is no performance issue, we stopped investing in building a workaround to make it somehow work. On the contrary, the current implementation with one testcontainer per test brings the further advantage that the individual tests are completely decoupled from each other.

The performance improves slightly, when you have the docker image already downloaded upfront. Within our CI/CD pipeline, we have a duration of ~1.5 minutes for nine integration tests. From my perspective, this is still fast enough to have integration tests within the pipeline, as it brings you a lot of confidence in the functionality of your service. Additionally, we do not see any need to invest more time in trying to improve the performance of the integration tests at all.

Summary

With the component tests in place, it was easy to introduce integration tests within our project. The cost of ~1.5 minutes for the integration tests running within our CI/CD pipeline is worth it. With these tests, we ensure all our DAOs are working correctly and especially our raw SQL statements are correct. This was not the case, while having unit tests, and it was not guaranteed that the component test covers all edge cases.

Integration tests are another powerful set of tests within our testing pyramid to ensure everything is working as expected and allow us to release the service with confidence.

Image of the author

Published by Nikolas Rist

Visit author page