Introduction to unit testing

One of the most important parts of writing maintainable code in any language, and that includes C++, is writing unit tests. There are some people who might disagree, and not all of those disagreements are completely ill-conceived, but leveraging unit tests is a huge part of my process and so they’ll be a large part of entries in this blog. This post provides a very brief introduction to get us started.

Even developers who don’t agree with unit testing generally agree that developers need to do some kind of testing of their own code before passing it on. In my experience its best if that is automated and that’s easiest if done in the same language that the production code is in. So even those who disagree with me that unit testing is an important part of writing sane code can leverage similar methods to write whatever kind of tests they’re writing.

I use Boost.Test for all my unit testing, and more even. There’s no great technical reason why I chose this framework over say googletest or one of the many new ones that keep coming out. I use it because it’s easy to use, I’m already using boost, and I find that it’s more than sufficient for the task. I’ve heard of there being issues around maintenence and the library author being a PITA about fixing things, but I’ve never run into any of that. There are many times when, “Who cares, it’s working,” are more than adequate reason to use something–you know, until it stops working. I do believe you need more than just C asserts so I’m not doing that–many people find asserts work just fine for them and so that’s a great answer too.

Integrating unit tests into the build

Unit tests should be easy to run as a target in the usual build setup that developers always use. For C++ this is generally make. Not many people are insane enough these days to use make directly but instead use something else that generates the makefile for them. I have so far found cmake to be good enough; better than autotools at least. It’s got its problems but it’s good enough and until I find something else I like, this blog will use it.

First thing you need to do specific to testing (see code on github for the rest) is to tell cmake that you will be adding tests. This must be done before you ever call add_test:

enable_testing()
include(CTest)

Next you need to tell cmake how to find Boost.Test. I do this in the CMakeLists.txt located in the test code directory:

find_package(Boost)
find_package(Boost COMPONENTS unit_test_framework)

I also like to add a check target to run the tests much like I would do with autotools in the past. This target builds all, builds the tests, and then runs them by activating cmake’s ctest system. I also add a tests target to just build the tests without running them:

add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND})
add_custom_target(tests)

Finally, you need to ensure that the boost headers can be found and I also like to have a function for adding unit test programs rather than needing to repeatedly do the same thing over and over:

include_directories(${Boost_INCLUDE_DIR})

function(add_boost_test name)
  add_executable(${name} EXCLUDE_FROM_ALL ${ARGN})

  target_link_libraries(${name} ${Boost_UNIT_TEST_FRAMEWORK_LIBRARY})

  add_test(${name} ./${name})

  add_dependencies(check ${name})
  add_dependencies(tests ${name})
endfunction()

Both cmake and Boost.Test support adding all of your unit tests into a single, monolithic executable and then running tests as a whole or individually. I have found it easier to use the old paradigm of keeping each suite of tests separate, or at least separate them by related modules.

Our first unit test

This example test won’t actually test any code. It instead just shows some of the features of Boost.Test that you’ll use in writing real tests, or that you’ll see here in this blog or attached code. I’ll just dump the whole code on you now and explain after:

#define BOOST_TEST_DYN_LINK
#define BOOST_TEST_MODULE helloworld
#include <boost/test/unit_test.hpp>

void fun() { throw std::runtime_error(""); }

BOOST_AUTO_TEST_CASE(what)
{
    // BOOST_FAIL("hello");

    BOOST_CHECK(0 == 0);
    BOOST_CHECK_EQUAL(0,0);
    BOOST_CHECK_NE(1,0);
    BOOST_CHECK_THROW(fun(), std::exception);

    std::vector<int> ints{3,2,1};

    BOOST_REQUIRE_EQUAL(ints.size(), 3U);
    BOOST_CHECK_EQUAL(ints[1], 2);
}

Lines 1-3 are the minimum you need for a unit test program. If you want to make a monolithic executable spanning multiple cpp files then this is all you’ll put in main.cpp. Otherwise you put it as the base for each test cpp file. Line #1 is some sort of magic that you need if you’re using dynamic libraries; leave it out if you are not. Line #2 defines the name of the top level test suite and also tells the preprocessor to generate the main function–the rules to do so are in the unit test headers. Line #3 includes the unit test framework so you can then use the various macros and other constructs to create unit tests and test suites.

Line #5 defines a function that throws an exception. It is there for illustration only.

Line #7 starts a new unit test function. The BOOST_AUTO_TEST_CASE macro creates and names a new test case and automatically registers it with the test framework. It does a bunch of stuff you can do by hand but I’ve found little reason not to use the macro. It defines a function so is followed by a brace enclosed block of code.

Line #9 is a good first version of any unit test. This ensures that your test is actually being compiled and run. The automated registration of tests that Boost.Test creates for us makes that less likely, but you might have forgotten to add the whole file to your build system. If you then do a bunch of coding assuming your tests are passing because you’re not seeing any failures…well that’s an ungood situation.

Lines 11-14 are a few examples of the common test macros you will use to assert behavior:

  • BOOST_CHECK: passes if the boolean expression yields true. If it fails you get the text of the test in the failure message.
  • BOOST_CHECK_EQUAL: passes if the two operands compare equal using ‘==’. If it fails you get text including the values of both operands. This is very useful but requires that both operands are streamable (can apply << to them).
  • BOOST_CHECK_NE: passes if the two operands compare an non-equal. It behaves similarly to the equal version.
  • BOOST_CHECK_THROW: passes if the expression given as the first operand results in an exception being thrown that is the same type or a subtype of the second operand.

Lines 16-19 illustrate how you protect your tests from crashing. All ‘check’ macros in Boost.Test allow the test to proceed, reporting the error but not aborting. If you naively checked the size of the vector ints with a check macro then the next check, where you retrieve a value within the vector, would possibly cause a crash. You can replace ‘CHECK’ with ‘REQUIRE’ in any of the boost macros to get a version that will abort any further tests in the current case if the check fails.

Test run

To run these tests you first initialize your build environment. I usually just make build directories in the source root, but you can actually make them anywhere; you just have to give the right path argument to cmake. So to make a debug build I do:

$ mkdir debug ; cd debug
$ cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_COMPILER=clang++ ..
$ make check

The second and third generate a bunch of text including the cmake analysis and the compilation. After all that is done though the tests are run via the check target and you’ll get something like this:

[100%] Built target hello_world
Test project /home/eddie/github/sanecpp/20140810-unittest/debug/test
    Start 1: hello_world
1/1 Test #1: hello_world ......................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.01 sec
[100%] Built target check

Conclusion

That’s it for now. This has been a very, VERY rudamentary introduction but you can actually use these tools to do quite a bit without needing to know a lot more. I encourage you to pull the code and play with it. You can use the CMakeLists.txt files as a basis for your own work if you’d like.