Monday, July 11, 2016

TDD Tools for C++ – Unit Testing & Mocking Frameworks

Smile
Test driven development (TDD) is a big buzz word when it comes to software development best practices. It has been well received in .NET and Java world and almost all the IDEs out of the box support tools/plugins required to ease the task for these languages.Computer
However when it comes to C/C++ it is a little bit different story. Your favorite IDE may not support the necessary tools/plugins for the TDD out of the box. But in fact there is a plethora of tools and plugins for major IDEs that you can use for a seamless development experience. Since the choices of IDEs and the tools/frameworks falls in a wide range, I do not plan to explain here how to use each and every tool/plugin. In fact there is enough instructions out there if you know how to Google things up. Winking smileRather I would list down a couple of popular tools/frameworks you could use along with the notable differences between them so you could choose the tools best suitable for your requirements. Also note that this is not an exhaustive coverage on all type of tools but should be good enough to get the most of TDD. Also it should be noted that I do not plan on discussing on TDD approach in general either.
In TDD, first step is to write an initially failing test case to test improvements or new functions that you need to add. In doing this you need a good Unit testing framework as well as a Mocking framework to set the expectations on interfaces of objects you would be using.
Unit Testing Frameworks.
Google Test[1]
Google Test is a good unit testing framework and has gained popularity among C++ developers recently. In Windows, it has a test runner plugin for Visual Studio. Also in Linux, It has plugins for both Qt Creator as well as Eclipse CDT. Google Test also supports value parameterized tests as well[2]. Though it is not that much intuitive like defining Theories on xUnit++. Google Tests is also well maintained and actively supported.
You can read more on getting started with Google Test here: https://github.com/google/googletest/blob/master/googletest/docs/Primer.md
Boost Unit Testing Framework[3]
If you are already using Boost libraries, Boost Unit Testing Framework is another good choice for unit testing. It also has a plugin support for both Visual Studio as well as Eclipse CDT. Like Google Test, Boost Unit Testing Framework also has support for value parameterized testing starting from Boost.Test v3 (Library version 1.59.0). For the older versions you can use unary function based test cases[4] Even though it’s not that elegant like theories defined in xUnit++ or the newer BOOST_DATA_TEST_CASE declarations.
Even though it is not a requirement of a unit testing framework, apart from CppUTest, Boost Unit Testing framework also supports Memory Leak Detection through its execution monitor[5]. However this feature is, at the moment, implemented only for the Microsoft family of compilers (and Intel, if it employs Microsoft C Runtime Library). Also it can not be tuned per instance of the monitor and is only triggered globally and reported after the whole program execution is done. Like Other Boost libraries Boost.Test is also well maintained and supported.
xUnit++[6]
xUnit++ is a unit testing platform for C++ inspired by xUnit.net. If you are already familiar with xUnit.net you would feel home at defining new tests (Facts) and Theories in similar fashion of code syntax. However xUnit++ have only support for Visual Studio Test Explorer. Also it should be noted that it is compatible with Visual Studio 2012 and compatibility with other versions may vary. Even though it’s open source since XUnit++ is not actively maintained currently, support or additional features availability cannot be guaranteed.
Microsoft Unit Testing Framework for C++
If you are using Microsoft Visual Studio to build native applications on Windows, you could use Microsoft Unit Testing Framework for C++ which is bundled with Visual Studio. It supports integrated Test Explorer to manage unit tests you write. However it should be noted that it does not support Data Driven Tests (Substitute for value parameterized unit tests) like in Microsoft unit test framework for managed code. And since Microsoft Unit Testing Framework is a proprietary piece of software there is no visibility into maintenance and you’ll be mostly limited to community support.
You can read more on getting started with Microsoft Unit Testing Framework for C++ here. https://msdn.microsoft.com/en-us/library/hh598953.aspx
CppUTest[7]
CppUTest is another unit testing framework developed using a limited C++ set and thus well  suitable to be used in embedded systems. But it works for any C/C++ project. It also has a Test Runner plugin support for Eclipse CDT Test Runner[8].
Even though it is not a requirement of a unit testing framework, one notable feature in the CppUTest is the Memory Leak detection support on a per-test level[9]. In this regard, the developers have made it possible to use CppUTest along with Google Test[10] so that it is possible to have Memory Leak Detection support along with all the other goodies from Google Test so giving out the best of both worlds.
CppUTest also has a C Interface so that it is easy to use it with pure C projects[11]. Like many other open source unit testing frameworks CppUTest is also actively maintained and supported.
Mocking Frameworks
The key concept behind most C++ mocking frameworks is ability to override virtual functions in derived classes.
Google Mock[12]
If you are already using Google Tests, Google Mock is a good choice of mocking framework. Even though Google Mock is implemented on top of Google Test, and depends on it, you can also easily configure Google Mock to work with another testing framework[13]. Also it should be noted that, when using Google Mock, you must use the bundled version of Google Test if you choose to use Google Test.
You can read more on using Google Mock here: https://github.com/google/googletest/blob/master/googlemock/docs/ForDummies.md
One thing to note is that Google Mock resides in a place between manual mocking and automated mocking. Expectation settings and validations are automated, yet you have to write some amount of manual code compared to other mocking frameworks like HippoMocks or FakeIt. However since Google Mock is included in Google Test framework it is also actively maintained and supported.
HippoMocks[14]
HippoMocks is provided as a single header file. So it is no brainer to use it with any C++ unit testing framework out there. Just include the header and you are good to go. HippoMocks is occasionally maintained and supported on github.
You can read more on using HippoMocks here. http://hippomocks.com/Main_Page
FakeIt[15]
Like HippoMocks, FakeIt is also provided as a single header file. However there is a couple of additional features that you might sometimes find it useful. Specially the ability to Spy on existing objects using a single line of code[16]. It also has very expressive Arrange-Act-Assert syntax. Similar to HippoMocks, FakeIt is also occasionally maintained and supported.
You can read more on using FakeIt here. https://github.com/eranpeer/FakeIt/wiki/Quickstart
CppUMock[17]
CppUTest also has support for building mocks. However unlike other mocking frameworks, the main idea is to make manual mocking easier and it does not support making automated mocks like other frameworks. Thus with regards to this, CppUTest supports using Google Mock directly in CppUTest[18]. Also since CppUMock is included in CppUTest it is also actively maintained and supported.
Test Runner Plugins for IDEs
Eclipse CDT
Eclipse CDT has C/C++ Unit Testing Support plugin available from marketplace which supports both Boost Test Runner and Google Test Runner.
Note: It may require to separately install “C/C++ Unit Testing Support” package under “Programming Languages” category by choosing “--All Available Sites” in Help->Install New Software in Eclipse CDT.
Apart from that if you are using CppUTest you may also install CppUTest Tests Runner from https://github.com/tcmak/CppUTestEclipseJunoTestRunner However at the moment it is not actively maintained so support for newer versions or bug fixes cannot be guaranteed.
Qt Creator
Qt Creator has a good plugin for Google Test.
Qt Creator Google Test Plugin: https://sourceforge.net/projects/qtc-gtest/ It is also regularly maintained for new Qt Creator releases.
Microsoft Visual Studio
Apart from Microsoft Unit Testing Framework for C++, there are plugins for Microsoft Visual Studio Test Explorer for xUnit++, Google Test framework as well as Boost Unit Test Framework. Some notable plugins are listed below.
GoogleTest Runner plugin for VS 2015: https://visualstudiogallery.msdn.microsoft.com/cbd1a9b2-790b-4d49-905f-d128ca21d3fb It also have occasional support and maintenance on github.
Boost Unit Test Adapter for Visual Studio: https://visualstudiogallery.msdn.microsoft.com/5f4ae1bd-b769-410e-8238-fb30beda987f It is also actively maintained and supported
xUnit++ Test Adapter for Visual Studio 2012: https://visualstudiogallery.msdn.microsoft.com/1212037b-a608-450c-a578-627b93f1b336 Since this plugin is included in xUnit++ which is not actively maintained or supported, support for newer versions cannot be guaranteed.
If above still has not catered to your requirement, you can find an exhaustive list of unit testing and mocking frameworks at https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B Also if you have any questions or think that anything else should be added please feel free to leave a comment below. I would also like to thank my colleague Ruvinda Dambarage for useful tips provided on compiling this post.
References
[1] Google Test: https://github.com/google/googletest
[2] Google Test Value Parameterized Tests: https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#value-parameterized-tests
[3] Boost Test Library : The Unit Test Framework: http://www.boost.org/doc/libs/1_61_0/libs/test/doc/html/index.html
[4] Unary function based test case: http://www.boost.org/doc/libs/1_56_0/libs/test/doc/html/utf/user-guide/test-organization/unary-test-case.html
[5] Boost Test Library Memory Leaks Detection: http://www.boost.org/doc/libs/1_58_0/libs/test/doc/html/execution-monitor/user-guide.html
[6] xUnit++: https://bitbucket.org/moswald/xunit/wiki/Home
[7] CppUTest https://cpputest.github.io/
[8] CppUTest Tests Runner: https://github.com/tcmak/CppUTestEclipseJunoTestRunner
[9] Memory Leak Detection on CppUTest https://cpputest.github.io/manual.html#memory_leak_detection
[10] Running Google Tests in CppUTest https://cpputest.github.io/manual.html#gtest
[11] CppUTest C Interface: https://cpputest.github.io/manual.html#c_interface
[12] Google Mock https://github.com/google/googletest/tree/master/googlemock
[13] Using Google Mock with Any Testing Framework: https://github.com/google/googletest/blob/master/googlemock/docs/ForDummies.md#using-google-mock-with-any-testing-framework
[14] HippoMocks https://github.com/dascandy/hippomocks
[15] FakeIt https://github.com/eranpeer/FakeIt
[16] Spying: https://github.com/eranpeer/FakeIt/wiki/Quickstart#spying
[17] CppUMock https://cpputest.github.io/mocking_manual.html
[18] Using Google Mock with CppUTest https://cpputest.github.io/manual.html#gmock
[Published with Open Live Writer on Windows 10]

1 comment:

Peter Sommerlad said...

The overview misses CUTE C++ Unit Testing Framework (www.cute-test.com) and Mockator (www.mockator.com) both integrated into the free Eclipse-based C++ IDE Cevelop (www.cevelop.com)