Use `code-coverage.cmake` from `cmake-scripts`

[mscofield0]

Currently, the template just applies -fprofile-arcs -ftest-coverage to the Greeter target from test. As far as I could tell, it doesn't really generate the code coverage data locally, and I think it'd be useful to be able to do that.

The code-coverage.cmake script exposes targets that we can run if we want to generate the data locally too. It also does checks that the current ModernCppStarter solution doesn't.

[mscofield0]

I've been trying for an hour to get it to work and it seems target_code_coverage() just doesn't find Greeter...
Worth noting that it works with GreeterTests.

Maybe you will have more luck?

Okay I figured out why the Greeter target doesn't work. It's because target_code_coverage() only generates the ccov-* targets if the passed-in target is an executable one.

What is even weirder is that when generating the code coverage data, it includes doctest in the report, but not Greeter.

[mscofield0]

Okay, I don't know how I didn't see this before. It's so simple. To instrument a library target and an executable one, just call target_code_coverage() on both. The library one won't get its custom ccov-* targets, but the executable will. The report that cmake --build build/test --target ccov now generates includes the greeter.cpp file.

Although it also imports fmt now along with doctest. I'll try to figure out which directory to exclude so that it doesn't show up anymore.

[TheLartians]

Currently, the template just applies -fprofile-arcs -ftest-coverage to the Greeter target from test. As far as I could tell, it doesn't really generate the code coverage data locally, and I think it'd be useful to be able to do that.

It should though, e.g. using Makefiles on Mac, the code coverage data is stored in <build_dir>/CMakeFiles/GreeterTests.dir/. This is generated by simply calling the executable, e.g. after

cmake -Htest -Bbuild/test -DENABLE_TEST_COVERAGE=1
cmake --build build/test
./build/test/GreeterTests

To summarise the data you could simply call gcovr.

> gcovr
------------------------------------------------------------------------------
                           GCC Code Coverage Report
Directory: .
------------------------------------------------------------------------------
File                                       Lines    Exec  Cover   Missing
------------------------------------------------------------------------------
source/greeter.cpp                             8       8   100%   
------------------------------------------------------------------------------
TOTAL                                          8       8   100%
------------------------------------------------------------------------------

TBH I don't do much local analysis myself and rely on platforms such as codecov.io for a nice visualisation. But all the data should be there and you should be able to easily generate the html reports locally.

I'm not really sure about the advantage of adding additional abstraction for the coverage, as the flags are straightforward to use and give most user control without additional required code.

To read more about gcov and how the flags work, see the wikipedia entry.

[mscofield0]

As far as I see it, the current solution doesn't support llvm-cov? Also, the script makes it really simple and automate by exposing ccov, ccov-all and ccov-SpecificTarget targets. It already comes with the cmake-scripts dependency so it really isn't adding too much complexity. I made sure the previous option ENABLE_TEST_COVERAGE remains there.

[acgetchell]

This doesn't seem to work anymore:

cmake -Htest -Bbuild/test -DENABLE_TEST_COVERAGE=1
-- CPM: Adding package doctest@2.4.9 (v2.4.9)
-- CPM: Adding package Format.cmake@1.7.3 (v1.7.3)
-- CPM: Adding package CDT@ (/Users/adam/projects/cdt/test/..)
-- CPM: CDT: Adding package PackageProject.cmake@1.8.0 (v1.8.0)
-- CPM: CDT: Adding package fmt@9.1.0 (9.1.0)
-- Module support is disabled.
-- Version: 9.1.0
-- Build type:
-- CXX_STANDARD: 11
-- Required features: cxx_variadic_templates
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/adam/projects/cdt/build/test
cmake --build build/test
[ 37%] Built target fmt
[ 62%] Built target CDT
[100%] Built target CDTTests
./build/test/CDTTests
[doctest] doctest version is "2.4.9"
[doctest] run with "--help" for options
===============================================================================
[doctest] test cases: 2 | 2 passed | 0 failed | 0 skipped
[doctest] assertions: 5 | 5 passed | 0 failed |
[doctest] Status: SUCCESS!

gcovr
zsh: correct 'gcovr' to 'gcov' [nyae]? n
------------------------------------------------------------------------------
                           GCC Code Coverage Report
Directory: .
------------------------------------------------------------------------------
File                                       Lines    Exec  Cover   Missing
------------------------------------------------------------------------------
build/test/_deps/doctest-src/doctest/doctest.h
                                            1995     679    34%   829,831,835,862,1024-1025,1029,1032,1036-1039,1045-1046,1078-1079,1123,1127,1152-1153,1256-1257,1300,1316,1318,1794,1801,1830,3238,3240,3283,3289-3291,3294-3296,3298-3302,3306-3307,3310-3311,3503,3506-3507,3510-3513,3516,3518-3521,3532,3563-3565,3593-3594,3602,3610-3615,3617,3619-3620,3622,3624,3626-3629,3631,3633-3634,3636-3638,3640-3642,3644,3646,3648,3650-3651,3653,3657,3669,3677-3678,3681-3684,3691,3695-3698,3701-3706,3709-3711,3714-3720,3723-3728,3741,3743-3744,3746-3748,3750,3754-3755,3758-3759,3762-3765,3779,3786-3789,3791,3793,3795,3797,3799,3801,3803-3808,3810-3811,3813,3819-3826,3832,3834-3840,3844,3849-3852,3855-3860,3863,3866-3868,3872-3875,3888,3890,3892,3894-3896,3898-3908,3915-3919,3926-3928,3936-3946,3948-3949,3955-3956,3963-3966,4034-4036,4038-4042,4044,4048-4050,4075-4077,4079-4087,4092,4108-4109,4113-4114,4116-4118,4121-4122,4125-4129,4131,4134-4137,4139,4143-4148,4150,4153-4157,4159,4161-4164,4166-4167,4169-4175,4177-4180,4189-4191,4193,4195-4198,4200,4204,4208-4209,4214,4217,4225,4273,4282,4284,4293-4299,4316,4319,4323-4327,4331-4335,4350,4352,4355-4359,4414-4416,4419,4421-4425,4429-4437,4466-4467,4473-4476,4478-4479,4482,4518-4521,4526-4527,4530-4532,4534-4535,4545,4547,4551-4553,4555,4746-4752,4755-4757,4823-4825,4829-4830,4832,4834-4836,4839,4841,4843,4864,4866,4873-4875,4880,4882,4884,4886,4888,4892,4899,4901,4905,4910,4913-4915,4917,4920,4922,4928-4930,4933-4937,4940-4942,4945,4947-4950,4953,4955,4958-4960,4963-4964,4967-4969,5012-5014,5040-5043,5062-5063,5078-5080,5082-5083,5085-5086,5088,5091-5093,5095-5096,5098-5099,5101,5104-5109,5114-5117,5119,5123-5127,5131-5132,5134-5135,5138-5139,5141-5142,5149-5151,5155-5157,5166-5169,5172,5174-5176,5181-5186,5191,5193-5195,5197,5199-5200,5204-5205,5207-5208,5213-5215,5218-5220,5236-5238,5241-5243,5246,5251-5253,5256-5263,5266-5269,5272-5277,5280,5282-5284,5287-5290,5293-5296,5299-5301,5304-5311,5313,5333-5336,5340-5341,5344-5347,5364,5366-5368,5370-5378,5383,5385-5390,5394,5397-5399,5402-5407,5409-5414,5421-5438,5440-5448,5450,5453-5454,5457,5463-5465,5468-5477,5480-5482,5484-5486,5488-5494,5496,5499-5501,5504,5506-5516,5518,5521-5522,5524-5526,5529-5534,5537,5539-5541,5543,5545-5549,5551,5553-5554,5556-5561,5563,5565,5568-5569,5571-5574,5576,5578,5580,5583-5587,5594-5595,5597-5598,5600-5611,5613-5614,5616,5618-5620,5623-5624,5626-5629,5632-5640,5642,5658,5660,5664-5665,5671,5675,5677-5678,5683-5684,5686-5687,5694-5695,5702-5703,5706-5709,5712-5716,5719-5721,5724-5726,5730-5731,5738,5740-5742,5744,5750-5751,5754-5755,5758,5760,5765-5772,5774-5775,5777-5782,5784,5786-5790,5793-5796,5799,5801-5802,5805-5807,5810-5813,5815-5816,5819-5822,5825-5827,5830-5831,5834,5836-5838,5840,5842-5844,5846-5848,5851,5853,5855-5858,5860-5864,5875-5876,5879-5882,5915,5917-5919,5922-5924,5927,5929-5930,5933-5936,5938-5942,5946,5950,5952-5954,5957-5959,5961-5969,5971-5973,5976-5980,5984,5986,6003-6005,6007-6016,6018-6020,6022-6036,6038-6112,6116-6117,6120-6127,6129-6131,6140,6142,6144,6162-6163,6165-6166,6168,6170-6175,6186,6226-6227,6232,6238,6241-6242,6245-6246,6249,6251,6253,6255-6256,6258-6259,6262,6267-6270,6272,6274-6275,6277-6278,6280-6286,6289,6292-6295,6298-6300,6307,6309,6311-6312,6314,6316,6319-6321,6323,6325,6327-6332,6335,6388-6389,6395-6399,6401,6435-6439,6441-6442,6444-6453,6455,6457-6462,6466-6467,6469-6470,6488,6491-6494,6496,6498-6499,6502-6505,6507-6509,6513,6526,6530-6533,6632-6633,6637-6638,6642-6643,6652-6653,6657-6658,6663,6666-6668,6672-6673,6677-6678,6682-6685,6689,6691,6693,6695,6707,6709-6711,6736,6739-6741,6753,6765,6783,6792,6794,6807-6811,6814-6816,6818,6820-6821,6823,6843,6846,6848,6850,6852,6854,6856,6864,6867-6869,6874,6884-6887,6889,6935-6937,6939,6945-6947,6951,6964,6983-6985,6988-6990,6998
build/test/_deps/fmt-src/include/fmt/core.h
                                              52      52   100%
include/cdt/greeter.h                          0       0    --%
source/greeter.cpp                             8       8   100%
test/source/greeter.cpp                       10      10   100%
test/source/main.cpp                           0       0    --%
------------------------------------------------------------------------------
TOTAL                                       2065     749    36%
------------------------------------------------------------------------------

[ClausKlein]

@acgetchell try a gcovr config file too like this: https://github.com/ClausKlein/ModernCmakeStarter/blob/develop/gcovr.cfg

[acgetchell]

@ClausKlein That fixes it, thanks!