Use C++ testing framework

Yuriy Skalko yuriy.skalko at gmail.com
Thu Jan 14 23:24:10 UTC 2021


> Thanks for continuing this conversation, Yuriy.

Scott, thanks for such detailed reply.


> I'm strongly in favor of a unit testing framework. That said, I just don't 
> think most LyX developers are. I don't know if it's because writing tests is 
> not fun or if the belief is that the time spent to write tests would be better 
> spent working directly on bugs/features. Even if it's the "fun" argument, I 
> think that's understandable. People contribute to LyX in their own free time. I 
> often choose what to work on based on what I find fun. In any case, I should 
> also say that I don't have experience with unit tests so I cannot make any 
> substantive claim based on my own experience.

I also believe that unit tests would be very useful. They can provide 
some kind of safety net to save already working features.


> I don't know if it's worth going forward without the support of most 
> developers. I'm not sure at all in what I'm about to say (given my lack of 
> experience), but I feel like writing unit tests is the sort of thing where each 
> developer kind of thinks along the lines of "OK I don't like writing tests, no 
> one *enjoys* writing tests... but if you do it I'll do it." And if every 
> developer takes that approach, and we hold each other accountable, over time we 
> get more coverage and the code base becomes more robust. Indeed I see a lot of 
> regressions on master that I wonder "hmmm I bet we could have had a test that 
> would have caught that." Most of the time these regressions are short-lived 
> (but even these are annoying and time-consuming), but some of them go unnoticed 
> for a while.
> 
> If just a few developers write tests, then I think there might be more 
> frustration than benefit. I will document below the situation with ctests. I 
> think that ctests are very different (especially the first point below :)) from 
> the potential unit test framework, but perhaps the description will be useful 
> to you in some way.
> 
> I see 6 reasons why developers aren't interested in the ctests: (1) the ctests 
> are not unit tests; (2) to understand the full suite of our ctests is complex 
> and non-standard and would take developers a lot of time. We tried to document 
> things in Development.lyx, but it is still confusing (I'm willing to clear up 
> any confusion in the document if anyone has a specific question). At least for 
> commits that change LaTeX output, it would be really nice if the developer ran 
> the ctests to see which fail that weren't failing before. LaTeX is so complex 
> and there are so many interactions and things that cause unexpected failures in 
> compilation that the ctests catch a lot of regressions. However, (3) running 
> all of the ctests takes a long time and perhaps because of power bills people 
> do not want to leave their computer running the tests overnight and push their 
> commit in the morning. Kornel and I even offer to take care of running the 
> ctests on a patch/branch before someone commits it to master, but I think 
> developers just (4) don't want to wait or have any extra step in their commit 
> chain. And I think that most developers (5) are fine with the idea of "eh let's 
> just fix it on master, master is not meant to be stable anyway." Finally, (6) 
> the ctests are tied to CMake and a lot of developers prefer autotools.

Yes, current ctests that used mostly for testing export outputs as are 
very different. I think the main issue that they are not popular is the 
long time to run them. I'm sure if ctests could complete in several 
seconds/minutes they would be run much frequently. But I presume there 
are not much possibilities to accelerate them, running LaTeX is time 
consuming task.

So we should learn from ctests adoption and try to do better for unit tests:

1. It should be simple to write and fast to execute many (hundreds) tests.

So no more separate executables for every testcase, shell scripts for 
checking the outputs, data files with correct results (as in current LyX 
unit tests in `tests` directories).

I think the best solution will be integrating tests into main LyX 
executable in debug mode (or even in release mode, depending on build 
system option) and providing --test command-line option to run them. The 
test runner of the framework is flexible and allows to choose what tests 
to run, so more heavy tests can be skipped if necessary. All tests will 
be located in src/test directory and linked into executable only in 
debug mode.

Doctest framework even suggests to write unit tests directly in the same 
file as tested function/class. But I think this approach is too radical 
for the LyX project.

2. We should try to get along without mocks/dummy_functions since LyX 
doesn't use potentially time-consuming external dependencies like 
databases and networking. But if tests are part of the full LyX 
executable, it simplifies this task.

(This approach is somewhat different than my first try in initial 
message in this thread. But the updated patch is largely ready.)

> 
> I hope that developers will respond to this and say "The ctests suck. If they 
> were simple unit tests then I would contribute to writing them and I would run 
> them before I commit!". I will (seriously) be happy to see such messages.
> 
> In any case, my main point is that I don't recommend going forward with the 
> unit tests if just a few people are on board. That said, a few people *could* 
> be sufficient. I would have abandoned the ctests long ago if Kornel were not 
> also going through the same frustration that others break the tests and we have 
> to report them and adapt the tests. It's really not fun. But it is useful, and 
> other developers are usually very responsive and helpful in fixing regressions 
> found by the ctests. Thus, if you ignore my warnings and decide you are willing 
> to suffer, I guess I would jump in with you whenever I get more free time back 
> (not for a while).
> 
> Scott


Thanks for the support, Scott. If there will be no objections from other 
developers this can start after 2.4 release (or earlier, in feature branch?)


> As far as unit tests are concerned, I have even less experience with
> those, so I'm not really in a position to have a substantive view. But
> from what little reading I've done, I'm sure they would be useful if the
> Coding Fairies would write them for us. And certainly, if someone
> (Yuriy) wants to expend the effort to get us started, then I'm certainly
> not opposed to that. But I agree that this would likely end up like the
> ctests are now: One or two people maintaining them and running them. But
> maybe if they proved their value, then others of us would start writing
> tests, too. Who knows?
> 
> Riki


Even if only small number of people will maintain them, it still should 
be useful. Existing check_lstrings.cpp and check_convert.cpp (written 
long time ago) helped me to be confident that some my patches don't 
break things. But I hope that more people will get involved if it will 
be really easy to add new tests. I'm ready to spend efforts to make 
tests writing as smooth as possible.


Yuriy


More information about the lyx-devel mailing list