Use C++ testing framework

Scott Kostyshak skostysh at lyx.org
Wed Jan 13 16:54:38 UTC 2021


On Wed, Jan 13, 2021 at 11:53:25AM +0200, Yuriy Skalko wrote:
> On 16.12.2020 22:14, Yuriy Skalko wrote:
> > There were already several discussions here in 2013, 2015 and even in
> > 1999 about using some unit testing framework, but without any tangible
> > result. I think it is now time to change this :)
> > 
> > There are several widely used frameworks, I've chosen "doctest"
> > (https://github.com/onqtam/doctest/). It is not the most popular, but is
> > the fasest and has small size -- it consists of only one file. Also it
> > is very convenient to use
> > (https://github.com/onqtam/doctest/blob/master/doc/markdown/tutorial.md).
> > 
> > Now LyX has some simple hand-made unit testing in `tests` directories
> > that is used to test a few base classes/functions. I've converted some
> > of these tests to the new framework.
> > 
> > I only updated/added CMake files and these surely will require further
> > improvements.
> > 
> > So, what is your opinion about this change? Comments are very appreciated.
> > 
> > 
> > Yuriy
> 
> Ping...

Thanks for continuing this conversation, Yuriy.

I'm strongly in favor of a unit testing framework. That said, I just don't think most LyX developers are. I don't know if it's because writing tests is not fun or if the belief is that the time spent to write tests would be better spent working directly on bugs/features. Even if it's the "fun" argument, I think that's understandable. People contribute to LyX in their own free time. I often choose what to work on based on what I find fun. In any case, I should also say that I don't have experience with unit tests so I cannot make any substantive claim based on my own experience.

I don't know if it's worth going forward without the support of most developers. I'm not sure at all in what I'm about to say (given my lack of experience), but I feel like writing unit tests is the sort of thing where each developer kind of thinks along the lines of "OK I don't like writing tests, no one *enjoys* writing tests... but if you do it I'll do it." And if every developer takes that approach, and we hold each other accountable, over time we get more coverage and the code base becomes more robust. Indeed I see a lot of regressions on master that I wonder "hmmm I bet we could have had a test that would have caught that." Most of the time these regressions are short-lived (but even these are annoying and time-consuming), but some of them go unnoticed for a while.

If just a few developers write tests, then I think there might be more frustration than benefit. I will document below the situation with ctests. I think that ctests are very different (especially the first point below :)) from the potential unit test framework, but perhaps the description will be useful to you in some way.

I see 6 reasons why developers aren't interested in the ctests: (1) the ctests are not unit tests; (2) to understand the full suite of our ctests is complex and non-standard and would take developers a lot of time. We tried to document things in Development.lyx, but it is still confusing (I'm willing to clear up any confusion in the document if anyone has a specific question). At least for commits that change LaTeX output, it would be really nice if the developer ran the ctests to see which fail that weren't failing before. LaTeX is so complex and there are so many interactions and things that cause unexpected failures in compilation that the ctests catch a lot of regressions. However, (3) running all of the ctests takes a long time and perhaps because of power bills people do not want to leave their computer running the tests overnight and push their commit in the morning. Kornel and I even offer to take care of running the ctests on a patch/branch before someone commits it to master, but I think developers just (4) don't want to wait or have any extra step in their commit chain. And I think that most developers (5) are fine with the idea of "eh let's just fix it on master, master is not meant to be stable anyway." Finally, (6) the ctests are tied to CMake and a lot of developers prefer autotools.

I hope that developers will respond to this and say "The ctests suck. If they were simple unit tests then I would contribute to writing them and I would run them before I commit!". I will (seriously) be happy to see such messages.

In any case, my main point is that I don't recommend going forward with the unit tests if just a few people are on board. That said, a few people *could* be sufficient. I would have abandoned the ctests long ago if Kornel were not also going through the same frustration that others break the tests and we have to report them and adapt the tests. It's really not fun. But it is useful, and other developers are usually very responsive and helpful in fixing regressions found by the ctests. Thus, if you ignore my warnings and decide you are willing to suffer, I guess I would jump in with you whenever I get more free time back (not for a while).

Scott
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20210113/6e4d2ed8/attachment.asc>


More information about the lyx-devel mailing list