Use C++ testing framework

Richard Kimberly Heck rikiheck at lyx.org
Wed Jan 13 17:18:52 UTC 2021


On 1/13/21 11:54 AM, Scott Kostyshak wrote:
> On Wed, Jan 13, 2021 at 11:53:25AM +0200, Yuriy Skalko wrote:
>> On 16.12.2020 22:14, Yuriy Skalko wrote:
>>> There were already several discussions here in 2013, 2015 and even in
>>> 1999 about using some unit testing framework, but without any tangible
>>> result. I think it is now time to change this :)
>>>
>>> There are several widely used frameworks, I've chosen "doctest"
>>> (https://github.com/onqtam/doctest/). It is not the most popular, but is
>>> the fasest and has small size -- it consists of only one file. Also it
>>> is very convenient to use
>>> (https://github.com/onqtam/doctest/blob/master/doc/markdown/tutorial.md).
>>>
>>> Now LyX has some simple hand-made unit testing in `tests` directories
>>> that is used to test a few base classes/functions. I've converted some
>>> of these tests to the new framework.
>>>
>>> I only updated/added CMake files and these surely will require further
>>> improvements.
>>>
>>> So, what is your opinion about this change? Comments are very appreciated.
>>>
>>>
>>> Yuriy
>> Ping...
> Thanks for continuing this conversation, Yuriy.
>
> I'm strongly in favor of a unit testing framework. That said, I just don't think most LyX developers are. I don't know if it's because writing tests is not fun or if the belief is that the time spent to write tests would be better spent working directly on bugs/features. Even if it's the "fun" argument, I think that's understandable. People contribute to LyX in their own free time. I often choose what to work on based on what I find fun. In any case, I should also say that I don't have experience with unit tests so I cannot make any substantive claim based on my own experience.
>
> I don't know if it's worth going forward without the support of most developers. I'm not sure at all in what I'm about to say (given my lack of experience), but I feel like writing unit tests is the sort of thing where each developer kind of thinks along the lines of "OK I don't like writing tests, no one *enjoys* writing tests... but if you do it I'll do it." And if every developer takes that approach, and we hold each other accountable, over time we get more coverage and the code base becomes more robust. Indeed I see a lot of regressions on master that I wonder "hmmm I bet we could have had a test that would have caught that." Most of the time these regressions are short-lived (but even these are annoying and time-consuming), but some of them go unnoticed for a while.
>
> If just a few developers write tests, then I think there might be more frustration than benefit. I will document below the situation with ctests. I think that ctests are very different (especially the first point below :)) from the potential unit test framework, but perhaps the description will be useful to you in some way.
>
> I see 6 reasons why developers aren't interested in the ctests: (1) the ctests are not unit tests; (2) to understand the full suite of our ctests is complex and non-standard and would take developers a lot of time. We tried to document things in Development.lyx, but it is still confusing (I'm willing to clear up any confusion in the document if anyone has a specific question). At least for commits that change LaTeX output, it would be really nice if the developer ran the ctests to see which fail that weren't failing before. LaTeX is so complex and there are so many interactions and things that cause unexpected failures in compilation that the ctests catch a lot of regressions. However, (3) running all of the ctests takes a long time and perhaps because of power bills people do not want to leave their computer running the tests overnight and push their commit in the morning. Kornel and I even offer to take care of running the ctests on a patch/branch before someone commits it to master, but I think developers just (4) don't want to wait or have any extra step in their commit chain. And I think that most developers (5) are fine with the idea of "eh let's just fix it on master, master is not meant to be stable anyway." Finally, (6) the ctests are tied to CMake and a lot of developers prefer autotools.
>
> I hope that developers will respond to this and say "The ctests suck. If they were simple unit tests then I would contribute to writing them and I would run them before I commit!". I will (seriously) be happy to see such messages.
>
> In any case, my main point is that I don't recommend going forward with the unit tests if just a few people are on board. That said, a few people *could* be sufficient. I would have abandoned the ctests long ago if Kornel were not also going through the same frustration that others break the tests and we have to report them and adapt the tests. It's really not fun. But it is useful, and other developers are usually very responsive and helpful in fixing regressions found by the ctests. Thus, if you ignore my warnings and decide you are willing to suffer, I guess I would jump in with you whenever I get more free time back (not for a while).

Regarding the ctests, I don't run them myself mostly out of ignorance.
But that shouldn't be confused with my not finding them helpful. It was
my general sense (perhaps wrongly) that, so long as *someone* is running
them, they are doing their job. So maybe I'm in the (5) camp (with a bit
of (4) added in): The ctests will catch problems, but those are rare
enough that it's not necessary to run them before making a commit. So
just let me be clear that I, and I think the other developers,
appreciate the work that you and Kornel put into this. I'll try to be
less grumpy when a test fails!

As far as unit tests are concerned, I have even less experience with
those, so I'm not really in a position to have a substantive view. But
from what little reading I've done, I'm sure they would be useful if the
Coding Fairies would write them for us. And certainly, if someone
(Yuriy) wants to expend the effort to get us started, then I'm certainly
not opposed to that. But I agree that this would likely end up like the
ctests are now: One or two people maintaining them and running them. But
maybe if they proved their value, then others of us would start writing
tests, too. Who knows?

Riki





More information about the lyx-devel mailing list