Use C++ testing framework

Scott Kostyshak skostysh at lyx.org
Wed Jan 13 17:34:40 UTC 2021


On Wed, Jan 13, 2021 at 12:18:52PM -0500, Richard Kimberly Heck wrote:
> On 1/13/21 11:54 AM, Scott Kostyshak wrote:
> > On Wed, Jan 13, 2021 at 11:53:25AM +0200, Yuriy Skalko wrote:
> >> On 16.12.2020 22:14, Yuriy Skalko wrote:
> >>> There were already several discussions here in 2013, 2015 and even in
> >>> 1999 about using some unit testing framework, but without any tangible
> >>> result. I think it is now time to change this :)
> >>>
> >>> There are several widely used frameworks, I've chosen "doctest"
> >>> (https://github.com/onqtam/doctest/). It is not the most popular, but is
> >>> the fasest and has small size -- it consists of only one file. Also it
> >>> is very convenient to use
> >>> (https://github.com/onqtam/doctest/blob/master/doc/markdown/tutorial.md).
> >>>
> >>> Now LyX has some simple hand-made unit testing in `tests` directories
> >>> that is used to test a few base classes/functions. I've converted some
> >>> of these tests to the new framework.
> >>>
> >>> I only updated/added CMake files and these surely will require further
> >>> improvements.
> >>>
> >>> So, what is your opinion about this change? Comments are very appreciated.
> >>>
> >>>
> >>> Yuriy
> >> Ping...
> > Thanks for continuing this conversation, Yuriy.
> >
> > I'm strongly in favor of a unit testing framework. That said, I just don't think most LyX developers are. I don't know if it's because writing tests is not fun or if the belief is that the time spent to write tests would be better spent working directly on bugs/features. Even if it's the "fun" argument, I think that's understandable. People contribute to LyX in their own free time. I often choose what to work on based on what I find fun. In any case, I should also say that I don't have experience with unit tests so I cannot make any substantive claim based on my own experience.
> >
> > I don't know if it's worth going forward without the support of most developers. I'm not sure at all in what I'm about to say (given my lack of experience), but I feel like writing unit tests is the sort of thing where each developer kind of thinks along the lines of "OK I don't like writing tests, no one *enjoys* writing tests... but if you do it I'll do it." And if every developer takes that approach, and we hold each other accountable, over time we get more coverage and the code base becomes more robust. Indeed I see a lot of regressions on master that I wonder "hmmm I bet we could have had a test that would have caught that." Most of the time these regressions are short-lived (but even these are annoying and time-consuming), but some of them go unnoticed for a while.
> >
> > If just a few developers write tests, then I think there might be more frustration than benefit. I will document below the situation with ctests. I think that ctests are very different (especially the first point below :)) from the potential unit test framework, but perhaps the description will be useful to you in some way.
> >
> > I see 6 reasons why developers aren't interested in the ctests: (1) the ctests are not unit tests; (2) to understand the full suite of our ctests is complex and non-standard and would take developers a lot of time. We tried to document things in Development.lyx, but it is still confusing (I'm willing to clear up any confusion in the document if anyone has a specific question). At least for commits that change LaTeX output, it would be really nice if the developer ran the ctests to see which fail that weren't failing before. LaTeX is so complex and there are so many interactions and things that cause unexpected failures in compilation that the ctests catch a lot of regressions. However, (3) running all of the ctests takes a long time and perhaps because of power bills people do not want to leave their computer running the tests overnight and push their commit in the morning. Kornel and I even offer to take care of running the ctests on a patch/branch before someone commits i
>  t to master, but I think developers just (4) don't want to wait or have any extra step in their commit chain. And I think that most developers (5) are fine with the idea of "eh let's just fix it on master, master is not meant to be stable anyway." Finally, (6) the ctests are tied to CMake and a lot of developers prefer autotools.
> >
> > I hope that developers will respond to this and say "The ctests suck. If they were simple unit tests then I would contribute to writing them and I would run them before I commit!". I will (seriously) be happy to see such messages.
> >
> > In any case, my main point is that I don't recommend going forward with the unit tests if just a few people are on board. That said, a few people *could* be sufficient. I would have abandoned the ctests long ago if Kornel were not also going through the same frustration that others break the tests and we have to report them and adapt the tests. It's really not fun. But it is useful, and other developers are usually very responsive and helpful in fixing regressions found by the ctests. Thus, if you ignore my warnings and decide you are willing to suffer, I guess I would jump in with you whenever I get more free time back (not for a while).
> 
> Regarding the ctests, I don't run them myself mostly out of ignorance.
> But that shouldn't be confused with my not finding them helpful. It was
> my general sense (perhaps wrongly) that, so long as *someone* is running
> them, they are doing their job. So maybe I'm in the (5) camp (with a bit
> of (4) added in): The ctests will catch problems, but those are rare
> enough that it's not necessary to run them before making a commit.

Indeed the number of regressions as a proportion of the number of
commits is small. If on the denominator we only include commits that
change LaTeX output, I do not think it is that small.

> So
> just let me be clear that I, and I think the other developers,
> appreciate the work that you and Kornel put into this.

Thanks!

> I'll try to be
> less grumpy when a test fails!

You are not the grumpy one, I am :). I realize I should not be though.
Different things are important to different people, and that's what in
the end makes for a well-balanced development team. I'll try to be less
grumpy. I think also I have not searched enough for intermediate
solutions. I'll try an explicit attempt here:

  If anyone has a patch/branch that changes LaTeX output, I would be
  appreciative to have the chance to run the ctests and report whether
  there are any failures introduced before you commit to master. You
  should expect a one or two day turnaround in 99% of the cases, and for
  the other 1% of cases where I am traveling or sick, we could ask
  Kornel for the favor.

Scott

> As far as unit tests are concerned, I have even less experience with
> those, so I'm not really in a position to have a substantive view. But
> from what little reading I've done, I'm sure they would be useful if the
> Coding Fairies would write them for us. And certainly, if someone
> (Yuriy) wants to expend the effort to get us started, then I'm certainly
> not opposed to that. But I agree that this would likely end up like the
> ctests are now: One or two people maintaining them and running them. But
> maybe if they proved their value, then others of us would start writing
> tests, too. Who knows?
> 
> Riki
> 
> 
> 
> -- 
> lyx-devel mailing list
> lyx-devel at lists.lyx.org
> http://lists.lyx.org/mailman/listinfo/lyx-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20210113/593ae682/attachment.asc>


More information about the lyx-devel mailing list