ctests: How to protect against tests accidentally being ignored? i.e., ctest -N changing

[Thibaut Cuvelier]

On Thu, 26 Nov 2020 at 22:44, Scott Kostyshak <skostysh at lyx.org> wrote:

> On Thu, Nov 26, 2020 at 07:19:55PM +0100, Kornel Benko wrote:
> > Am Thu, 26 Nov 2020 19:11:14 +0100
> > schrieb Kornel Benko <kornel at lyx.org>:
> >
> > > Am Thu, 26 Nov 2020 13:03:41 -0500
> > > schrieb Scott Kostyshak <skostysh at lyx.org>:
> > >
> > > > On Thu, Nov 26, 2020 at 05:32:41PM +0000, José Abílio Matos wrote:
> > > > > On Thursday, November 26, 2020 5:30:10 PM WET Kornel Benko wrote:
>
> > > > > > We could output (at configure time) how many tests are ignored.
> > > > > >
> > > > > >         Kornel
> > > > >
> > > > > That seems a nice first step. :-)
> > > >
> > > > Good idea.
> > > >
> > > > A further step might be to assert that the number of ignored tests
> > > > equals a saved number. Whenever we updated the ignored tests, we
> need to
> > > > update the number. i.e., if I add a regex to ignoredTests that I
> expect
> > > > will ignore 5 tests, I increase the number by 5.
> > > >
> > > > What are your thoughts on that? The disadvantage to this workflow is
> > > > that if we do not remember to update the number, the CMake command
> would
> > > > fail. This could be annoying. I think it would be worth the annoyance
> > > > and that we would get used to remembering to update the number of
> > > > ignored tests, but I would understand if others disagreed.
> > > >
> > > > Scott
> > >
> > > Possible, and not even difficult. The saved value would reside in the
> 'Testing'
> > > directory. We could also accumulate this file with entries like
> date-stamp or commit.
> > >
> > >     Kornel
> >
> > I misread your mail. What I meant was
> > 1.) Protocol the date, commit, and number of ignored tests in a file
> > so we can always check when the count made a big step forward.
>
> Ah I see. So there would be no automatic checking/enforcement. In that
> case, instead of having a separate file, why not put a comment above each
> ignored pattern for how many tests it is expected to ignore? e.g.,
>
>   # nignored:5
>
> or
>
>   # nignored=5
>
> Then we could make a different script that parses those and sums up the
> number so we can easily check whether it equals the actual number of tests
> ignored.
>

The problem with a script that you *can* run, but that is not automatically
run, is that people will forget it at times (and I'll probably be the first
to!). Maybe such a script could be run as part of CMake configuration?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20201126/ac81e97b/attachment-0001.html>