ctests: How to protect against tests accidentally being ignored? i.e., ctest -N changing

[Kornel Benko]

Am Thu, 26 Nov 2020 23:55:57 +0100
schrieb Thibaut Cuvelier <tcuvelier at lyx.org>:

> On Thu, 26 Nov 2020 at 22:44, Scott Kostyshak <skostysh at lyx.org> wrote:
> 
> > On Thu, Nov 26, 2020 at 07:19:55PM +0100, Kornel Benko wrote:
> > > Am Thu, 26 Nov 2020 19:11:14 +0100
> > > schrieb Kornel Benko <kornel at lyx.org>:
> > >
> > > > Am Thu, 26 Nov 2020 13:03:41 -0500
> > > > schrieb Scott Kostyshak <skostysh at lyx.org>:
> > > >
> > > > > On Thu, Nov 26, 2020 at 05:32:41PM +0000, José Abílio Matos wrote:
> > > > > > On Thursday, November 26, 2020 5:30:10 PM WET Kornel Benko wrote:
> >
> > > > > > > We could output (at configure time) how many tests are ignored.
> > > > > > >
> > > > > > >         Kornel
> > > > > >
> > > > > > That seems a nice first step. :-)
> > > > >
> > > > > Good idea.
> > > > >
> > > > > A further step might be to assert that the number of ignored tests
> > > > > equals a saved number. Whenever we updated the ignored tests, we
> > need to
> > > > > update the number. i.e., if I add a regex to ignoredTests that I
> > expect
> > > > > will ignore 5 tests, I increase the number by 5.
> > > > >
> > > > > What are your thoughts on that? The disadvantage to this workflow is
> > > > > that if we do not remember to update the number, the CMake command
> > would
> > > > > fail. This could be annoying. I think it would be worth the annoyance
> > > > > and that we would get used to remembering to update the number of
> > > > > ignored tests, but I would understand if others disagreed.
> > > > >
> > > > > Scott
> > > >
> > > > Possible, and not even difficult. The saved value would reside in the
> > 'Testing'
> > > > directory. We could also accumulate this file with entries like
> > date-stamp or commit.
> > > >
> > > >     Kornel
> > >
> > > I misread your mail. What I meant was
> > > 1.) Protocol the date, commit, and number of ignored tests in a file
> > > so we can always check when the count made a big step forward.
> >
> > Ah I see. So there would be no automatic checking/enforcement. In that
> > case, instead of having a separate file, why not put a comment above each
> > ignored pattern for how many tests it is expected to ignore? e.g.,
> >
> >   # nignored:5
> >
> > or
> >
> >   # nignored=5
> >
> > Then we could make a different script that parses those and sums up the
> > number so we can easily check whether it equals the actual number of tests
> > ignored.
> >
> 
> The problem with a script that you *can* run, but that is not automatically
> run, is that people will forget it at times (and I'll probably be the first
> to!).

Me too.

> Maybe such a script could be run as part of CMake configuration?

This looks very error prone. Mark that removing a file may dismiss 10 (or more) tests
(dvi*, pdf*, lyx2*, xhtml, docbook5, ... etc)

Search for this would not be easy.
Committed a first version at 7e5b1d09.

The file with infos is Testing/IgnoredCount

	Kornel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: Digitale Signatur von OpenPGP
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20201127/a26a8e4c/attachment.asc>