ctests: How to protect against tests accidentally being ignored? i.e., ctest -N changing

[Scott Kostyshak]

On Fri, Nov 27, 2020 at 12:56:42AM +0100, Kornel Benko wrote:
> Am Thu, 26 Nov 2020 23:55:57 +0100
> schrieb Thibaut Cuvelier <tcuvelier at lyx.org>:
> 
> > On Thu, 26 Nov 2020 at 22:44, Scott Kostyshak <skostysh at lyx.org> wrote:
> > 
> > > On Thu, Nov 26, 2020 at 07:19:55PM +0100, Kornel Benko wrote:
> > > > Am Thu, 26 Nov 2020 19:11:14 +0100
> > > > schrieb Kornel Benko <kornel at lyx.org>:
> > > >
> > > > > Am Thu, 26 Nov 2020 13:03:41 -0500
> > > > > schrieb Scott Kostyshak <skostysh at lyx.org>:
> > > > >
> > > > > > On Thu, Nov 26, 2020 at 05:32:41PM +0000, José Abílio Matos wrote:
> > > > > > > On Thursday, November 26, 2020 5:30:10 PM WET Kornel Benko wrote:
> > >
> > > > > > > > We could output (at configure time) how many tests are ignored.
> > > > > > > >
> > > > > > > >         Kornel
> > > > > > >
> > > > > > > That seems a nice first step. :-)
> > > > > >
> > > > > > Good idea.
> > > > > >
> > > > > > A further step might be to assert that the number of ignored tests
> > > > > > equals a saved number. Whenever we updated the ignored tests, we
> > > need to
> > > > > > update the number. i.e., if I add a regex to ignoredTests that I
> > > expect
> > > > > > will ignore 5 tests, I increase the number by 5.
> > > > > >
> > > > > > What are your thoughts on that? The disadvantage to this workflow is
> > > > > > that if we do not remember to update the number, the CMake command
> > > would
> > > > > > fail. This could be annoying. I think it would be worth the annoyance
> > > > > > and that we would get used to remembering to update the number of
> > > > > > ignored tests, but I would understand if others disagreed.
> > > > > >
> > > > > > Scott
> > > > >
> > > > > Possible, and not even difficult. The saved value would reside in the
> > > 'Testing'
> > > > > directory. We could also accumulate this file with entries like
> > > date-stamp or commit.
> > > > >
> > > > >     Kornel
> > > >
> > > > I misread your mail. What I meant was
> > > > 1.) Protocol the date, commit, and number of ignored tests in a file
> > > > so we can always check when the count made a big step forward.
> > >
> > > Ah I see. So there would be no automatic checking/enforcement. In that
> > > case, instead of having a separate file, why not put a comment above each
> > > ignored pattern for how many tests it is expected to ignore? e.g.,
> > >
> > >   # nignored:5
> > >
> > > or
> > >
> > >   # nignored=5
> > >
> > > Then we could make a different script that parses those and sums up the
> > > number so we can easily check whether it equals the actual number of tests
> > > ignored.
> > >
> > 
> > The problem with a script that you *can* run, but that is not automatically
> > run, is that people will forget it at times (and I'll probably be the first
> > to!).
> 
> Me too.
> 
> > Maybe such a script could be run as part of CMake configuration?
> 
> This looks very error prone. Mark that removing a file may dismiss 10 (or more) tests
> (dvi*, pdf*, lyx2*, xhtml, docbook5, ... etc)

Ah good point about when we remove a file. Same for adding a file.

> Search for this would not be easy.
> Committed a first version at 7e5b1d09.
> 
> The file with infos is Testing/IgnoredCount

Thanks for that quick commit! Indeed the number is larger than I would
have expected. I think it is because we ignore some large classes of
tests.

Scott
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20201127/74ec2c1b/attachment.asc>