ctests: How to protect against tests accidentally being ignored? i.e., ctest -N changing

[Scott Kostyshak]

On Thu, Nov 26, 2020 at 07:19:55PM +0100, Kornel Benko wrote:
> Am Thu, 26 Nov 2020 19:11:14 +0100
> schrieb Kornel Benko <kornel at lyx.org>:
> 
> > Am Thu, 26 Nov 2020 13:03:41 -0500
> > schrieb Scott Kostyshak <skostysh at lyx.org>:
> > 
> > > On Thu, Nov 26, 2020 at 05:32:41PM +0000, José Abílio Matos wrote:  
> > > > On Thursday, November 26, 2020 5:30:10 PM WET Kornel Benko wrote:    
> > > > > We could output (at configure time) how many tests are ignored.
> > > > > 
> > > > >         Kornel    
> > > > 
> > > > That seems a nice first step. :-)    
> > > 
> > > Good idea.
> > > 
> > > A further step might be to assert that the number of ignored tests
> > > equals a saved number. Whenever we updated the ignored tests, we need to
> > > update the number. i.e., if I add a regex to ignoredTests that I expect
> > > will ignore 5 tests, I increase the number by 5.
> > > 
> > > What are your thoughts on that? The disadvantage to this workflow is
> > > that if we do not remember to update the number, the CMake command would
> > > fail. This could be annoying. I think it would be worth the annoyance
> > > and that we would get used to remembering to update the number of
> > > ignored tests, but I would understand if others disagreed.
> > > 
> > > Scott  
> > 
> > Possible, and not even difficult. The saved value would reside in the 'Testing'
> > directory. We could also accumulate this file with entries like date-stamp or commit.
> > 
> > 	Kornel
> 
> I misread your mail. What I meant was
> 1.) Protocol the date, commit, and number of ignored tests in a file
> so we can always check when the count made a big step forward.

Ah I see. So there would be no automatic checking/enforcement. In that case, instead of having a separate file, why not put a comment above each ignored pattern for how many tests it is expected to ignore? e.g.,

  # nignored:5

or

  # nignored=5

Then we could make a different script that parses those and sums up the number so we can easily check whether it equals the actual number of tests ignored.

Scott
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.lyx.org/pipermail/lyx-devel/attachments/20201126/a6b1b77b/attachment.asc>