[squeak-dev] Skipping tests (was: [CRON] Fixed: squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518))

Chris Muller asqueaker at gmail.com
Mon Apr 20 22:53:35 UTC 2020


When I find myself swimming in "solutions", I like to take a step back and
refocus on the problem for guidance.  What am I _really_ wanting, and why?
For me, it's about

   #1 -- a meaningful, overall *status universalis *(e.g., red, yellow,
green)
   #2 -- ability to defer broken tests until later so their individual
status won't affect the overall output of #1
        (2b -- see / edit / run the list of deferred tests)
   #3 -- "notification" of when a deferred test is fixed, but still in the
deferred list

It really boils down to making the "overall status" useful by deciding
which tests I want to decouple from it, combined with helping me keep that
list maintained in the similar way as it informs me about broken tests --
by flagging those tests in some way ("unexpected success" or whatever -- I
care a lot less about what words we use in the selectors as much as the
ability to meet the above requirements).

 - Chris

On Mon, Apr 20, 2020 at 1:22 PM Jakob Reschke <forums.jakob at resfarm.de>
wrote:

> JUnit has assumptions for the "not suitable" case:
> https://junit.org/junit5/docs/5.0.0/api/org/junit/jupiter/api/Assumptions.html
> That would match "platform-specific circumstance" and TestInconclusive
> <https://github.com/hpi-swa-teaching/SpreadSheetTool/tree/master/packages/SpreadSheetTool-Tests.package/TestInconclusive.class>
> .
>
> Then there is the other case to "ignore", "skip", or "disable" a certain
> test for the time being, for whatever reason.
> https://junit.org/junit5/docs/current/api/org.junit.jupiter.api/org/junit/jupiter/api/Disabled.html
>
> What about the other suggestion to use pragmas? I'd like that (more than
> listing selectors in another method). But I think it is only practical for
> the "disable" case because otherwise you would have to put the conditions
> for "suitable" in an expression in the pragma and would have to evaluate
> that in the runner ...
>
> Am Mo., 20. Apr. 2020 um 17:09 Uhr schrieb Marcel Taeumel <
> marcel.taeumel at hpi.de>:
>
>> Hmm... could be another can-o-worms. :-) Skipping is not the same as an
>> expected failure. Could as well be a platform-specific circumstance that
>> makes a test not suitable.
>>
>> Best,
>> Marcel
>>
>> Am 18.04.2020 15:28:33 schrieb Thiede, Christoph <
>> christoph.thiede at student.hpi.uni-potsdam.de>:
>>
>> By the way, this also reminds me of some previous ideas about skipping
>> tests in SUnit.
>>
>>
>> SqueakSheet contains an implementation of TestInconclusive
>> <https://github.com/hpi-swa-teaching/SpreadSheetTool/tree/master/packages/SpreadSheetTool-Tests.package/TestInconclusive.class> and
>> a <skip> pragma
>> <https://github.com/hpi-swa-teaching/SpreadSheetTool/blob/master/packages/SpreadSheetTool-Tests.package/TestCase.extension/class/wantsToTest..st>
>> .
>>
>> Leon chose a similar approach in the PragmaTestCase
>> <https://github.com/MrModder/PragmaTestCase> package.
>>
>> Maybe we should consider adding something similar to the Trunk?
>> Personally, I find it more convenient to tag expected failures or "test
>> that take too much time for the moment" directly at the relevant test case,
>> not in an extra #expectedFailures method.
>>
>> Best,
>> Christoph
>>
>> ------------------------------
>> *Von:* Squeak-dev <squeak-dev-bounces at lists.squeakfoundation.org> im
>> Auftrag von Thiede, Christoph
>> *Gesendet:* Samstag, 18. April 2020 15:22 Uhr
>> *An:* The general-purpose Squeak developers list
>> *Betreff:* Re: [squeak-dev] [CRON] Fixed:
>> squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518)
>>
>>
>> Thanks for your arguments! Yeah, we should really take CI more seriously,
>> a red bar shouldn't be something you see every day.
>>
>> > Note that we're already excluding some test cases:
>> >
>> https://github.com/squeak-smalltalk/squeak-app/blob/b063518c61711ab7a0ca89ffa3d2494925fac823/smalltalk-ci/Squeak64-trunk.ston#L8-L12
>>
>> Hm, for this purpose of "expected failures, don't run them at all" I
>> think it could be helpful if smalltalkCI printed these tests into the
>> output log.
>> Example:
>>
>> #########################
>> # 4 tests did not pass: #
>> #########################
>>
>> SocketTest
>>  ✗ #testSocketReuse (215ms)
>>  ✗ #testUDP (10007ms)
>> ...
>>
>> #########################
>> # 4 tests were skipped: #
>> #########################
>>
>> AllocationTest
>>  ⛔ #testOneGigAllocation
>>  ⛔ #testOneMegAllocation
>>  ⛔ #testOutOfMemorySignal
>> ...
>>
>>
>> What do you think? :-)
>>
>> Best,
>> Christoph
>>
>> ------------------------------
>> *Von:* Squeak-dev <squeak-dev-bounces at lists.squeakfoundation.org> im
>> Auftrag von Fabio Niephaus <lists at fniephaus.com>
>> *Gesendet:* Freitag, 17. April 2020 17:58 Uhr
>> *An:* The general-purpose Squeak developers list
>> *Betreff:* Re: [squeak-dev] [CRON] Fixed:
>> squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518)
>>
>> Note that we're already excluding some test cases:
>>
>> https://github.com/squeak-smalltalk/squeak-app/blob/b063518c61711ab7a0ca89ffa3d2494925fac823/smalltalk-ci/Squeak64-trunk.ston#L8-L12
>>
>> On Fri, Apr 17, 2020 at 5:57 PM Fabio Niephaus <lists at fniephaus.com>
>> wrote:
>>
>>> On Fri, Apr 17, 2020 at 5:53 PM Thiede, Christoph <
>>> Christoph.Thiede at student.hpi.uni-potsdam.de> wrote:
>>>
>>>> > Hmmm... in any case, we want to new bundles to appear on
>>>> files.squeak.org/trunk right?
>>>>
>>>>
>>>> Of course, but that does not mean that we have to handle the build as
>>>> successful, does it? We can also upload the new bundles and then let the
>>>> build fail.
>>>>
>>> Or we could fix (or ignore) the four failing tests and move on:
>>>
>>> [image: image.png]
>>>
>>> Fabio
>>>
>>>>
>>>> Best,
>>>>
>>>> Christoph
>>>> ------------------------------
>>>> *Von:* Squeak-dev <squeak-dev-bounces at lists.squeakfoundation.org> im
>>>> Auftrag von Taeumel, Marcel
>>>> *Gesendet:* Freitag, 17. April 2020 17:27:29
>>>> *An:* gettimothy via Squeak-dev
>>>> *Betreff:* Re: [squeak-dev] [CRON] Fixed:
>>>> squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518)
>>>>
>>>> Hmmm... in any case, we want to new bundles to appear on
>>>> files.squeak.org/trunk right? That's why it is called "ALPHA". :-)
>>>>
>>>> Best,
>>>> Marcel
>>>>
>>>> Am 17.04.2020 17:22:16 schrieb Thiede, Christoph <
>>>> christoph.thiede at student.hpi.uni-potsdam.de>:
>>>>
>>>> Sorry for the imprecise question. What I find a bit confusing is the
>>>> following: Not all tests are passing, but the Travis job still ends up with
>>>> exit code 0 so we get the mail "build was fixed", and the badge of the repo
>>>> becomes green. Wouldn't it be more intuitive if the mails sent to the list
>>>> would keep reporting "still failing" until we have a green bar in the Trunk?
>>>>
>>>>
>>>> This is the relevant place in the code (in the function
>>>> check_test_status):
>>>>
>>>> squeak-app/prepare_image.sh
>>>> <https://github.com/squeak-smalltalk/squeak-app/blob/b063518c61711ab7a0ca89ffa3d2494925fac823/prepare_image.sh#L90-L94>
>>>>
>>>> Lines 90 to 94 in b063518
>>>> <https://github.com/squeak-smalltalk/squeak-app/commit/b063518c61711ab7a0ca89ffa3d2494925fac823>
>>>> # Temporarily disable test status check for trunk builds. Remove this
>>>> check as
>>>> # soon as all tests are running in trunk (hopefully soon).
>>>> if is_trunk; then
>>>> return 0
>>>> fi
>>>>
>>>>
>>>> Couldn't we remove this edge case and run check_test_status *after* the
>>>> deployment stuff in prepare.sh?
>>>>
>>>>
>>>> Best,
>>>>
>>>> Christoph
>>>>
>>>>
>>>> <http://www.hpi.de/>
>>>>
>>>> ------------------------------
>>>> *Von:* Squeak-dev <squeak-dev-bounces at lists.squeakfoundation.org> im
>>>> Auftrag von Fabio Niephaus <lists at fniephaus.com>
>>>> *Gesendet:* Freitag, 17. April 2020 16:25:40
>>>> *An:* The general-purpose Squeak developers list
>>>> *Betreff:* Re: [squeak-dev] [CRON] Fixed:
>>>> squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518)
>>>>
>>>> On Fri, Apr 17, 2020 at 4:10 PM Thiede, Christoph <
>>>> Christoph.Thiede at student.hpi.uni-potsdam.de> wrote:
>>>>
>>>>> Do we really want to get exit code 0 before every single test passes?
>>>>>
>>>>
>>>> Sorry, but I don't understand your question. Maybe because not all
>>>> tests were passing [1]?
>>>>
>>>> [1]
>>>> https://travis-ci.org/github/squeak-smalltalk/squeak-app/jobs/676189826#L789
>>>>
>>>>
>>>>> ------------------------------
>>>>> *Von:* Squeak-dev <squeak-dev-bounces at lists.squeakfoundation.org> im
>>>>> Auftrag von Travis CI <builds at travis-ci.org>
>>>>> *Gesendet:* Freitag, 17. April 2020 15:41:44
>>>>> *An:* squeak-dev at lists.squeakfoundation.org
>>>>> *Betreff:* [squeak-dev] [CRON] Fixed:
>>>>> squeak-smalltalk/squeak-app#1648 (squeak-trunk - b063518)
>>>>>
>>>>>
>>>>> squeak-smalltalk
>>>>>
>>>>> /
>>>>>
>>>>> squeak-app
>>>>>
>>>>> <https://travis-ci.org/github/squeak-smalltalk/squeak-app?utm_medium=notification&utm_source=email>
>>>>>
>>>>> [image: branch icon]squeak-trunk
>>>>> <https://github.com/squeak-smalltalk/squeak-app/tree/squeak-trunk>
>>>>> [image: build has passed]
>>>>> Build #1648 was fixed
>>>>> <https://travis-ci.org/github/squeak-smalltalk/squeak-app/builds/676189824?utm_medium=notification&utm_source=email>
>>>>> [image: arrow to build time]
>>>>> [image: clock icon]20 mins and 26 secs
>>>>>
>>>>> [image: Fabio Niephaus avatar]Fabio Niephaus
>>>>> b063518 CHANGESET →
>>>>> <https://github.com/squeak-smalltalk/squeak-app/compare/76ccf898d33362c1a577fb2cccdcb3dd4ae9b253...b063518c61711ab7a0ca89ffa3d2494925fac823>
>>>>>
>>>>> Send email notifications to squeak-dev [ci skip]
>>>>>
>>>>> Want to know about upcoming build environment updates?
>>>>>
>>>>> Would you like to stay up-to-date with the upcoming Travis CI build
>>>>> environment updates? We set up a mailing list for you!
>>>>> SIGN UP HERE <http://eepurl.com/9OCsP>
>>>>>
>>>>> [image: book icon]
>>>>>
>>>>> Documentation <https://docs.travis-ci.com/> about Travis CI
>>>>> Have any questions? We're here to help. <support at travis-ci.com>
>>>>> Unsubscribe
>>>>> <https://travis-ci.org/account/preferences/unsubscribe?repository=8901856&utm_medium=notification&utm_source=email>
>>>>> from build emails from the squeak-smalltalk/squeak-app repository.
>>>>> To unsubscribe from *all* build emails, please update your settings
>>>>> <https://travis-ci.org/account/preferences/unsubscribe?utm_medium=notification&utm_source=email>.
>>>>>
>>>>> [image: black and white travis ci logo] <https://travis-ci.com>
>>>>>
>>>>> Travis CI GmbH, Rigaer Str. 8, 10427 Berlin, Germany | GF/CEO: Randy
>>>>> Jacops | Contact: contact at travis-ci.com | Amtsgericht Charlottenburg,
>>>>> Berlin, HRB 140133 B | Umsatzsteuer-ID gemäß §27 a Umsatzsteuergesetz:
>>>>> DE282002648
>>>>>
>>>>>
>>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20200420/1030aa61/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 19029 bytes
Desc: not available
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20200420/1030aa61/attachment-0001.png>


More information about the Squeak-dev mailing list