[squeak-dev] The Inbox: SUnit-cmm.116.mcz
frank.shearar at gmail.com
Wed May 29 16:34:06 UTC 2019
On Wed., May 29, 2019, 00:16 Marcel Taeumel, <marcel.taeumel at hpi.de> wrote:
> Hi Chris,
> if an unexpected pass would be treated as a failure, *you would encourage*
> tests like this:
> self assert: 5 / 0.
> ^ #(testSomething)
> A programmer's goal should always be to write passing tests. An expected
> failure should only be temporary. If that expected failure passes at some
> point without the programmer noticing and updating #expectedFailures,
> that's not a bad thing. It's progress. The programmer will update
> #expectedFailures eventually.
> The list of expected failures is just an add-on to your test suite to
> document not-yet-working stuff so that fellow programmers do not treat
> those failing tests as some new regression.
> If you treat expected failures as something more inherent in your test
> suite, you would encourage tests like the one sketched above. I wouldn't do
But where is the incentive for a programmer to fix a test and remember to
On the other hand there might be tests that flake, but no one has gotten
around to deflaking it and we suppress the test with #expectedFailures, when
it passes 80% of the time we want to not fail builds 20% of the time. (Ok
really we do, but not many of us are paid to keep the build green.)
> Am 29.05.2019 00:42:50 schrieb Chris Muller <ma.chris.m at gmail.com>:
> Hey Marcel,
> I think we either disagree what #expectedFailures is for, or one of us
> forgot (possibly me). When I "zoomed out" earlier in the thread to explain
> my real world use-case, it was an implicit invitation for alternative
> solutions. Would you mind helping me out with one then?
> Because I assume we agree on these fundamental requirements of SUnit..
> 1) allow user to know the result of running tests via a gross glance
> from across the room (e.g., if I see "green", I know all is good).
> 2) Be able to interrogate the result of running tests *programmatically*
> (e.g., "myResult failures isEmpty"...?).
> 3) Be able to document functionality *expected* to fail (independent of
> "should"), while keeping the above requirements in line with these
> ..but by saying that expectedFailures is just "an FYI list off-to-the-side
> for documentation" -- and treating them the same as regular failures
> basically neuters all the above basic capabilities of SUnit. We're forcing
> it to report false negatives *100% of the time*, which means it'll *never
> report green*.
> > But expectedFailures are a subset *within* a domain-specific reason that
> > not only *should* (IMO), but *need* to be separated out, because SUnit
>> > provides registration of expectedFailures ONLY at the #test... method
>> > level.
>> Hmm... no. Given that expected failures work at the level of a #test...
>> method, programmers can just work with that. Actually, SUnit relies on the
>> programmer to figure out the best granularity for both passing and failing
>> tests. There is no need to do some analysis that goes sub-method here.
> Despite the "no", I actually agree with everything you said after that...
> I just think that "figuring out the best granularity" involves doing
> "sub-method" analysis automatically, without even realizing it.
>> Your proposed change in TestResult >> #unexpectedPasses renders the code
>> practically unreadable:
>> ^ failures select: [:each | each shouldPass not]
>> What does it say? Well, somehow, a "pass" got stored in the variable
>> "failures" and if you happen to know that #shouldPass can sort out those
>> hidden passes... way too much magic.
> The fact you put "pass" in quotes, above, means, deep down, I think you
> agree with me, :) that going against the programmers *declared
> expectations* is NOT a pass, which is why it's rightly found in the
> failures collection.
>> This was your initial fallacy for this proposal:
>> > The Result of a Test which is tagged as an #expectedFailure IS a
>> failure if the test passed.
>> No, it is not. :-) If anything, the programmer could be happy that an
>> expected failure passes. It is *not a failure*. Maybe something third,
>> which we don't have yet. Serendipity? Happy accident? Not a failure.
> You can't possibly believe that and articulate a rational reason for the
> existence of #expectedFailures.
> LOL, and it wouldn't be "happy" because you wouldn't even know it
> happened! The SUnit browser would just be the same yellow it always is,
> with a same-looking smattering of "expectedFailures" listed at the top,
> maybe even with one REAL failure mixed in, so that the size of the list
> looks just right, and so you won't even bother checking manually the actual
> testMethodNames which failed to make sure they're truly ALL the "expected"
> ones. It could be until years later, when another programmer would find it
> sitting in "expectedFailures" and get confused why its in there.
> Please help me to rediscover the capabilities of SUnit that made it
> actually useful.
> - Chris
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Squeak-dev