Hi Eliot,

...snip...
>  *BUT* (and it's a big BUT) only if the test signals can be seen.
...snip...

> So lumping lots of tests together so that the one persistently failing
> test marks the whole suite as having failed isn't helpful. For
> example, if our SUnit tests were grouped by package then we could see
> much sooner when a commit of a package broke something.  Right now,
> with everything lumped together the fact that we have some failing
> tests (an inevitability in complex projects with lots going on, and
> long term issues that need long term actions to solve) obscures the
> test results, and means we're not able to use them as we should. If we
> broke out the test results in sub groups, and didn't report the
> overall signal, but emphasised the deltas between successive runs of
> individual groups, we would see when things break.
Fully agree. I never know what to make of the report that goes to the
list. An I would be frustrated searching the problem with that little
information.

What if Squeak's TestRunner iterates over the packages  and on each
failure writes a text file with the packages's name. Jenkins appends the
directory list of these files to the mail going to Squeak dev.

Sorry I know nothing about Squeak's build system but at my employer
ASCII files are what binds the system tests together. Audio processing,
GUI, hardware, different cultures, different programming languages,
incompatible tools, but we get one detailed report where we can see the
single test that failed.

>
> Right now the CI for VM builds aggregates in exactly this way so when
> I look in my email I see that the tests have failed (so what's new"). 
> Well, yesterday I had time to look (I don't always) and found that the
> builds are OK up until a run of a build of a newspeak VM, and the VM
> build is OK, it is the Newspeak bootstrap that fals, and since
> Newspeak (I think I'm right in thinking) is no longer maintained, this
> is not a surprise.
>
> But instead of simply discarding the Newspeak build to get green tests
> again, better would be to
> a) not chain the builds so that an early failure prevents potentially
> successful subsequent builds, i.e. attempt to build everything
> b) report to the mailing list a message that specifies which builds
> have failed, so the subject line would be something like "Travis CI: M
> builds out of N have failed", not the depressing "the build has failed"

No, 'This specific build has failed' please. Even with a few builds you
usually start looking at the wrong end. At least I do.

Cheers,

Herbert

>
>     I'm bitten by a workflow of quick commits (we are also using Git)
>     as a QA person in my day job where they also add hasty reviews to
>     the quick commits. So I kind of cringed when I read Christoph's 
>     post :-).
>
>
>     Still a complex or tedious workflow is also bad for quality and
>     productivity so that's the point of the post I fully agree with.
>
>
>     I'm too far away from Squeak but would it be a possibility to have
>     small commits in Git (or elsewhere) and consolidated bigger ones
>     for a proper ancestry? At least we need to be very careful when
>     changing the community process.
>
>
> We have all the infrastructure we need in Monticello (it supports
> branching and merging, and multiple repositories).  We even have
> something close to what you suggest in production, with release
> repositories that allow people to stay with e.g. squeak53, and we can
> push improvements and bug fixes there. But I agree, we want something
> like a high-frequency trunk and a low-frequency trunk.  So we could
> implement e.g. treatedtrunk, and, say, automate pushing the 3 day, or
> 7 day old, trunk to treatedtrunk.  It would be easy for those wanting
> a stable trunk process to stay with the treatedtrunk update stream,
> and to load from trunk the latest greatest that they really want,
> since the delta between treatedtrunk and trunk would be small, and the
> delta would be no more than a few days worth of commits.
>
>     Best regards,
>
>
>     Herbert
>
>
>
>     Am 29.09.20 um 01:38 schrieb Thiede, Christoph:
>>
>>     > Maybe for git-based projects, a commit-first, fix-later
>>     strategy is what that culture likes and that system can support,
>>     but it's not a good fit for Squeak's trunk.
>>
>>
>>     And that's why - sorry for rolling out my old chestnut again - I
>>     still do believe that a git-based workflow could make us as a
>>     community more efficient.
>>
>>     I don't question that manual testing is great, and I don't
>>     question that quality can increase if you give your patches time
>>     for maturing, but when I compare the resulting workflow to the
>>     "mainstream" workflow that I can use anywhere on GitHub, I
>>     repeatedly have the dissatisfying feeling that the inbox/trunk
>>     workflow is so slow that it ruins all the efficiency from the
>>     Smalltalkish development workflow (which, however, unquestionably
>>     outperforms the "mainstream" workflow in a dead, non-live UI
>>     without first-class objects for code and tools!).
>>
>>     This might apply most significantly to small changes that would
>>     form a PR of two or three commits in a git project because our
>>     inbox workflow does not scale so well for changes of such
>>     extent. I do not know how many hours I already have spent on
>>     fixing the ancestry of my versions, comparing them to their
>>     ancestors, or re-uploading them, but it has definitively been too
>>     many hours ...
>>
>>
>>     Did someone ever investigate this question seriously by doing a
>>     study or so? I would really find the results interesting.
>>
>>
>>     Best,
>>
>>     Christoph
>>
>>
>>     ------------------------------------------------------------------------
>>     *Von:* Squeak-dev <squeak-dev-bounces at lists.squeakfoundation.org>
>>     <mailto:squeak-dev-bounces at lists.squeakfoundation.org> im Auftrag
>>     von Chris Muller <asqueaker at gmail.com> <mailto:asqueaker at gmail.com>
>>     *Gesendet:* Dienstag, 29. September 2020 01:00 Uhr
>>     *An:* The general-purpose Squeak developers list
>>     *Betreff:* Re: [squeak-dev] tedious programming-in-the-debugger
>>     error needs fixing
>>     On Mon, Sep 28, 2020 at 10:07 AM Eliot Miranda
>>     <eliot.miranda at gmail.com <mailto:eliot.miranda at gmail.com>> wrote:
>>
>>         Hi Christoph,
>>
>>         On Fri, Sep 25, 2020 at 7:58 AM Thiede, Christoph
>>         <Christoph.Thiede at student.hpi.uni-potsdam.de
>>         <mailto:Christoph.Thiede at student.hpi.uni-potsdam.de>> wrote:
>>
>>             Hi Eliot,
>>
>>
>>             I'm very sorry for this bug, which was so unnecessary
>>             because my image has still a gigantic working copy ...!
>>             Tools-ct.988 should fix the issue, I tested it in a fresh
>>             trunk image. But it would be probably better if you could
>>             test it yourself, too. ;-)
>>
>>
>>         No need to apologise.  It's an easy mistake, and you fixed
>>         it.  As long as we're all patient with each other and take
>>         responsibility (Andreas said "if you break it, you fix it")
>>         we're going to get along fine and be collectively productive.
>>
>>
>>     The following is not addressed to Christoph or his commit, but to
>>     Eliots comment, above:  Patience should begin within our
>>     development methodology.  The words above are correct and sweet,
>>     and I agree with them, but I feel the need to caution against the
>>     implication that "everything's great as long as you fix it
>>     /afterward/." Maybe for git-based projects, a /commit-first,
>>     fix-later/ strategy is what that culture likes and that system
>>     can support, but it's not a good fit for Squeak's trunk. I
>>     believe Andreas understood this, and he indicated that "breaking
>>     the trunk is generally frowned upon."
>>
>>     When it comes to code less than 24 hours old, no matter how
>>     simple it seems, chances are about 80% that a subsequent "oops,
>>     sorry" commit will need to follow.  With "older," (e.g., even
>>     only just 48 hours!) _tested_ code, that chance drops
>>     significantly. Patience.  Restraint.  Please.  :)  Let our
>>     methodology put time to work /for/ us, by living with our changes
>>     for a bit (as, it sounds like, Christoph did!) and witness them
>>     work in context.  Maybe not this time, but /generally, /you'll
>>     have a gist enough to know whether it should be loaded and tested
>>     separately in a clean trunk first.
>>
>>     Cheers,
>>       Chris
>>
>
>
>
> --
> _,,,^..^,,,_
> best, Eliot
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20200929/ea7bfb39/attachment-0001.html>