On Fri, 21 Jun 2019 at 4:20 pm, Bernhard Pieber <bernhard at pieber.com> wrote:

> Thanks for the clarification. I did not know there was a separate
> mechanism for ignoring tests in smalltalkCI beside #expectedFailures in the
> image. Do you happen to know what is the rationale for that? Especially
> when the build succeeds anyway.


During the preparation of, I think, the Squeak 5.1 release, we wanted to
fix all tests, so we set up the CI to give us a todo list. Eventually, I
wanted to let trunk builds depend on the test results, so that we'd always
get notified if test break again. Unfortunately, we couldn't fix all tests
in time for the release and some still fail. We currently run all builds on
macOS, so that we can sign the app bundles. We probably need something like
this for Windows too at some point. But as far as I can remember,
SqueakSSLTest have never passed on macOS.


>
> It is a pity that the trunk is not green all the time. I wonder how
> difficult it would be to include information on failing tests in the
> download packages say as a known_test_failures.txt alongside the image
> file. It might help to increase the urge to fix them. Besides it would make
> it easier to find out whether one has introduced a regression.


I agree. You could programmatically download and parse the last X build
results using Travis' REST API. That's why we haven't uploaded them
anywhere. I'd rather spend some time fixing bugs that are failing in the
latest trunk and then flip the switch, so that we get notified if someone
breaks something. Some test are documenting feature requests or VM-level
bugs that cannot be fixed in the image. So we probably still need to ignore
some in the smalltalkCI config. Also, sometimes builds timeout due to
fragile network handling or VM segfaults. Some hardening would also be
necessary I guess.

Fabio


>
> - Bernhard
>
> > Am 21.06.2019 um 15:05 schrieb Fabio Niephaus <lists at fniephaus.com>:
> >
> >
> >
> > On Fri, 21 Jun 2019 at 11:45 am, Bernhard Pieber <bernhard at pieber.com>
> wrote:
> > Thank you for taking the time to answer. At least I know I am not the
> only one.
> >
> > Your link to the travis job log helped me find the travis config:
> > https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159/config
> >
> > The SUnit tests are being executed by smalltalkCI and in its config,
> SqueakSSLTest is excluded [1].
> > BTW: I think builds succeed, even if tests fail. Sometimes, the VM even
> segfaults during testing.
> >
> > Fabio
> >
> > [1]
> https://github.com/squeak-smalltalk/squeak-app/blob/squeak-trunk/smalltalk-ci/Squeak-trunk-64.ston#L10
> >
> >
> >
> > According to this the build seems to be run on macOS 10.11. So it is
> strange that the test failures don't show up in the job log. Maybe they
> succeed on travis. However, that is hard to imagine.
> >
> > Anyone else on a Mac for whom the SqueakSSLTest succeeds?
> >
> > - Bernhard
> >
> > > Am 20.06.2019 um 20:07 schrieb Tobias Pape <Das.Linux at gmx.de>:
> > >
> > > Hi
> > >
> > >> On 20.06.2019, at 19:53, Bernhard Pieber <bernhard at pieber.com> wrote:
> > >>
> > >> Hi,
> > >>
> > >> I have one failure in #testFacebookAPI and three errors in
> #testSSLSockets, #testSocketAccept, and #testSocketAccept in SqueakSSLTest
> on macOS Mojave 10.14.5.
> > >>
> > >> I tried the latest trunk images 18618 with the release VMs and the
> latest VMs on bintray, both 32bit and 64bit:
> > >>
> > >> Croquet Closure Cog[Spur] VM [CoInterpreterPrimitives
> VMMaker.oscog-eem.2530] 5.0.201906130203
> > >> Mac OS X built on Jun 13 2019 02:12:03 GMT Compiler: 4.2.1 Compatible
> Apple LLVM 8.1.0 (clang-802.0.42)
> > >> platform sources revision VM: 201906130203
> https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Wed Jun 12
> 19:03:53 2019 CommitHash: a85579a1 Plugins: 201906130203
> https://github.com/OpenSmalltalk/opensmalltalk-vm.git
> > >> CoInterpreter VMMaker.oscog-eem.2530 uuid:
> 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
> > >> StackToRegisterMappingCogit VMMaker.oscog-eem.2530 uuid:
> 4d90ede0-0700-4d15-8173-2aaf2360b7d1 Jun 13 2019
> > >>
> > >> I would like to know if other Mac users can confirm these. If yes,
> does anyone know anything about these problems?
> > >
> > > The failure/errors have been there since I have looked at all that
> code.
> > > I have no idea why they do not work. They use the SecureSocket variant
> of SqueakSSL, the rest of the system uses the SecureSocketStream IIRC, and
> that works…
> > >
> > >>
> > >> I assume the tests on macOS are not run during the trunk build, right?
> > >
> > > I am not sure. It seems that these are not part of the failed tests on
> travis:
> > >
> > >
> https://travis-ci.org/squeak-smalltalk/squeak-app/jobs/548131159#L2326
> > >
> > > But Really, I have no idea…
> > >
> > > Best regards
> > >       -Tobias
> > >
> > >
> > >>
> > >> Cheers,
> > >> Bernhard
> >
> >
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20190621/70093039/attachment.html>