Hi Marcel,

Actually, I'm not seeing the problem on my system (Linux). The tests in NetworkTests amd the WebClient tests are green, regardless of whether the "Enable IPv6" preference is true or false.

I do see intermittent error failures in WebClientServerTest. Usually all tests pass, but I have seen failures in #testStreaming and #testPersistentAutRedirectSession so far. It is very intermittent, but I know that the new resolver primitives introduce some new possibilities for concurrency issue, so there may be some problems in that regard.

But the failures you see are presumably consistent, and are much different from what I am seeing on my system.

I have to suspect platform differences at this point. I think that you are using OS X (is that right?) and I expect that the network resolver may behave quite differently on different platforms, so we'll need to collect a few more data points to see where the issues are.

Can others please try running WebClientServerTest both with "Enable IPv6" true and with it false, and report back on whether the tests pass on your Unix/Linux/Windows/OS X system?

Thanks, Dave

On Wed, Apr 22, 2020 at 11:31:30AM -0400, David T. Lewis wrote:

Oh my, that's going to take some tidying up.

And I expect that in order to get full coverage, we will need to arrange for tests that switch the preference to cover both the old and the new network settings when we run the full test suite.

I can't look at it right now but will do so soon-ish.

Dave

On Wed, Apr 22, 2020 at 03:56:37PM +0200, Marcel Taeumel wrote:

...

Hi Dave! :-)

Now we have to fix those tests that suddenly got nervous and fail or time out -- at least on my machine:

YELLOW - FAIL

WebClientServerTest>>#testBasicAuth WebClientServerTest>>#testBerarerAuth WebClientServerTest>>#testContentTypeCharsetEncoding WebClientServerTest>>#testDigestAuth WebClientServerTest>>#testHtmlSubmit WebClientServerTest>>#testHttpDelete WebClientServerTest>>#testInvalidCookies WebClientServerTest>>#testNestedAction

RED - ERROR

ArbitraryObjectSocketTestCase>>#testBasics SocketTest>>#testClientConnect SocketTest>>#testDataReceive SocketTest>>#testDataSending SocketTest>>#testLocalAddress SocketTest>>#testLocalPort SocketTest>>#testPeerName SocketTest>>#testReceiveTimeout SocketTest>>#testRemoteAddress SocketTest>>#testRemotePort SocketTest>>#testSendTimeout SocketTest>>#testServerAccept SocketTest>>#testStringFromAddress SocketTest>>#testUDP SqueakSSLTest>>#testGooglePopStream StringSocketTestCase>>#testBasics StringSocketTestCase>>#testBogusInput1 StringSocketTestCase>>#testBogusInput2 WebClientServerTest>>#testListenOnInterface

;o)

Best, Marcel Am 07.04.2020 17:12:43 schrieb commits@source.squeak.org commits@source.squeak.org: David T. Lewis uploaded a new version of Network to project The Trunk: http://source.squeak.org/trunk/Network-dtl.240.mcz

==================== Summary ====================

Name: Network-dtl.240 Author: dtl Time: 6 April 2020, 10:33:07.751386 pm UUID: a8f00e08-7398-4d4a-8cd2-09cedcada729 Ancestors: Network-eem.239

This update enables the Squeak new networking and IPv6 support by default. The update is in the package postscript.

IPv6 network support for the VM (SocketPlugin) and image were implemented by Ian Piumarta circa 2007. This included IPv6 support and a more general image-side representation of network addressing. In 2012, this work was merged into Squeak trunk, with a preference to switch between the old networking support or the new networking support with IPv6. At that time, there were concerns with performance of NetNameResolver on some Windows platforms, and for this reason the "Enable IPv6 and new network support" preference was left disabled for official Squeak distributions.

It now 2020, and operating systems have improved over the years, so it is time to enable the new networking preference by default.

=============== Diff against Network-eem.239 ===============

Item was changed:

• (PackageInfo named: 'Network') postscript: '"Enable IPv6 and new network support by default"
• NetNameResolver enableIPv6: true
• '!

• (PackageInfo named: 'Network') postscript: '"Rebuild CSNonAtom."
• MailAddressTokenizer cleanUp: false'!

...

The network resolvers are definitely different in Linux and MacOS, the one in MacOS is less forgiving, or maybe it’s less robust.

/—————————————————————/ For encrypted mail use jgpfersich@protonmail.com - Free account at ProtonMail.com Web: https://objectnets.net and https://objectnets.org https://datascilv.com https://datascilv.org

On Apr 22, 2020, at 18:36, David T. Lewis lewis@mail.msen.com wrote:

I have to suspect platform differences at this point. I think that you are using OS X (is that right?) and I expect that the network resolver may behave quite differently on different platforms, so we'll need to collect a few more data points to see where the issues are.

On Thu, 23 Apr 2020, Marcel Taeumel wrote:

Hi all! I am on Windows 10. This is my output for the usual calls. Maybe it helps:

SocketAddressInformation forHost: 'localhost' service: '12345' flags: 0 addressFamily: 0 socketType: SocketAddressInformation socketTypeStream protocol: SocketAddressInformation protocolTCP.

" an OrderedCollection(::1(MOBILUS-TAB-NEW),12345(12345)-inet6-stream-tcp 127.0.0.1(MOBILUS-TAB-NEW),12345(12345)-inet4-stream-tcp)"

NetNameResolver addressForName: 'localhost'.

" ::1(MOBILUS-TAB-NEW),0(0)"

---

Socket >> #connectTo: times out. Basically "WebClient httpGet: 'http://localhost:12345/test/auth'" does not work. ConnectionRefused.

I suspect that the server only listens on the IPv4 address while, due to its undeterministic nature (it returns the first address no matter how many are there, and the order of the addresses is not defined), #addressForName: gives you the IPv6 address, so the client can't connect to the server.

Another issue is that listening on IPv6 addresses is not possible (at least on my machine):

IPv4 works:

s := Socket newTCP. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '127.0.0.1'). s "==> a Socket[waitingForConnection]"

IPv6 does not work. It just fails silently:

s := Socket newTCP. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]"

Levente

Best, Marcel

  Am 23.04.2020 09:42:00 schrieb John Pfersich via Squeak-dev <squeak-dev@lists.squeakfoundation.org>:

  The network resolvers are definitely different in Linux and MacOS, the one in MacOS is less forgiving, or maybe it’s less robust.

  /—————————————————————/For encrypted mail use jgpfersich@protonmail.com - Free account at ProtonMail.comWeb: https://objectnets.net and https://objectnets.org

https://datascilv.com https://datascilv.org

  On Apr 22, 2020, at 18:36, David T. Lewis <lewis@mail.msen.com> wrote:

  I have to suspect platform differences at this point. I think that you
  are using OS X (is that right?) and I expect that the network resolver
  may behave quite differently on different platforms, so we'll need to
  collect a few more data points to see where the issues are.

On Thu, 23 Apr 2020, Marcel Taeumel wrote:

Looking at "Socket newTCP", IPv6 is not even considered. One has to call "Socket newTCP: SocketAddressInformation addressFamilyINET6". Maybe out IPv6 preferences needs to be considered more often?

Right, so #newTCP defines the family prematurely. In my opinion, it should happen when the interface (which is actually an address not an interface) is specified because the two address families can contradict, as the example showed.

Let's do it the "right" way:

s := Socket newTCP: SocketAddressInformation addressFamilyINET6. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]".

Still destroyed. Why? Perhaps because #listenOn:backlogSize:interface: tries to convert the address with #asByteArray and (NetNameResolver addressForName: '::1') asByteArray returns #[1] (what?). So, let's remove that #asByteArray send hoping that the VM can handle the new opaque addresses.

s := Socket newTCP: SocketAddressInformation addressFamilyINET6. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]".

Still destroyed, so that didn't help. Why? Let's have a look at the plugin code (unix this time)[1]:

void sqSocketListenOnPortBacklogSizeInterface(SocketPtr s, sqInt port, sqInt backlogSize, sqInt addr)

Okay, addr must be a pointer to one of those opaque blobs.

saddr.sin_port= htons((short)port);

That short was meant to be unsigned short, right?

saddr.sin_addr.s_addr= htonl(addr);

The VM code expects addr to be a 4-byte unsigned integer (where's the cast?) representing an IPv4 address in host byte order. So, too bad, #listenOn:backlogSize:interface: (primitiveSocketListenOnPortBacklogInterface) does not support IPv6. It never did.

Levente

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/pl...

Best, Marcel

  Am 23.04.2020 16:57:44 schrieb Levente Uzonyi <leves@caesar.elte.hu>:

  On Thu, 23 Apr 2020, Marcel Taeumel wrote:

  > Hi all!
  > I am on Windows 10. This is my output for the usual calls. Maybe it helps:
  >
  > SocketAddressInformation
  > forHost: 'localhost'
  > service: '12345'
  > flags: 0
  > addressFamily: 0
  > socketType: SocketAddressInformation socketTypeStream
  > protocol: SocketAddressInformation protocolTCP.
  >
  > " an OrderedCollection(::1(MOBILUS-TAB-NEW),12345(12345)-inet6-stream-tcp 127.0.0.1(MOBILUS-TAB-NEW),12345(12345)-inet4-stream-tcp)"
  >
  > NetNameResolver addressForName: 'localhost'.
  >
  > " ::1(MOBILUS-TAB-NEW),0(0)"
  >
  > ***
  >
  > Socket >> #connectTo: times out. Basically "WebClient httpGet: 'http://localhost:12345/test/auth'" does not work. ConnectionRefused.

  I suspect that the server only listens on the IPv4 address while, due to
  its undeterministic nature (it returns the first address no matter how
  many are there, and the order of the addresses is not defined),
  #addressForName: gives you the IPv6 address, so the client can't
  connect to the server.

  Another issue is that listening on IPv6 addresses is not possible (at
  least on my machine):

  IPv4 works:

  s := Socket newTCP.
  s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver
  addressForName: '127.0.0.1').
  s "==> a Socket[waitingForConnection]"

  IPv6 does not work. It just fails silently:

  s := Socket newTCP.
  s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver
  addressForName: '::1').
  s "==> a Socket[destroyed]"


  Levente

  >
  > Best,
  > Marcel
  >
  > Am 23.04.2020 09:42:00 schrieb John Pfersich via Squeak-dev :
  >
  > The network resolvers are definitely different in Linux and MacOS, the one in MacOS is less forgiving, or maybe it’s less robust.
  >
  > /—————————————————————/For encrypted mail use jgpfersich@protonmail.com - Free account at ProtonMail.comWeb: https://objectnets.net and https://objectnets.org
  > https://datascilv.com https://datascilv.org
  >
  >
  > On Apr 22, 2020, at 18:36, David T. Lewis wrote:
  >
  > I have to suspect platform differences at this point. I think that you
  > are using OS X (is that right?) and I expect that the network resolver
  > may behave quite differently on different platforms, so we'll need to
  > collect a few more data points to see where the issues are.
  >
  >
  >

Hi Christoph --

In my trunk image, "NetNameResolver enableIPv6" is true. Connections (including SSL) seem to work fine. Yet, those failing tests are concerning ....

Best, Marcel Am 19.09.2021 16:01:09 schrieb christoph.thiede@student.hpi.uni-potsdam.de christoph.thiede@student.hpi.uni-potsdam.de: Hi Dave, Hi all,

I have checked two images that I am updating frequently via the update stream and in both, EnableIPv6 is still set to false. In one of these fresh images, without new networking, the only error in the Network-Tests package is SocketTest>>#testTCPSocketReuse (SocketPrimitiveFailed). If I turn on new networking, I have 15 failures, 2 errors in total (Windows 21H1).

So there are two considerations:

1. The postscript apparently has not worked, or the change was lost by accident during Marcel's pragma refactoring. 2. To me as an outsider, IPv6 does not yet look ready for productive use (at least on Windows). Are we able to get this working prior to the next release, or should we officially revert to the old networking for now?

Thanks in advance!

Best, Christoph

--- Sent from Squeak Inbox Talk [https://github.com/hpi-swa-lab/squeak-inbox-talk]

On 2020-04-23T17:36:12+02:00, leves@caesar.elte.hu wrote:

On Thu, 23 Apr 2020, Marcel Taeumel wrote:

...

Looking at "Socket newTCP", IPv6 is not even considered. One has to call "Socket newTCP: SocketAddressInformation addressFamilyINET6". Maybe out IPv6 preferences needs to be considered more often?

Right, so #newTCP defines the family prematurely. In my opinion, it should happen when the interface (which is actually an address not an interface) is specified because the two address families can contradict, as the example showed.

Let's do it the "right" way:

s := Socket newTCP: SocketAddressInformation addressFamilyINET6. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]".

Still destroyed. Why? Perhaps because #listenOn:backlogSize:interface: tries to convert the address with #asByteArray and (NetNameResolver addressForName: '::1') asByteArray returns #[1] (what?). So, let's remove that #asByteArray send hoping that the VM can handle the new opaque addresses.

s := Socket newTCP: SocketAddressInformation addressFamilyINET6. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]".

Still destroyed, so that didn't help. Why? Let's have a look at the plugin code (unix this time)[1]:

void sqSocketListenOnPortBacklogSizeInterface(SocketPtr s, sqInt port, sqInt backlogSize, sqInt addr)

Okay, addr must be a pointer to one of those opaque blobs.

saddr.sin_port= htons((short)port);

That short was meant to be unsigned short, right?

saddr.sin_addr.s_addr= htonl(addr);

The VM code expects addr to be a 4-byte unsigned integer (where's the cast?) representing an IPv4 address in host byte order. So, too bad, #listenOn:backlogSize:interface: (primitiveSocketListenOnPortBacklogInterface) does not support IPv6. It never did.

Levente

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/pl...

...

Best, Marcel

Am 23.04.2020 16:57:44 schrieb Levente Uzonyi <leves at caesar.elte.hu>:

On Thu, 23 Apr 2020, Marcel Taeumel wrote:

...

Hi all! I am on Windows 10. This is my output for the usual calls. Maybe it helps:

SocketAddressInformation forHost: 'localhost' service: '12345' flags: 0 addressFamily: 0 socketType: SocketAddressInformation socketTypeStream protocol: SocketAddressInformation protocolTCP.

" an OrderedCollection(::1(MOBILUS-TAB-NEW),12345(12345)-inet6-stream-tcp 127.0.0.1(MOBILUS-TAB-NEW),12345(12345)-inet4-stream-tcp)"

NetNameResolver addressForName: 'localhost'.

" ::1(MOBILUS-TAB-NEW),0(0)"

---

Socket >> #connectTo: times out. Basically "WebClient httpGet: 'http://localhost:12345/test/auth'" does not work. ConnectionRefused.

I suspect that the server only listens on the IPv4 address while, due to its undeterministic nature (it returns the first address no matter how many are there, and the order of the addresses is not defined), #addressForName: gives you the IPv6 address, so the client can't connect to the server.

Another issue is that listening on IPv6 addresses is not possible (at least on my machine):

IPv4 works:

s := Socket newTCP. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '127.0.0.1'). s "==> a Socket[waitingForConnection]"

IPv6 does not work. It just fails silently:

s := Socket newTCP. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]"

Levente

...

Best, Marcel

Am 23.04.2020 09:42:00 schrieb John Pfersich via Squeak-dev :

The network resolvers are definitely different in Linux and MacOS, the one in MacOS is less forgiving, or maybe its less robust.

//For encrypted mail use jgpfersich at protonmail.com - Free account at ProtonMail.comWeb: https://objectnets.net and https://objectnets.org https://datascilv.com https://datascilv.org

On Apr 22, 2020, at 18:36, David T. Lewis wrote:

I have to suspect platform differences at this point. I think that you are using OS X (is that right?) and I expect that the network resolver may behave quite differently on different platforms, so we'll need to collect a few more data points to see where the issues are.

Hi Christoph,

Here is my best effort to give some background. I do not know the full story, but here is a summary:

Ian Piumarta designed and wrote the IPv6 support in the VM SocketPlugin in 2007. I don't know the background on the project but based on the method stamps, it was a significant chunk of work that Ian did in 2007.

Ian and Andreas Raab did the VM integration for the Unix and Windows VMs. I expect that they worked together on this to create the platform support code for Windows and Unix to make the necessary primitives available for both Windows and Unix. This probably also happened in 2007.

At that point, the primitives would have been available for the Unix and Windows VMs (I don't know the status of Mac VMs at that point, but most likely they would have adopted the Unix code).

The next step would have been to integrate the new VM support for IPv6 into the image. However for reasons I cannot explain, this apparently did not happen. Maybe the key developers moved on to other interests, or perhaps some project funding dried up, I don't know. But whatever the reason, Ian's and Andreas' work on IPv6 support did not really get included into Squeak as intended.

Some years later, I had noticed that the VM support was bit-rotting in the VM without anyone using it, so I decided to try to integrate it into the image. It was not clear if the IPv6 stuff would be reliable (or even if it was fully developed), so I spliced it into the existing network support with a preference for "EnableIPv6 and new network support" to use the original IPv4 code by default and to switch to the new network model for bleeding-edge Squeakers.

Back in those days, Windows operating systems in general (and the network resolver support in particular) were less advanced than you might expect today. In particular, there were concerns that name resolution on Windows could have very serious performance problems when running the new IPv6 support in Squeak. So when we got the spliced networking support included into trunk a number of years ago, we made the safe choice of keeping the EnableIPv6 preference set to false.

In a perfect world, we would have had lots of developers who would set the preference true and work out all of the VM and image bugs. But in the real world this did not happen, so some time back in our current release cycle, we decided to flip the bit and see if we could identify and resolve the open issues.

At this point I would say that things look pretty good for the Unix VMs (one failing test, probably a VM issue), and maybe not so good for Windows just because the Windows platform may need to get some VM updates (after all, 2007 was a long time ago, and things have changed on Windows since then).

My advice now: Unless there are some real-world problems with EnableIPv6 for today's Windows users, then keep it enabled. Think of the failing network tests as a "to-do list" of issues that still need work in the VM and/or the image.

Dave

On Sun, Sep 19, 2021 at 01:13:49AM +0200, christoph.thiede@student.hpi.uni-potsdam.de wrote:

Hi Dave, Hi all,

I have checked two images that I am updating frequently via the update stream and in both, EnableIPv6 is still set to false. In one of these fresh images, without new networking, the only error in the Network-Tests package is SocketTest>>#testTCPSocketReuse (SocketPrimitiveFailed). If I turn on new networking, I have 15 failures, 2 errors in total (Windows 21H1).

So there are two considerations:

1. The postscript apparently has not worked, or the change was lost by accident during Marcel's pragma refactoring.
2. To me as an outsider, IPv6 does not yet look ready for productive use (at least on Windows). Are we able to get this working prior to the next release, or should we officially revert to the old networking for now?

Thanks in advance!

Best, Christoph

---

Sent from Squeak Inbox Talk

On 2020-04-23T17:36:12+02:00, leves@caesar.elte.hu wrote:

...

On Thu, 23 Apr 2020, Marcel Taeumel wrote:

...

Looking at "Socket newTCP", IPv6 is not even considered. One has to call "Socket?newTCP: SocketAddressInformation addressFamilyINET6". Maybe out IPv6 preferences needs to be considered more often?

Right, so #newTCP defines the family prematurely. In my opinion, it should happen when the interface (which is actually an address not an interface) is specified because the two address families can contradict, as the example showed.

Let's do it the "right" way:

s := Socket newTCP: SocketAddressInformation addressFamilyINET6. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]".

Still destroyed. Why? Perhaps because #listenOn:backlogSize:interface: tries to convert the address with #asByteArray and (NetNameResolver addressForName: '::1') asByteArray returns #[1] (what?). So, let's remove that #asByteArray send hoping that the VM can handle the new opaque addresses.

s := Socket newTCP: SocketAddressInformation addressFamilyINET6. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]".

Still destroyed, so that didn't help. Why? Let's have a look at the plugin code (unix this time)[1]:

void sqSocketListenOnPortBacklogSizeInterface(SocketPtr s, sqInt port, sqInt backlogSize, sqInt addr)

Okay, addr must be a pointer to one of those opaque blobs.

saddr.sin_port= htons((short)port);

That short was meant to be unsigned short, right?

saddr.sin_addr.s_addr= htonl(addr);

The VM code expects addr to be a 4-byte unsigned integer (where's the cast?) representing an IPv4 address in host byte order. So, too bad, #listenOn:backlogSize:interface: (primitiveSocketListenOnPortBacklogInterface) does not support IPv6. It never did.

Levente

[1] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/Cog/platforms/unix/pl...

...

Best, Marcel

Am 23.04.2020 16:57:44 schrieb Levente Uzonyi <leves at caesar.elte.hu>:

On Thu, 23 Apr 2020, Marcel Taeumel wrote:

...

Hi all! I am on Windows 10. This is my output for the usual calls. Maybe it helps:

SocketAddressInformation forHost: 'localhost' service: '12345' flags: 0 addressFamily: 0 socketType: SocketAddressInformation socketTypeStream protocol: SocketAddressInformation protocolTCP.

" an OrderedCollection(::1(MOBILUS-TAB-NEW),12345(12345)-inet6-stream-tcp 127.0.0.1(MOBILUS-TAB-NEW),12345(12345)-inet4-stream-tcp)"

NetNameResolver addressForName: 'localhost'.

" ::1(MOBILUS-TAB-NEW),0(0)"

---

Socket >> #connectTo: times out. Basically "WebClient httpGet: 'http://localhost:12345/test/auth'" does not work. ConnectionRefused.

I suspect that the server only listens on the IPv4 address while, due to its undeterministic nature (it returns the first address no matter how many are there, and the order of the addresses is not defined), #addressForName: gives you the IPv6 address, so the client can't connect to the server.

Another issue is that listening on IPv6 addresses is not possible (at least on my machine):

IPv4 works:

s := Socket newTCP. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '127.0.0.1'). s "==> a Socket[waitingForConnection]"

IPv6 does not work. It just fails silently:

s := Socket newTCP. s listenOn: 60000 backlogSize: 8 interface: (NetNameResolver addressForName: '::1'). s "==> a Socket[destroyed]"

Levente

...

Best, Marcel

Am 23.04.2020 09:42:00 schrieb John Pfersich via Squeak-dev :

The network resolvers are definitely different in Linux and MacOS, the one in MacOS is less forgiving, or maybe its less robust.

//For encrypted mail use jgpfersich at protonmail.com - Free?account at ProtonMail.comWeb: https://objectnets.net and https://objectnets.org https://datascilv.com https://datascilv.org

On Apr 22, 2020, at 18:36, David T. Lewis wrote:

I have to suspect platform differences at this point. I think that you are using OS X (is that right?) and I expect that the network resolver may behave quite differently on different platforms, so we'll need to collect a few more data points to see where the issues are.