OK, build.squeak.org, I have to admit, you win... http://build.squeak.org/job/SqueakTrunk/855/console

Ddin't we see this failure already?

Segmentation fault Sun May 25 00:39:10 2014

Squeak VM version: 4.0-2776 #1 Thu Aug 22 10:35:56 PDT 2013 gcc 4.1.2 [Production ITHB VM] Built from: CoInterpreter VMMaker.oscog-eem.331 uuid: 37d2e4b0-2f37-4e2d-8313-c63637785e59 Aug 22 2013 With: StackToRegisterMappingCogit VMMaker.oscog-eem.333 uuid: 84da9cb8-7f30-4cb7-b4fb-239a11f63b54 Aug 22 2013 Revision: VM: r2776 http://www.squeakvm.org/svn/squeak/branches/Cog Plugins: r2545 http://squeakvm.org/svn/squeak/trunk/platforms/Cross/plugins Build host: Linux mcqfes 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux plugin path: /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776 [default: /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/lib/squeak/4.0-2776/]

C stack backtrace: /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak[0x805d051] /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak[0x805d228] [0xf57fe40c] /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak[0x80e5a59] [0x473c7848] /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak(interpret+0x1eb)[0x808107b] /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak(main+0x397)[0x805d717] /lib/i686/nosegneg/libc.so.6(__libc_start_main+0xe6)[0xb7560ca6] /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak[0x805b101]

Smalltalk stack dump: 0xbfa178a4 M LargePositiveInteger>\ 0x4899127c: a(n) LargePositiveInteger

2014-05-25 0:06 GMT+02:00 Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com>:

Ah, no, the build is still failing! Since http://build.squeak.org/job/SqueakTrunk/834/ it's another problem with:

MCClassDefinition>>createClass

I encountered this one while updating some of my old trunk images. The definition changed in Monticello-cwp.589:

•                   inEnvironment: (CurrentEnvironment signal ifNil:
  

[superClass environment])

•                   inEnvironment: (EnvironmentRequest signal ifNil:
  

[superClass environment])

But there is no EnvironmentRequest anymore when loading this definition, so we signal nil, and the load fails. We would have expected this, it's just be renamed in:

Environments-cwp.47 Time: 22 March 2014, 7:53:17.666 pm Rename EnvironmentRequest to CurrentEnvironment and use it to implement Environment class>>current.

To be safe, the rename should have been performed in two stages:

1. create the new class and migrate customers from the old to the new one.
2. remove the old one

It would thus have required two updates.mcm and not a single one (MC lacks a semantic #rename: so it does not work like in-image-tools) Unfortunately, update-cwp.48 did perform the two operations at once...

Curiously, if I take an artefact from build server and update, the upgrade is then processing fine... Don't ask why. Since no one complained, I told to myself that it was me not following the standard process, I patched my images manually and restarted the upgrade. But I now see that it's not just me, there is a CI machine too using Squeak trunk (anyone else???)

I have to cheat again with mcm surgery:

• publish an Environments-nice.47 that adds the CurrentEnvironment, but

not removes EnvironmentRequest

• patch update-cwp.48 to point to Environments-nice.47 rather than

Environments-cwp.47 That'll be in a few minutes...

2014-05-24 22:58 GMT+02:00 Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com>:

...

2014-05-24 22:36 GMT+02:00 Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com>:

Hi,

...

it seems that most of the builds are red for 2 months. I checked the last artefact for squeak trunk: http://build.squeak.org/job/SqueakTrunk/853/

It's still blocked with the [Warning signal: 'About to serialize an empty package.'] With this warning, a test never time out (I don't know why exactly, but it seems that theProcess monitored by the timeout watchdog is already suspended by the Warning) As a result, the tests do never complete and are aborted after 20 minutes...

Chris has fixed the problem since, but it seems that the CI does not upload latest trunk. Why? It seems to be the same problem as the TestRunner: updating triggers the same [Warning signal: 'About to serialize an empty package.'] Ah Chris, it's irony that it's you who precisely keep on fighting modal UI ;)

Now the question: how do we escape from this situation? Shall I cheat and modify the update that is blocking?

So I decided to cheat:

I brought update-eem.279 to the surgery block and reverted CommandLine-fbs.3 -> CommandLine-fbs.2 CommandLine-fbs.3 tried to be smart and workaround the Warning problem, but before it had a chance to load, it precisely triggered this Warning!!! Since this update is also bringing Chris fix, let's defer CommandLine-fbs.3 to next update (eem-280) and cross fingers...

Hi, thanks for noticing and investigating this! Seeing this now, I think we should signal an Error rather than a Warning to be more TestCase friendly but also because persisting empty packages is so painful, it should be an error, hands down. Better to force a resolution to the issue than silently persist modules of future-pain...

On Sat, May 24, 2014 at 6:13 PM, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:

Now http://build.squeak.org/job/SqueakTrunk/856/ is back to a better red state: one that time out after 20 minutes because of the [Warning signal: 'About to serialize an empty package.'] (like I told in my first mail and like in builds http://build.squeak.org/job/SqueakTrunk/823/ to http://build.squeak.org/job/SqueakTrunk/833/)

So applying Chris' fix and repairing the update-stream was not enough to get rid of it.

The Warning is triggered by #testMcdSerialization It appears that (self mockDiffyVersion) has no changes... Though, mockDiffyVersion tries to change: #a toReturn: 'a2'. Something in this test is not working as intended, but here I'll stop for tonight.

2014-05-25 0:48 GMT+02:00 Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:

...

OK, build.squeak.org, I have to admit, you win... http://build.squeak.org/job/SqueakTrunk/855/console

Ddin't we see this failure already?

Segmentation fault Sun May 25 00:39:10 2014

Squeak VM version: 4.0-2776 #1 Thu Aug 22 10:35:56 PDT 2013 gcc 4.1.2 [Production ITHB VM] Built from: CoInterpreter VMMaker.oscog-eem.331 uuid: 37d2e4b0-2f37-4e2d-8313-c63637785e59 Aug 22 2013 With: StackToRegisterMappingCogit VMMaker.oscog-eem.333 uuid: 84da9cb8-7f30-4cb7-b4fb-239a11f63b54 Aug 22 2013 Revision: VM: r2776 http://www.squeakvm.org/svn/squeak/branches/Cog Plugins: r2545 http://squeakvm.org/svn/squeak/trunk/platforms/Cross/plugins Build host: Linux mcqfes 2.6.18-128.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux plugin path: /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776 [default: /var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/lib/squeak/4.0-2776/]

C stack backtrace:

/var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak[0x805d051]

/var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak[0x805d228] [0xf57fe40c]

/var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak[0x80e5a59] [0x473c7848]

/var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak(interpret+0x1eb)[0x808107b]

/var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak(main+0x397)[0x805d717] /lib/i686/nosegneg/libc.so.6(__libc_start_main+0xe6)[0xb7560ca6]

/var/lib/jenkins/workspace/SqueakTrunk/target/cog.r2776/coglinux/bin/../lib/squeak/4.0-2776/squeak[0x805b101]

Smalltalk stack dump: 0xbfa178a4 M LargePositiveInteger>\ 0x4899127c: a(n) LargePositiveInteger

2014-05-25 0:06 GMT+02:00 Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:

...

Ah, no, the build is still failing! Since http://build.squeak.org/job/SqueakTrunk/834/ it's another problem with:

MCClassDefinition>>createClass

I encountered this one while updating some of my old trunk images. The definition changed in Monticello-cwp.589:

•                   inEnvironment: (CurrentEnvironment signal ifNil:
  

[superClass environment])

•                   inEnvironment: (EnvironmentRequest signal ifNil:
  

[superClass environment])

But there is no EnvironmentRequest anymore when loading this definition, so we signal nil, and the load fails. We would have expected this, it's just be renamed in:

Environments-cwp.47 Time: 22 March 2014, 7:53:17.666 pm Rename EnvironmentRequest to CurrentEnvironment and use it to implement Environment class>>current.

To be safe, the rename should have been performed in two stages:

1. create the new class and migrate customers from the old to the new

one. 2) remove the old one It would thus have required two updates.mcm and not a single one (MC lacks a semantic #rename: so it does not work like in-image-tools) Unfortunately, update-cwp.48 did perform the two operations at once...

Curiously, if I take an artefact from build server and update, the upgrade is then processing fine... Don't ask why. Since no one complained, I told to myself that it was me not following the standard process, I patched my images manually and restarted the upgrade. But I now see that it's not just me, there is a CI machine too using Squeak trunk (anyone else???)

I have to cheat again with mcm surgery:

• publish an Environments-nice.47 that adds the CurrentEnvironment, but

not removes EnvironmentRequest

• patch update-cwp.48 to point to Environments-nice.47 rather than

Environments-cwp.47 That'll be in a few minutes...

2014-05-24 22:58 GMT+02:00 Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:

...

2014-05-24 22:36 GMT+02:00 Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:

...

Hi, it seems that most of the builds are red for 2 months. I checked the last artefact for squeak trunk: http://build.squeak.org/job/SqueakTrunk/853/

It's still blocked with the [Warning signal: 'About to serialize an empty package.'] With this warning, a test never time out (I don't know why exactly, but it seems that theProcess monitored by the timeout watchdog is already suspended by the Warning) As a result, the tests do never complete and are aborted after 20 minutes...

Chris has fixed the problem since, but it seems that the CI does not upload latest trunk. Why? It seems to be the same problem as the TestRunner: updating triggers the same [Warning signal: 'About to serialize an empty package.'] Ah Chris, it's irony that it's you who precisely keep on fighting modal UI ;)

Now the question: how do we escape from this situation? Shall I cheat and modify the update that is blocking?

So I decided to cheat: I brought update-eem.279 to the surgery block and reverted CommandLine-fbs.3 -> CommandLine-fbs.2 CommandLine-fbs.3 tried to be smart and workaround the Warning problem, but before it had a chance to load, it precisely triggered this Warning!!! Since this update is also bringing Chris fix, let's defer CommandLine-fbs.3 to next update (eem-280) and cross fingers...

I have been working on a variation of class DateAndTime that replaces its instance variables (seconds offset jdn nanos) with two instance variables, utcMicroseconds to represent microseconds elapsed since the Posix epoch, and localOffsetSeconds to represent the local time zone offset. When instantiating the time now, A single call primitiveUtcWithOffset is used to obtain these two values atomically as reported by the underlying platform.

There are several advantages to this representation of DateAndTime, the most important of which is that its magnitude is unambiguous regardless of daylight savings transitions in local time zones.

This is my attempt to address some historical baggage in Squeak. The VM reports time related to the local time zone, and the image attempts to convert to UTC (sometimes incorrectly). A UTC based representation makes the implementation of time zone tables more straightforward (see for example the Olson time zone tables in TimeZoneDatabase on SqueakMap).

I am attaching the source code as a SAR file that can be loaded into a fully updated Squeak trunk image. The conversion process is slow, so be patient if you load it.

This can be run on either an intepreter VM or Cog, but if you use Cog, please use a version dated June 2013 or later (the VM in the Squeak 4.5 all-in-one is fine).

I am also attaching a copy of LXTestDateAndTimePerformance, which can be used to compare the performance of some basic DateAndTime functions.

Performance of the UTC based DateAndTime is generally favorable compared to the original. Here is what I see on my system (smaller numbers are better).

LXTestDateAndTimePerformance test results using the original Squeak DateAndTime on an interpreter VM: { #testNow->10143 . #testEquals->30986 . #testGreaterThan->80199 . #testLessThan->75912 . #testPrintString->10429 . #testStringAsDateAndTime->44657 }

LXTestDateAndTimePerformance test results using the new UTC based DateAndTime on an interpreter VM: { #testNow->6423 . #testEquals->31625 . #testGreaterThan->22999 . #testLessThan->18514 . #testPrintString->12502 . #testStringAsDateAndTime->32912 }

(CC to Brent Pinkney, author of the excellent Squeak Chronology package)

Dave

Hi Craig, you've mentioned this a few times, and sometimes I wondered if I was doing it right. Could you please confirm?

You're asking if one is replying to a email, but changing the subject-matter of that conversation, then please change the Subject-line of the email. And, optionally(?), put the old subject line inside parentheses prefiexed by "was:").

The purpose of this convention is to let emails be well-formed; their subjects matching the bodies and not wandering into other subjects. This would be helpful for searching email later too..

On Sun, May 25, 2014 at 3:04 PM, Craig Latta craig@netjam.org wrote:

Hoi David--

 Cool! But please use a new thread when starting a new thread on the

mailing list? I kill threads when I'm no longer interested in them, and don't want to miss out on stuff which is actually new.

 thanks,

-C

-- Craig Latta www.netjam.org +31 6 2757 7177 (SMS ok)

• 1 415 287 3547 (no SMS)

On Sun, May 25, 2014 at 7:34 PM, David T. Lewis lewis@mail.msen.com wrote:

On Sun, May 25, 2014 at 05:40:10PM -0500, Chris Muller wrote:

...

On Sun, May 25, 2014 at 3:04 PM, Craig Latta craig@netjam.org wrote:

...

Hoi David--

 Cool! But please use a new thread when starting a new thread on

the

...

...

mailing list? I kill threads when I'm no longer interested in them, and don't want to miss out on stuff which is actually new.

Hi Craig, you've mentioned this a few times, and sometimes I wondered if I was doing it right. Could you please confirm?

You're asking if one is replying to a email, but changing the subject-matter of that conversation, then please change the Subject-line of the email. And, optionally(?), put the old subject line inside parentheses prefiexed by "was:").

The purpose of this convention is to let emails be well-formed; their subjects matching the bodies and not wandering into other subjects. This would be helpful for searching email later too..

Hi Chris,

In this case, I think that caused a problem for a different reason. Out of convenience (and this is something I will not do again), I sometimes start a new mail message by "replying" to an old one, then changing to my new subject line. I had assumed (without ever bothering to check) that mail threading is done based on subject line, and had not noticed that the mail headers contain the "In-Reply-To:" information that no doubt is used for connecting the mail threads.

So in the future I will not use "reply to" unless actually replying to something.

Surely it is the software at fault? If you change the subject line thats clearly starting a new thread. Something I do too. Craig?

Dave

On Sun, May 25, 2014 at 7:58 PM, Craig Latta craig@netjam.org wrote:

...

In this case, I think that caused a problem for a different reason. Out of convenience (and this is something I will not do again), I sometimes start a new mail message by "replying" to an old one, then changing to my new subject line. I had assumed (without ever bothering to check) that mail threading is done based on subject line, and had not noticed that the mail headers contain the "In-Reply-To:" information that no doubt is used for connecting the mail threads.

 Yes; in my case the gmane.org mail/NNTP gateway uses that to make

its references headers, and Mozilla Thunderbird uses those headers to display and manipulate threads when I read the gmane.comp.lang.smalltalk.squeak.general newsgroup.

...

So in the future I will not use "reply to" unless actually replying to something.

 Thanks again!

Once again the tail wags the dog. So to do something natural like reply and change subject is outlawed because fixable software is broken? If we continue to tolerate the broken, guess what?

-C

-- Craig Latta www.netjam.org +31 6 2757 7177 (SMS ok)

• 1 415 287 3547 (no SMS)

On Sun, May 25, 2014 at 10:04:22PM +0200, Craig Latta wrote:

Hoi David--

 Cool! But please use a new thread when starting a new thread on the

mailing list? I kill threads when I'm no longer interested in them, and don't want to miss out on stuff which is actually new.

I apologize, I didn't realized that I was messing up the mail threads. I'll take care not to do that in the future.

Dave

On Mon, May 26, 2014 at 02:17:47PM +0000, J. Vuletich (mail lists) wrote:

Hi David, Folks,

Quoting "David T. Lewis" lewis@mail.msen.com:

...

I have been working on a variation of class DateAndTime that replaces its instance variables (seconds offset jdn nanos) with two instance variables, utcMicroseconds to represent microseconds elapsed since the Posix epoch, and localOffsetSeconds to represent the local time zone offset. When instantiating the time now, A single call primitiveUtcWithOffset is used to obtain these two values atomically as reported by the underlying platform.

There are several advantages to this representation of DateAndTime, the most important of which is that its magnitude is unambiguous regardless of daylight savings transitions in local time zones.

This is my attempt to address some historical baggage in Squeak. The VM reports time related to the local time zone, and the image attempts to convert to UTC (sometimes incorrectly). A UTC based representation makes the implementation of time zone tables more straightforward (see for example the Olson time zone tables in TimeZoneDatabase on SqueakMap). ... Dave

I very much support this approach. I did a bit of testing of <primitive: 'primitiveUtcWithOffset'> . I found that on a Mac, with 'Croquet Closure Cog VM [CoInterpreter VMMaker.oscog-eem.331] Squeak Cog 4.0.2776' 'Mac OS' 'intel' '1092' (from Eliot's site), the second element I get (time zone offset) is -140473411.

The correct value would be -10800, as answered in Windows. I could not test on Linux yet (could not get the vm to run in Ubuntu 14.04 64 bit :( ).

Any clue on what's wrong on Mac OS?

BTW, which would be the current non-Cog VMs to try?

Oops, I mistakenly said that the Cog VMs could be used. But it looks like there is a regression or code merge problem of some sort. I'm afraid that I was testing with my own locally compiled Cog VM and did not notice the problem.

A unix Mac VM from squeakvm.org/unix should demonstrate the correct behavior.

CC to vm-dev list:

Eliot, the fix for this was here (but it seems to have been overridden by a more recent change):

Name: VMMaker.oscog-dtl.286 Author: dtl Time: 4 May 2013, 11:29:25.237 am UUID: 8be237d9-7812-4792-9723-90f9cff0c2e9 Ancestors: VMMaker.oscog-eem.285

Replace broken primitiveUtcWithOffset with a version that works.

Dave

On Mon, May 26, 2014 at 04:41:10PM +0000, J. Vuletich (mail lists) wrote:

Quoting "David T. Lewis" lewis@mail.msen.com:

...

On Mon, May 26, 2014 at 02:17:47PM +0000, J. Vuletich (mail lists) wrote:

...

Hi David, Folks, ... The correct value would be -10800, as answered in Windows. I could not test on Linux yet (could not get the vm to run in Ubuntu 14.04 64 bit :( ).

Any clue on what's wrong on Mac OS?

BTW, which would be the current non-Cog VMs to try?

Oops, I mistakenly said that the Cog VMs could be used. But it looks like there is a regression or code merge problem of some sort. I'm afraid that I was testing with my own locally compiled Cog VM and did not notice the problem.

A unix Mac VM from squeakvm.org/unix should demonstrate the correct behavior.

CC to vm-dev list:

Eliot, the fix for this was here (but it seems to have been overridden by a more recent change):

Name: VMMaker.oscog-dtl.286 Author: dtl Time: 4 May 2013, 11:29:25.237 am UUID: 8be237d9-7812-4792-9723-90f9cff0c2e9 Ancestors: VMMaker.oscog-eem.285

Replace broken primitiveUtcWithOffset with a version that works.

Dave

Thanks Dave,

I could run on 64 bits Ubuntu with the VM from squeakvm.org/unix. I'll try the Mac VM when I get the chance to borrow a Mac again.

One thing that seems to be missing is a Windows interpreter with the new primitives, although I don't know if there is a real need for that.

Ian did a build of the Windows VM that should have the necessary support. Try the Squeak4.1.2.2612 VM from http://squeakvm.org/win32/.

One thing to note - if the primitive is not present, DateAndTime will fall back on the old logic, and it should produce reasonable results.

Additionally, besides getting rid of <primitive: 137> (primLocalSecondsClock that will overflow in 2037), it would be great to stop using <primitive: 135> (primLocalSecondsClock), that overflows every six days. But for this, we would need a new <primitive: 136> (primSignal:atMilliseconds:) as it uses on the same time base. This would enable a serious simplification of Delay.

I think that Eliot is planning to update Squeak to use the microsecond clock primitive, which removes any 2037 issues. I'm not sure if that would include a change to primSignal:atMilliseconds:

Dave

On Mon, May 26, 2014 at 06:16:03PM +0000, J. Vuletich (mail lists) wrote:

Quoting "David T. Lewis" lewis@mail.msen.com:

...

...

...

Ian did a build of the Windows VM that should have the necessary support. Try the Squeak4.1.2.2612 VM from http://squeakvm.org/win32/.

That VM fails for <primitive: 'primitiveUtcWithOffset'> <primitive: 240> primUtcMicrosecondClock <primitive: 241> primLocalMicrosecondClock

...

One thing to note - if the primitive is not present, DateAndTime will fall back on the old logic, and it should produce reasonable results.

Yes, Cuis already does that... I'd prefer to be able to assume that any future VM will provide <primitive: 'primitiveUtcWithOffset'> and clean the code. Would that asking for too much?

You should expect the following two primitives to be present in all VMs (comments are from the VMM trunk implementations):

primitiveUtcWithOffset "Answer an array with UTC microseconds since the Posix epoch and the current seconds offset from GMT in the local time zone. An empty two element array may be supplied as a parameter. This is a named (not numbered) primitive in the null module (ie the VM)"

primitiveUTCMicrosecondClock "Answer the UTC microseconds since the Smalltalk epoch. The value is derived from the Posix epoch (see primitiveUTCMicrosecondClock) with a constant offset corresponding to elapsed microseconds between the two epochs according to RFC 868."

Dave

...

...

Additionally, besides getting rid of <primitive: 137> (primLocalSecondsClock that will overflow in 2037), it would be great to stop using <primitive: 135> (primLocalSecondsClock), that overflows every six days. But for this, we would need a new <primitive: 136> (primSignal:atMilliseconds:) as it uses on the same time base. This would enable a serious simplification of Delay.

I think that Eliot is planning to update Squeak to use the microsecond clock primitive, which removes any 2037 issues. I'm not sure if that would include a change to primSignal:atMilliseconds:

Dave

I see. But given that the Delay code is fragile and not trivial at all, the advantages of of relying on a clock that never rolls over would be significant.

Please, Eliot, consider this when you work on this code.

Cheers, Juan Vuletich

Hi Dave,

May I respectfully ask why localOffsetSeconds (to represent the local time zone offset) is needed? It seems to me a UTC time is enough. Is there really a need for the timezone offset the instance was created in? Does every DateAndTime instance need to carry this offset around with it? I would think the offset is only needed if one wants to display a date/time as a local value and then one could get the local offset from the VM or a program setting the user had previously supplied regardless of where the computer was setup to run. I guess there might be some historic interest as to the timezone an instances (or many instances) was created in but one could just keep that as a separate value.

Lou

On Sun, 25 May 2014 13:48:44 -0400, "David T. Lewis" lewis@mail.msen.com wrote:

I have been working on a variation of class DateAndTime that replaces its instance variables (seconds offset jdn nanos) with two instance variables, utcMicroseconds to represent microseconds elapsed since the Posix epoch, and localOffsetSeconds to represent the local time zone offset. When instantiating the time now, A single call primitiveUtcWithOffset is used to obtain these two values atomically as reported by the underlying platform.

There are several advantages to this representation of DateAndTime, the most important of which is that its magnitude is unambiguous regardless of daylight savings transitions in local time zones.

This is my attempt to address some historical baggage in Squeak. The VM reports time related to the local time zone, and the image attempts to convert to UTC (sometimes incorrectly). A UTC based representation makes the implementation of time zone tables more straightforward (see for example the Olson time zone tables in TimeZoneDatabase on SqueakMap).

I am attaching the source code as a SAR file that can be loaded into a fully updated Squeak trunk image. The conversion process is slow, so be patient if you load it.

This can be run on either an intepreter VM or Cog, but if you use Cog, please use a version dated June 2013 or later (the VM in the Squeak 4.5 all-in-one is fine).

I am also attaching a copy of LXTestDateAndTimePerformance, which can be used to compare the performance of some basic DateAndTime functions.

Performance of the UTC based DateAndTime is generally favorable compared to the original. Here is what I see on my system (smaller numbers are better).

LXTestDateAndTimePerformance test results using the original Squeak DateAndTime on an interpreter VM: { #testNow->10143 . #testEquals->30986 . #testGreaterThan->80199 . #testLessThan->75912 . #testPrintString->10429 . #testStringAsDateAndTime->44657 }

LXTestDateAndTimePerformance test results using the new UTC based DateAndTime on an interpreter VM: { #testNow->6423 . #testEquals->31625 . #testGreaterThan->22999 . #testLessThan->18514 . #testPrintString->12502 . #testStringAsDateAndTime->32912 }

(CC to Brent Pinkney, author of the excellent Squeak Chronology package)

Dave

----------------------------------------------------------- Louis LaBrunda Keystone Software Corp. SkypeMe callto://PhotonDemon mailto:Lou@Keystone-Software.com http://www.Keystone-Software.com

On 26.05.2014, at 17:16, David T. Lewis lewis@mail.msen.com wrote:

On Mon, May 26, 2014 at 10:48:16AM -0400, Louis LaBrunda wrote:

...

On Sun, 25 May 2014 13:48:44 -0400, "David T. Lewis" lewis@mail.msen.com wrote:

...

I have been working on a variation of class DateAndTime that replaces its instance variables (seconds offset jdn nanos) with two instance variables, utcMicroseconds to represent microseconds elapsed since the Posix epoch, and localOffsetSeconds to represent the local time zone offset. When instantiating the time now, A single call primitiveUtcWithOffset is used to obtain these two values atomically as reported by the underlying platform.

There are several advantages to this representation of DateAndTime, the most important of which is that its magnitude is unambiguous regardless of daylight savings transitions in local time zones.

Hi Dave,

May I respectfully ask why localOffsetSeconds (to represent the local time zone offset) is needed? It seems to me a UTC time is enough. Is there really a need for the timezone offset the instance was created in? Does every DateAndTime instance need to carry this offset around with it? I would think the offset is only needed if one wants to display a date/time as a local value and then one could get the local offset from the VM or a program setting the user had previously supplied regardless of where the computer was setup to run. I guess there might be some historic interest as to the timezone an instances (or many instances) was created in but one could just keep that as a separate value.

Hi Lou,

Good question. In fact, one of the reasons I like the UTC implementation is that it helps clarify the two main responsibilities of DateAndTime. One is to represent time as a magnitude (for duration calculation, etc). The other is to display time in the frame of reference of a local time zone. It is not at all clear to me that those two responsibilies belong in the same class.

Dave

We need to be able to distinguish between local and universal time. It would be rather inconvenient if asking a DateAndTime for e.g. the hour would not be made to answer the local hour. Arguably the local time offset could be moved to a subclass, but having a single DateAndTime class is appealing for simplicity reasons, too.

- Bert -

On Mon, May 26, 2014 at 11:12 AM, Louis LaBrunda Lou@keystone-software.com wrote:

On Mon, 26 May 2014 17:29:19 +0200, Bert Freudenberg bert@freudenbergs.de wrote:

...

On 26.05.2014, at 17:16, David T. Lewis lewis@mail.msen.com wrote:

...

On Mon, May 26, 2014 at 10:48:16AM -0400, Louis LaBrunda wrote:

...

On Sun, 25 May 2014 13:48:44 -0400, "David T. Lewis" lewis@mail.msen.com wrote:

...

I have been working on a variation of class DateAndTime that replaces its instance variables (seconds offset jdn nanos) with two instance variables, utcMicroseconds to represent microseconds elapsed since the Posix epoch, and localOffsetSeconds to represent the local time zone offset. When instantiating the time now, A single call primitiveUtcWithOffset is used to obtain these two values atomically as reported by the underlying platform.

There are several advantages to this representation of DateAndTime, the most important of which is that its magnitude is unambiguous regardless of daylight savings transitions in local time zones.

Hi Dave,

May I respectfully ask why localOffsetSeconds (to represent the local time zone offset) is needed? It seems to me a UTC time is enough. Is there really a need for the timezone offset the instance was created in? Does every DateAndTime instance need to carry this offset around with it? I would think the offset is only needed if one wants to display a date/time as a local value and then one could get the local offset from the VM or a program setting the user had previously supplied regardless of where the computer was setup to run. I guess there might be some historic interest as to the timezone an instances (or many instances) was created in but one could just keep that as a separate value.

Hi Lou,

Good question. In fact, one of the reasons I like the UTC implementation is that it helps clarify the two main responsibilities of DateAndTime. One is to represent time as a magnitude (for duration calculation, etc). The other is to display time in the frame of reference of a local time zone. It is not at all clear to me that those two responsibilies belong in the same class.

Dave

We need to be able to distinguish between local and universal time. It would be rather inconvenient if asking a DateAndTime for e.g. the hour would not be made to answer the local hour. Arguably the local time offset could be moved to a subclass, but having a single DateAndTime class is appealing for simplicity reasons, too.

• Bert -

I guess a DateAndTime class (without offset) could always answer 0 for the offset and a DateAndTimeWithOffset subclass could carry and answer the offset. Methods could be provided to morph one into the other if desired.

No, one of the core requirements of a DateAndTime has always been to be able to answer the local time. It's fine for its internal representation to be in UTC, but that requirement cannot go away.

If you want all UTC DateAndTime's then just specify an offset of 0.

David T. Lewis wrote

the two main responsibilities of DateAndTime. One is to represent time as a magnitude (for duration calculation, etc). The other is to display time in the frame of reference of a local time zone. It is not at all clear to me that those two responsibilies belong in the same class.

There is definitely something fishy with the combination. Consider Pharo's Date, which retroactively applies the current timezone to dates. This is bad because e.g.: '7/10/2014' asDate "evaluated the day before DST" ~= '7/10/2014' asDate "evaluated the day after DST"

I created an issue, which has been sitting on the bug tracker. Here's an excerpt from the discussion (https://pharo.fogbugz.com/default.asp?12147#BugEvent.90264):

I see that a particular location could reasonably be implied in a Date, e.g. if we're saying that a treaty was signed on such and such a date, we mean at the place where it was signed... it may have already been the next day in another time zone.

Now, what is beyond doubt, is that the timezone that would be implied is the timezone of the date itself, not an artifact of when the instance was created. In my example, I create '11/12/2013' asDate, and it assumes my timezone. Since it's the summer in NY, that means -4 offset. Now, I evaluate '11/12/2013' asDate again in December. Again it implies my timezone, the offset of which has changed to -5! So now '11/12/2013' asDate (in NY) ~= '11/12/2013' asDate (in NY). Even though the offset of the date in question, which depends only on whether DST was in effect on 11/12/2013, has not changed.

So the current behavior of two equal dates being considered not-equal is as jarring as your counter example of two non-equal dates (different timezone) being considered equal.

There doesn't seem to be an easy, direct solution because I'm assuming we don't have a primitive for "the offset at this location on X date in the past". So the next best thing might be a PlatonicDate (obviously a name-in-progress :)) that represents the abstract idea of a Date without regard to location, and then an object to connect a particular place/offset to that concept (but the second one would have the same problem as we currently have).

----- Cheers, Sean -- View this message in context: http://forum.world.st/What-s-up-on-build-squeak-org-tp4760266p4774978.html Sent from the Squeak - Dev mailing list archive at Nabble.com.

...

So the current behavior of two equal dates being considered not-equal is as jarring as your counter example of two non-equal dates (different timezone) being considered equal.

Indeed!

There doesn't seem to be an easy, direct solution because I'm assuming we don't have a primitive for "the offset at this location on X date in the past". So the next best thing might be a PlatonicDate (obviously a name-in-progress :)) that represents the abstract idea of a Date without regard to location, and then an object to connect a particular place/offset to that concept (but the second one would have the same problem as we currently have).

Well, for me at least, the solution we chose for Squeak has been working well. Here's the main discussion:

http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-February/162872....

Hi Dave, as someone who works with large systems in Squeak, I'm always interested in _storage efficiency_ as much as execution efficiency.

DateAndTime, in particular, is a very common domain element with a high potential for there to be many millions of instances in a given domain model.

Apps which have millions of objects with merely a Date attribute can canonicalize them. And, apps which have millions of Time objects can canonicalize them.

But LargeInteger's are not easy to canonicalize (e.g., utcMicroseconds). So a database system with millions of DateAndTime's would have to do _two_ reads for every DateAndTime instance instead of just one today (because SmallIntegers are immediate, while LargeIntegers require their own storage buffer).

One thing I really like about the current implementation of DateAndTime is how it carefully avoids LargeIntegers by having large-grained "platforms" to arrive at the current time. e.g., each 'jdn' is a chunk of (1000000*60*60*24) microseconds. Your new implementation reflects an increase of 86 BILLION utcMicroseconds for every 1 jdn.

Small, all-in-memory benchmarks may show faster with the LI, but I'm concerned that large-scale apps might be significantly impacted in the opposite way..

Would it be possible to re-optimize this part of the representation while still maintaining internal UTC represenation to solve your concern about daylight-savings?

Thanks.

On Sun, May 25, 2014 at 12:48 PM, David T. Lewis lewis@mail.msen.com wrote:

I have been working on a variation of class DateAndTime that replaces its instance variables (seconds offset jdn nanos) with two instance variables, utcMicroseconds to represent microseconds elapsed since the Posix epoch, and localOffsetSeconds to represent the local time zone offset. When instantiating the time now, A single call primitiveUtcWithOffset is used to obtain these two values atomically as reported by the underlying platform.

There are several advantages to this representation of DateAndTime, the most important of which is that its magnitude is unambiguous regardless of daylight savings transitions in local time zones.

This is my attempt to address some historical baggage in Squeak. The VM reports time related to the local time zone, and the image attempts to convert to UTC (sometimes incorrectly). A UTC based representation makes the implementation of time zone tables more straightforward (see for example the Olson time zone tables in TimeZoneDatabase on SqueakMap).

I am attaching the source code as a SAR file that can be loaded into a fully updated Squeak trunk image. The conversion process is slow, so be patient if you load it.

This can be run on either an intepreter VM or Cog, but if you use Cog, please use a version dated June 2013 or later (the VM in the Squeak 4.5 all-in-one is fine).

I am also attaching a copy of LXTestDateAndTimePerformance, which can be used to compare the performance of some basic DateAndTime functions.

Performance of the UTC based DateAndTime is generally favorable compared to the original. Here is what I see on my system (smaller numbers are better).

LXTestDateAndTimePerformance test results using the original Squeak DateAndTime on an interpreter VM: { #testNow->10143 . #testEquals->30986 . #testGreaterThan->80199 . #testLessThan->75912 . #testPrintString->10429 . #testStringAsDateAndTime->44657 }

LXTestDateAndTimePerformance test results using the new UTC based DateAndTime on an interpreter VM: { #testNow->6423 . #testEquals->31625 . #testGreaterThan->22999 . #testLessThan->18514 . #testPrintString->12502 . #testStringAsDateAndTime->32912 }

(CC to Brent Pinkney, author of the excellent Squeak Chronology package)

Dave

On Mon, May 26, 2014 at 01:09:06PM -0500, Chris Muller wrote:

Hi Dave, as someone who works with large systems in Squeak, I'm always interested in _storage efficiency_ as much as execution efficiency.

DateAndTime, in particular, is a very common domain element with a high potential for there to be many millions of instances in a given domain model.

Apps which have millions of objects with merely a Date attribute can canonicalize them. And, apps which have millions of Time objects can canonicalize them.

But LargeInteger's are not easy to canonicalize (e.g., utcMicroseconds). So a database system with millions of DateAndTime's would have to do _two_ reads for every DateAndTime instance instead of just one today (because SmallIntegers are immediate, while LargeIntegers require their own storage buffer).

One thing I really like about the current implementation of DateAndTime is how it carefully avoids LargeIntegers by having large-grained "platforms" to arrive at the current time. e.g., each 'jdn' is a chunk of (1000000*60*60*24) microseconds. Your new implementation reflects an increase of 86 BILLION utcMicroseconds for every 1 jdn.

Understood. But to clarify: The name "utcMicroseconds" reflects only the precision of the time scale, it is not meant to imply what kind of number is used to represent it. In fact, a DateAndTime with nanosecond precision will typically appear as a Fraction rather than a LargeInteger. But microsecond precision is what is currently reported by the primitives, so these are LargeInteger relative to the Posix epoch.

For saving to a database, you could certainly shift the time origin and/or limit the precision of the time representation. That's more or less with the current jnd/seconds/nanos does.

Small, all-in-memory benchmarks may show faster with the LI, but I'm concerned that large-scale apps might be significantly impacted in the opposite way..

Would it be possible to re-optimize this part of the representation while still maintaining internal UTC represenation to solve your concern about daylight-savings?

Sure, but just to clarify: This is not something that I am proposing for Squeak trunk. It is a follow up project to my TimeZoneDatabase that I have been meaning to do for the last 15 years. I finally got around to trying it, so I figured I'd go ahead and publish the code :)

Dave

Thanks.

On Sun, May 25, 2014 at 12:48 PM, David T. Lewis lewis@mail.msen.com wrote:

...

I have been working on a variation of class DateAndTime that replaces its instance variables (seconds offset jdn nanos) with two instance variables, utcMicroseconds to represent microseconds elapsed since the Posix epoch, and localOffsetSeconds to represent the local time zone offset. When instantiating the time now, A single call primitiveUtcWithOffset is used to obtain these two values atomically as reported by the underlying platform.

There are several advantages to this representation of DateAndTime, the most important of which is that its magnitude is unambiguous regardless of daylight savings transitions in local time zones.

This is my attempt to address some historical baggage in Squeak. The VM reports time related to the local time zone, and the image attempts to convert to UTC (sometimes incorrectly). A UTC based representation makes the implementation of time zone tables more straightforward (see for example the Olson time zone tables in TimeZoneDatabase on SqueakMap).

I am attaching the source code as a SAR file that can be loaded into a fully updated Squeak trunk image. The conversion process is slow, so be patient if you load it.

This can be run on either an intepreter VM or Cog, but if you use Cog, please use a version dated June 2013 or later (the VM in the Squeak 4.5 all-in-one is fine).

I am also attaching a copy of LXTestDateAndTimePerformance, which can be used to compare the performance of some basic DateAndTime functions.

Performance of the UTC based DateAndTime is generally favorable compared to the original. Here is what I see on my system (smaller numbers are better).

LXTestDateAndTimePerformance test results using the original Squeak DateAndTime on an interpreter VM: { #testNow->10143 . #testEquals->30986 . #testGreaterThan->80199 . #testLessThan->75912 . #testPrintString->10429 . #testStringAsDateAndTime->44657 }

LXTestDateAndTimePerformance test results using the new UTC based DateAndTime on an interpreter VM: { #testNow->6423 . #testEquals->31625 . #testGreaterThan->22999 . #testLessThan->18514 . #testPrintString->12502 . #testStringAsDateAndTime->32912 }

(CC to Brent Pinkney, author of the excellent Squeak Chronology package)

Dave

The issue actually relates purely to Squeak domain models. Consider the case of an all-in-memory object model in Squeak, with no database involved at all. It is very feasible an app would want to import a flat-file dataset that involves creation a few million DateAndTime instances (along with other objects, of course) to the point where memory constraints begin to be noticed.

When dealing with this level of prolifigation-potential of a particular class, and for such a base data-type we don't want to endure changing again, I want us to strongly scrutinize the internal representation.

In this case, the use of 'utcMicroseconds' introduces a lot of duplicate bit-patterns in memory that are very hard, if not impossible, to share.

The simplest case are two equivalent instances of DateAndTime (read from separate files). Despite being equivalent, their utcMicroseconds' will be separate objects each consuming separate memory space. There is no easy way to share the same 'utcMicroseconds' instance between them.

But fully-equivalent DateAndTime's is not even half of the concern -- the high-order bits of every DateAndTime's 'utcMicroseconds' duplicates the same bit pattern, again and again, eating up memory.

That doesn't happen when the internal representations are, or can be, canonicalized, as in the case of using SmallIntegers. Yes, Brent's original representation requires two additional slots per instance, but the _contents_ of those slots are SmallIntegers -- shared memory.

On Mon, May 26, 2014 at 8:29 PM, David T. Lewis lewis@mail.msen.com wrote:

On Mon, May 26, 2014 at 01:09:06PM -0500, Chris Muller wrote:

...

Hi Dave, as someone who works with large systems in Squeak, I'm always interested in _storage efficiency_ as much as execution efficiency.

DateAndTime, in particular, is a very common domain element with a high potential for there to be many millions of instances in a given domain model.

Apps which have millions of objects with merely a Date attribute can canonicalize them. And, apps which have millions of Time objects can canonicalize them.

But LargeInteger's are not easy to canonicalize (e.g., utcMicroseconds). So a database system with millions of DateAndTime's would have to do _two_ reads for every DateAndTime instance instead of just one today (because SmallIntegers are immediate, while LargeIntegers require their own storage buffer).

Hi Chris,

I do not have a lot of experience with database systems, so I would like to better understand the issue for storage of large numeric values.

I was under the impression that modern SQL databases provide direct support for large integer data types (e.g. bigint for SQL server), and my assumption was that object databases such as Magma or GemStone would make this a non-issue. Why is it that a large (64 bit) integer should be any more or less difficult to persist than a small integer?

This may be a dumb question but I am curious.

Thanks, Dave

On Tue, May 27, 2014 at 09:55:33PM +0200, Nicolas Cellier wrote:

2014-05-27 4:30 GMT+02:00 Chris Muller ma.chris.m@gmail.com:

...

The issue actually relates purely to Squeak domain models. Consider the case of an all-in-memory object model in Squeak, with no database involved at all. It is very feasible an app would want to import a flat-file dataset that involves creation a few million DateAndTime instances (along with other objects, of course) to the point where memory constraints begin to be noticed.

When dealing with this level of prolifigation-potential of a particular class, and for such a base data-type we don't want to endure changing again, I want us to strongly scrutinize the internal representation.

In this case, the use of 'utcMicroseconds' introduces a lot of duplicate bit-patterns in memory that are very hard, if not impossible, to share.

The simplest case are two equivalent instances of DateAndTime (read from separate files). Despite being equivalent, their utcMicroseconds' will be separate objects each consuming separate memory space. There is no easy way to share the same 'utcMicroseconds' instance between them.

But fully-equivalent DateAndTime's is not even half of the concern -- the high-order bits of every DateAndTime's 'utcMicroseconds' duplicates the same bit pattern, again and again, eating up memory.

That doesn't happen when the internal representations are, or can be, canonicalized, as in the case of using SmallIntegers. Yes, Brent's original representation requires two additional slots per instance, but the _contents_ of those slots are SmallIntegers -- shared memory.

Well, in current 32 bit image format, SmallInteger are not exactly shared, they are immediate values. Each consumes exactly 32 bits.

For a compact class like LargePosOrNegInteger, I don't remember what is the header size exactly, but you get 64 bits for data, I would be surprised to see a major difference wrt consumed memory.

Smalltalk compactClassesArray includes: DateAndTime ==> false Smalltalk compactClassesArray includes: LargePositiveInteger ==> true

So for the traditional DateAndTime implementation, an instance requires:

2 words of header (64 bits) 3 words for the small integer jdn/seconds/nanos variables 1 word for the pointer to the offset object, which is an instance of Duration

In practice, most instances of DateAndTime within an image will share the same offset object, so for purposes of estimation assume that this takes no extra space.

Thus each instance requires 6 words of space in the object memory (maybe a bit more on average if the DateAndTime instances are not sharing the same Duration instance for one reason or another).

For the UTC based implementation of DateAndTime, each instance requires:

2 words of header 1 word for the small integer localOffsetSeconds variable 1 word for the pointer to the LargePositiveInteger representing utcMicroSeconds 1 word of header for the large positive integer 2 words of data for the value of the large positive integer

Thus each instance requires 7 words of space in the object memory.

So there is a difference, but it would probably not be a large effect on overall space utilization, even assuming complete sharing of the offset Duration instances.

Dave

It's probably possible to get it down to 3 words if DateAndTime were represented as one canonicalized 'date', one canonicalized 'time' (precise to the second), and one SmallInteger for millis or micros..

The more and higher-level parts a DateAndTime can be constructed with, the better the opportunity for memory-optimization. Conversely, the more an implementation moves more toward a 'binary data' representation, the fewer of those bits can be shared and, therefore, the more that will be duplicated across instances.

On Mon, Jun 2, 2014 at 2:21 PM, Levente Uzonyi leves@elte.hu wrote:

On Tue, 27 May 2014, David T. Lewis wrote:

...

On Tue, May 27, 2014 at 09:55:33PM +0200, Nicolas Cellier wrote:

...

2014-05-27 4:30 GMT+02:00 Chris Muller ma.chris.m@gmail.com:

...

The issue actually relates purely to Squeak domain models. Consider the case of an all-in-memory object model in Squeak, with no database involved at all. It is very feasible an app would want to import a flat-file dataset that involves creation a few million DateAndTime instances (along with other objects, of course) to the point where memory constraints begin to be noticed.

When dealing with this level of prolifigation-potential of a particular class, and for such a base data-type we don't want to endure changing again, I want us to strongly scrutinize the internal representation.

In this case, the use of 'utcMicroseconds' introduces a lot of duplicate bit-patterns in memory that are very hard, if not impossible, to share.

The simplest case are two equivalent instances of DateAndTime (read from separate files). Despite being equivalent, their utcMicroseconds' will be separate objects each consuming separate memory space. There is no easy way to share the same 'utcMicroseconds' instance between them.

But fully-equivalent DateAndTime's is not even half of the concern -- the high-order bits of every DateAndTime's 'utcMicroseconds' duplicates the same bit pattern, again and again, eating up memory.

That doesn't happen when the internal representations are, or can be, canonicalized, as in the case of using SmallIntegers. Yes, Brent's original representation requires two additional slots per instance, but the _contents_ of those slots are SmallIntegers -- shared memory.

Well, in current 32 bit image format, SmallInteger are not exactly shared, they are immediate values. Each consumes exactly 32 bits.

For a compact class like LargePosOrNegInteger, I don't remember what is the header size exactly, but you get 64 bits for data, I would be surprised to see a major difference wrt consumed memory.

Smalltalk compactClassesArray includes: DateAndTime ==> false Smalltalk compactClassesArray includes: LargePositiveInteger ==> true

So for the traditional DateAndTime implementation, an instance requires:

2 words of header (64 bits) 3 words for the small integer jdn/seconds/nanos variables 1 word for the pointer to the offset object, which is an instance of Duration

In practice, most instances of DateAndTime within an image will share the same offset object, so for purposes of estimation assume that this takes no extra space.

Thus each instance requires 6 words of space in the object memory (maybe a bit more on average if the DateAndTime instances are not sharing the same Duration instance for one reason or another).

For the UTC based implementation of DateAndTime, each instance requires:

2 words of header 1 word for the small integer localOffsetSeconds variable 1 word for the pointer to the LargePositiveInteger representing utcMicroSeconds 1 word of header for the large positive integer 2 words of data for the value of the large positive integer

Thus each instance requires 7 words of space in the object memory.

So there is a difference, but it would probably not be a large effect on overall space utilization, even assuming complete sharing of the offset Duration instances.

I think it's possible to reduce the number of words to 5 at the cost of reusing integer primitives. If DateAndTime is a variable byte class, then it can hold the utcMicroSeconds in 8 variable slots (2 words). I don't know if the LargeInteger primitives would work with it, but I think they should, so comparison and arithmetic methods could be based on them.

But it's probably not worth to care about this, because Spur will change these things.

Levente

...

Dave

On Mon, Jun 02, 2014 at 09:21:06PM +0200, Levente Uzonyi wrote:

On Tue, 27 May 2014, David T. Lewis wrote:

...

On Tue, May 27, 2014 at 09:55:33PM +0200, Nicolas Cellier wrote:

...

2014-05-27 4:30 GMT+02:00 Chris Muller ma.chris.m@gmail.com:

...

The issue actually relates purely to Squeak domain models. Consider the case of an all-in-memory object model in Squeak, with no database involved at all. It is very feasible an app would want to import a flat-file dataset that involves creation a few million DateAndTime instances (along with other objects, of course) to the point where memory constraints begin to be noticed.

When dealing with this level of prolifigation-potential of a particular class, and for such a base data-type we don't want to endure changing again, I want us to strongly scrutinize the internal representation.

In this case, the use of 'utcMicroseconds' introduces a lot of duplicate bit-patterns in memory that are very hard, if not impossible, to share.

The simplest case are two equivalent instances of DateAndTime (read from separate files). Despite being equivalent, their utcMicroseconds' will be separate objects each consuming separate memory space. There is no easy way to share the same 'utcMicroseconds' instance between them.

But fully-equivalent DateAndTime's is not even half of the concern -- the high-order bits of every DateAndTime's 'utcMicroseconds' duplicates the same bit pattern, again and again, eating up memory.

That doesn't happen when the internal representations are, or can be, canonicalized, as in the case of using SmallIntegers. Yes, Brent's original representation requires two additional slots per instance, but the _contents_ of those slots are SmallIntegers -- shared memory.

Well, in current 32 bit image format, SmallInteger are not exactly shared, they are immediate values. Each consumes exactly 32 bits.

For a compact class like LargePosOrNegInteger, I don't remember what is the header size exactly, but you get 64 bits for data, I would be surprised to see a major difference wrt consumed memory.

Smalltalk compactClassesArray includes: DateAndTime ==> false Smalltalk compactClassesArray includes: LargePositiveInteger ==> true

So for the traditional DateAndTime implementation, an instance requires:

2 words of header (64 bits) 3 words for the small integer jdn/seconds/nanos variables 1 word for the pointer to the offset object, which is an instance of Duration

In practice, most instances of DateAndTime within an image will share the same offset object, so for purposes of estimation assume that this takes no extra space.

Thus each instance requires 6 words of space in the object memory (maybe a bit more on average if the DateAndTime instances are not sharing the same Duration instance for one reason or another).

For the UTC based implementation of DateAndTime, each instance requires:

2 words of header 1 word for the small integer localOffsetSeconds variable 1 word for the pointer to the LargePositiveInteger representing utcMicroSeconds 1 word of header for the large positive integer 2 words of data for the value of the large positive integer

Thus each instance requires 7 words of space in the object memory.

So there is a difference, but it would probably not be a large effect on overall space utilization, even assuming complete sharing of the offset Duration instances.

I think it's possible to reduce the number of words to 5 at the cost of reusing integer primitives. If DateAndTime is a variable byte class, then it can hold the utcMicroSeconds in 8 variable slots (2 words). I don't know if the LargeInteger primitives would work with it, but I think they should, so comparison and arithmetic methods could be based on them.

This probably would work, but I don't think that it would be a good thing to do. The "microseconds" in the variable name is intended to indicate the scale, not the actual numeric representation. For example, It would be a Fraction in the case of parsing a DateAndTime with nanosecond precision from a string.

But it's probably not worth to care about this, because Spur will change these things.

Yes, for sure.

Dave

On Wed, May 28, 2014 at 11:49:08AM +0200, Bert Freudenberg wrote:

On 25.05.2014, at 19:48, David T. Lewis lewis@mail.msen.com wrote:

...

Performance of the UTC based DateAndTime is generally favorable compared to the original. Here is what I see on my system (smaller numbers are better).

LXTestDateAndTimePerformance test results using the original Squeak DateAndTime on an interpreter VM: { #testNow->10143 . #testEquals->30986 . #testGreaterThan->80199 . #testLessThan->75912 . #testPrintString->10429 . #testStringAsDateAndTime->44657 }

LXTestDateAndTimePerformance test results using the new UTC based DateAndTime on an interpreter VM: { #testNow->6423 . #testEquals->31625 . #testGreaterThan->22999 . #testLessThan->18514 . #testPrintString->12502 . #testStringAsDateAndTime->32912 }

Hi Dave,

just curious: did you test the performance without the LargeInt primitives?

• Bert -

For the new UTC implementation with the primitive disabled, fallback code is much slower for DateAndTime class>>now.

{ #testNow->36939 . #testEquals->29015 . #testGreaterThan->21142 . #testLessThan->17586 . #testPrintString->11809 . #testStringAsDateAndTime->30918 }

Dave

On Wed, May 28, 2014 at 11:49:08AM +0200, Bert Freudenberg wrote:

On 25.05.2014, at 19:48, David T. Lewis lewis@mail.msen.com wrote:

...

Performance of the UTC based DateAndTime is generally favorable compared to the original. Here is what I see on my system (smaller numbers are better).

LXTestDateAndTimePerformance test results using the original Squeak DateAndTime on an interpreter VM: { #testNow->10143 . #testEquals->30986 . #testGreaterThan->80199 . #testLessThan->75912 . #testPrintString->10429 . #testStringAsDateAndTime->44657 }

LXTestDateAndTimePerformance test results using the new UTC based DateAndTime on an interpreter VM: { #testNow->6423 . #testEquals->31625 . #testGreaterThan->22999 . #testLessThan->18514 . #testPrintString->12502 . #testStringAsDateAndTime->32912 }

Hi Dave,

just curious: did you test the performance without the LargeInt primitives?

• Bert -

Hi Bert,

Sorry for the extremely late reply. I had never tried the UTC DateAndTime without a LargeIntegersPlugin, but I realize now that the question might be important for the SqueakJS VM, so a ran some tests using an interpreter VM with and without LargeIntegersPlugin.

The results were not at all what I expected. Very surprisingly (at least to me), the plugin seems to make very little difference in the performance of the UTC DateAndTime. The UTC DateAndTime seems to be a bit slower on printString and equality testing (?), and quite a bit faster on the other measures. But the LargeIntegersPlugin does not seem to make much of a difference at all.

Here is what I saw on my machine today.

Standard DateAndTime, standard VM: LXTestDateAndTimePerformance new test. ==> { #testNow->9147 . #testEquals->28252 . #testGreaterThan->27358 . #testLessThan->24664 . #testPrintString->10955 . #testStringAsDateAndTime->48505 }

Standard DateAndTime, no LargeIntegersPlugin: LXTestDateAndTimePerformance new test. ==> { #testNow->28324 . #testEquals->27985 . #testGreaterThan->27837 . #testLessThan->23783 . #testPrintString->11321 . #testStringAsDateAndTime->46679 }

UTC DateAndTime, standard VM: LXTestDateAndTimePerformance new test. ==> { #testNow->5807 . #testEquals->28168 . #testGreaterThan->19764 . #testLessThan->16117 . #testPrintString->13212 . #testStringAsDateAndTime->36652 }

UTC DateAndTime, no LargeIntegersPlugin: LXTestDateAndTimePerformance new test. ==> { #testNow->5759 . #testEquals->30079 . #testGreaterThan->19934 . #testLessThan->16315 . #testPrintString->14125 . #testStringAsDateAndTime->38208 }

On Sun, May 25, 2014 at 01:48:44PM -0400, David T. Lewis wrote:

I have been working on a variation of class DateAndTime that replaces its instance variables (seconds offset jdn nanos) with two instance variables, utcMicroseconds to represent microseconds elapsed since the Posix epoch, and localOffsetSeconds to represent the local time zone offset. When instantiating the time now, A single call primitiveUtcWithOffset is used to obtain these two values atomically as reported by the underlying platform.

I originally posted this as email to squeak-dev a couple of months ago. I have now added a page for it on the swiki at http://wiki.squeak.org/squeak/6197 with the original SAR file attached. I have also put the code into Monticello on SS3 at http://ss3.gemstone.com/ss/UTCDateAndTime. The SS3 repository has a few minor updates since my original posting.

The SS3 archive has MCM update maps that, if loaded in sequence, can be used to update a Squeak trunk image from the repository.

I'll probably put this on SqueakMap also, if I can find a nice way to do the the updates for an arbitrary image. This is a tricky business, because instances of DateAndTime need to change at the same time that the Monticello loader is making heavy use of DateAndTime to load the MCZs. A good old fashioned SAR is the safest way to handle it, but Monticello is probably more approachable for most of us.

Dave

There are several advantages to this representation of DateAndTime, the most important of which is that its magnitude is unambiguous regardless of daylight savings transitions in local time zones.

This is my attempt to address some historical baggage in Squeak. The VM reports time related to the local time zone, and the image attempts to convert to UTC (sometimes incorrectly). A UTC based representation makes the implementation of time zone tables more straightforward (see for example the Olson time zone tables in TimeZoneDatabase on SqueakMap).

I am attaching the source code as a SAR file that can be loaded into a fully updated Squeak trunk image. The conversion process is slow, so be patient if you load it.

This can be run on either an intepreter VM or Cog, but if you use Cog, please use a version dated June 2013 or later (the VM in the Squeak 4.5 all-in-one is fine).

I am also attaching a copy of LXTestDateAndTimePerformance, which can be used to compare the performance of some basic DateAndTime functions.

Performance of the UTC based DateAndTime is generally favorable compared to the original. Here is what I see on my system (smaller numbers are better).

LXTestDateAndTimePerformance test results using the original Squeak DateAndTime on an interpreter VM: { #testNow->10143 . #testEquals->30986 . #testGreaterThan->80199 . #testLessThan->75912 . #testPrintString->10429 . #testStringAsDateAndTime->44657 }

LXTestDateAndTimePerformance test results using the new UTC based DateAndTime on an interpreter VM: { #testNow->6423 . #testEquals->31625 . #testGreaterThan->22999 . #testLessThan->18514 . #testPrintString->12502 . #testStringAsDateAndTime->32912 }

(CC to Brent Pinkney, author of the excellent Squeak Chronology package)

Dave

On 25.05.2014, at 22:43, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:

2014-05-25 18:34 GMT+02:00 Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:

2014-05-25 2:39 GMT+02:00 Chris Muller asqueaker@gmail.com:

Hi, thanks for noticing and investigating this! Seeing this now, I think we should signal an Error rather than a Warning to be more TestCase friendly but also because persisting empty packages is so painful, it should be an error, hands down. Better to force a resolution to the issue than silently persist modules of future-pain...

In any case, I think you accidently uncovered bugs in Tests-Monticello. I'm looking at a fix for a few hours, and it's really messy.

Hooray! After publishing Tests-nice.297 there is now a trunk CI job that finished http://build.squeak.org/job/SqueakTrunk/857/

Now we can see we have some regressions... How could we live without CI before?

Of course, for dissecting which change introduced which regression after a 2 month interrupt, that's going to be more pain than necessary...

To not make that happen again, can we make the jenkins mail squeak-dev on important (read me, not all) information? Like: more tests fail errors (possibly one nightly information)

best -tobias

On 28 May 2014 21:06, Frank Shearar frank.shearar@gmail.com wrote:

On 25 May 2014 21:52, Tobias Pape Das.Linux@gmx.de wrote:

...

On 25.05.2014, at 22:43, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:

...

2014-05-25 18:34 GMT+02:00 Nicolas Cellier nicolas.cellier.aka.nice@gmail.com:

2014-05-25 2:39 GMT+02:00 Chris Muller asqueaker@gmail.com:

Hi, thanks for noticing and investigating this! Seeing this now, I think we should signal an Error rather than a Warning to be more TestCase friendly but also because persisting empty packages is so painful, it should be an error, hands down. Better to force a resolution to the issue than silently persist modules of future-pain...

In any case, I think you accidently uncovered bugs in Tests-Monticello. I'm looking at a fix for a few hours, and it's really messy.

Hooray! After publishing Tests-nice.297 there is now a trunk CI job that finished http://build.squeak.org/job/SqueakTrunk/857/

Now we can see we have some regressions... How could we live without CI before?

Of course, for dissecting which change introduced which regression after a 2 month interrupt, that's going to be more pain than necessary...

To not make that happen again, can we make the jenkins mail squeak-dev on important (read me, not all) information? Like: more tests fail errors (possibly one nightly information)

We possibly could, but I'd suggest _not_ doing so because even without Nicolas's work (thanks very much, Nicolas!) we _normally_ have failing tests, which means that the normal status of the SqueakTrunk build would not be green.

Really, the ideal place to be is for the job to usually be green, because then CI saying anything at all is cause for alarm, not yet another false positive. And we're just not there, on a number of fronts.

I should add that it doesn't help that build agents that run for a long time - Java processes - eventually run out of PermGen space and stop working. I've pinged the owner of two of the slaves - thanks, Tony! - and we're tentatively trying out a hack so disgusting I'm not going to talk about it.

frank

...

best -tobias