<div dir="ltr">Hi Christoph,<div><br></div><div>We talked about this in the board meeting today and the general consensus was that it should rather be distributed via the update stream than relying on the download from a third party website. A few hundred kilobytes are not expected to be much trouble.</div><div><br></div><div>We did not specifically talk about whether the release should wait for further changes here... when in doubt, probably not. Of course, if you do get it done in the next few days, it would be welcome, but I do not think that any pressure is warranted. :-)</div><div><br></div><div>Regarding the storage efficiency... Do you really need to store the whole UnicodeData.txt in the image or in Monticello? For example, what I had in mind on April 5 was like: let parseUnicodeData: generate a new initialize method rather than populate the GeneralCategory table directly. The generated initialize method would "hardcode" the GeneralCategory table based on the parsed file. Then you could commit that generated method and you would not need to keep the file in the package. That might reduce the size that Monticello has to deal with.</div><div><br></div><div>Kind regards,<br>Jakob</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Fr., 20. Mai 2022 um 14:50 Uhr schrieb Thiede, Christoph <<a href="mailto:Christoph.Thiede@student.hpi.uni-potsdam.de" target="_blank">Christoph.Thiede@student.hpi.uni-potsdam.de</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div id="gmail-m_6808335383074496777gmail-m_-2829760355532042190divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p>Hi Jakob, hi all,</p>
<p><br>
</p>
<p>I can see your point ... Given the current efficiency of Monticello (IMO less storage- and more transmission-related efficiency), I would hesitate to store the Unicode data in a package. But we could decide on running the database update indeed via the update
stream in situations (maybe wrap it into an exception handler so that server images behind a firewall do not get in trouble).</p>
<p><br>
</p>
<p>If you deem this important enough, we can also pin the version of the downloaded Unicode data (i.e., <a href="https://www.unicode.org/Public/14.0.0/ucd/UnicodeData.txt" id="gmail-m_6808335383074496777gmail-m_-2829760355532042190LPlnk972374" target="_blank">https://www.unicode.org/Public/14.0.0/ucd/UnicodeData.txt</a> instead
of <a href="https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt" id="gmail-m_6808335383074496777gmail-m_-2829760355532042190LPlnk817652" target="_blank">https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt</a>). But then we would need a process for updating the version
number ... Maybe I could write a reminder test for this ...</p>
<p><br>
</p>
<p>I don't know. Decide you (board) what is the right option and I will give my best to implement it. Please also decide whether this still should go into the current beta version. :-)</p>
<p><br>
</p>
<p>Best,</p>
<p>Christoph</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_6808335383074496777gmail-m_-2829760355532042190divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>Von:</b> Squeak-dev <<a href="mailto:squeak-dev-bounces@lists.squeakfoundation.org" target="_blank">squeak-dev-bounces@lists.squeakfoundation.org</a>> im Auftrag von Jakob Reschke <<a href="mailto:jakres%2Bsqueak@gmail.com" target="_blank">jakres+squeak@gmail.com</a>><br>
<b>Gesendet:</b> Sonntag, 8. Mai 2022 12:55:36<br>
<b>An:</b> The general-purpose Squeak developers list<br>
<b>Betreff:</b> Re: [squeak-dev] Unicode</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">I want to retract the batteries not included argument, because I conflated the timezones situation too much with the Unicode situation. For the Unicode, there is already something in the image, although it is outdated.<br>
</div>
<div dir="ltr"><br>
</div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Am So., 8. Mai 2022 um 12:23 Uhr schrieb Jakob Reschke <<a href="mailto:jakres%2Bsqueak@gmail.com" target="_blank">jakres+squeak@gmail.com</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">Also it is kind of inconsistent: on the one hand Squeak does not want to rely on foreign libraries (via FFI) by default, and have everything in Smalltalk or within the image, on the other hand it loads basic databases from the outside... Batteries
not included.</div>
<div dir="auto"><br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div> </div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto"><br>
<br>
<div class="gmail_quote" dir="auto">
<div dir="ltr" class="gmail_attr">Thiede, Christoph <<a href="mailto:Christoph.Thiede@student.hpi.uni-potsdam.de" target="_blank">Christoph.Thiede@student.hpi.uni-potsdam.de</a>> schrieb am Do., 5. Mai 2022, 19:24:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div id="gmail-m_6808335383074496777gmail-m_-2829760355532042190gmail-m_-9176390380828309032m_-5949262845342413945divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p>Hi Jakob, Hi Marcel,</p>
<p><br>
</p>
<p>the advantage of your proposed solution is that we would have more control over the process.</p>
<p>The disadvantage is that it would increase the package size and tangle logic + data together. At least, we're talking about ~300 kB for the Unicode data if I used SpaceTally correctly. :-)</p>
<p><br>
</p>
<p>Personally, I would prefer to stay with the existing practice because updating your Unicode data locally really seems to be optional at the moment.</p>
<p><br>
</p>
<p>Best,</p>
<p>Christoph</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_6808335383074496777gmail-m_-2829760355532042190gmail-m_-9176390380828309032m_-5949262845342413945divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>Von:</b> Squeak-dev <<a href="mailto:squeak-dev-bounces@lists.squeakfoundation.org" rel="noreferrer" target="_blank">squeak-dev-bounces@lists.squeakfoundation.org</a>> im Auftrag von
Taeumel, Marcel<br>
<b>Gesendet:</b> Freitag, 8. April 2022 09:56:36<br>
<b>An:</b> squeak-dev<br>
<b>Betreff:</b> Re: [squeak-dev] Unicode</font>
<div> </div>
</div>
<div>
<div id="gmail-m_6808335383074496777gmail-m_-2829760355532042190gmail-m_-9176390380828309032m_-5949262845342413945__MailbirdStyleContent" style="font-size:10pt;font-family:Arial;color:rgb(0,0,0);text-align:left" dir="ltr">
Hi Jakob --
<div><br>
</div>
<div>> <span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">One could run the download&generate step as needed to update the data. (CI, release bubild, manually)</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">To integrate it as part of ReleaseBuilder class >> #prepareEnvironment would be my preferred way. Then it would be part of the CI.</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">Best,</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">Marcel</span></div>
<div></div>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-top:20px;margin-left:0px;padding-left:10px">
<p style="color:rgb(170,170,170);margin-top:10px">Am 05.04.2022 14:41:14 schrieb Jakob Reschke <<a href="mailto:jakres%2Bsqueak@gmail.com" rel="noreferrer" target="_blank">jakres+squeak@gmail.com</a>>:</p>
<div style="font-family:Arial,Helvetica,sans-serif">
<div dir="ltr">Would it be possible/practical to separate this?
<div>* Download and transform into code/objects</div>
<div>* Distribute the generated code/objects via the update stream directly</div>
<div>One could run the download&generate step as needed to update the data. (CI, release build, manually)</div>
<div>Or are there any reasons not to do that?</div>
<div><br>
</div>
<div>I was asking myself the same thing recently about the package that provides time zone information. It needs a Unix timezone database from the operating system to initialize, rather than providing Smalltalk objects/code directly in Monticello, based on
the official online database.</div>
<div><br>
</div>
<div>Kind regards,<br>
Jakob</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Am Di., 5. Apr. 2022 um 08:59 Uhr schrieb Marcel Taeumel <<a href="mailto:marcel.taeumel@hpi.de" rel="noreferrer" target="_blank">marcel.taeumel@hpi.de</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div id="gmail-m_6808335383074496777gmail-m_-2829760355532042190gmail-m_-9176390380828309032m_-5949262845342413945gmail-m_-998108584214244266__MailbirdStyleContent" style="font-size:10pt;font-family:Arial;color:rgb(0,0,0);text-align:left" dir="ltr">
Hi Eliot, hi Christoph --
<div><br>
</div>
<div>> <span style="color:rgb(0,0,0)">Unicode</span><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(0,0,128)">reinitializeData</span><span style="color:rgb(0,0,0)">.</span></div>
<div><span style="color:rgb(0,0,0)"><br>
</span></div>
<div><span style="color:rgb(0,0,0)">I think that this method has an unfortunate name. Since it downloads data from the Internet, it should be called #downloadAndInitializeData.</span></div>
<div><span style="color:rgb(0,0,0)"><br>
</span></div>
<div><span style="color:rgb(0,0,0)">And that's the reason for it not being in the post-load script. We might put the raw info there, but it would be very surprising if "Update Squeak" fetches data other than from
<a href="http://source.squeak.org" rel="noreferrer" target="_blank">source.squeak.org</a>.</span></div>
<div><span style="color:rgb(0,0,0)"><br>
</span></div>
<div><span style="color:rgb(0,0,0)">Best,</span></div>
<div><span style="color:rgb(0,0,0)">Marcel</span></div>
<div></div>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-top:20px;margin-left:0px;padding-left:10px;min-width:500px">
<p style="color:rgb(170,170,170);margin-top:10px">Am 04.04.2022 21:17:17 schrieb Thiede, Christoph <<a href="mailto:christoph.thiede@student.hpi.uni-potsdam.de" rel="noreferrer" target="_blank">christoph.thiede@student.hpi.uni-potsdam.de</a>>:</p>
<div style="font-family:Arial,Helvetica,sans-serif">
<div id="gmail-m_6808335383074496777gmail-m_-2829760355532042190gmail-m_-9176390380828309032m_-5949262845342413945gmail-m_-998108584214244266divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p>> <span>If this is essential</span></p>
<p><span><br>
</span></p>
<p><span>Well, how do you define essential? You can still use your image with the old font definitions. However, for some newer codepoints such as 😁😎😍, <span>Unicode generalCategoryLabelOf: and friends will answers "not assigned" without the upgrade. You
can watch the difference by browsing any comprehensive font in the FontImporter. But I am not aware of any code path that relies on the presence of newer Unicode data.</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Apart from that, I was already discussing with Marcel what would be the consequences of downloading data from a third-party server during an image update. There might be any images, most likely server images, that do not have free internet access
due to a strict firewall. Hypothetically, this might even introduce any security issues. So in the end, we decided on leaving this optional for now. It will only break if any future patch of any package relies on exact Unicode data.</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>Best,</span></span></p>
<p><span><span>Christoph</span></span></p>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_6808335383074496777gmail-m_-2829760355532042190gmail-m_-9176390380828309032m_-5949262845342413945gmail-m_-998108584214244266divRplyFwdMsg" dir="ltr">
<span style="font-family:Calibri,sans-serif;color:rgb(0,0,0)"><b>Von:</b> Squeak-dev <<a href="mailto:squeak-dev-bounces@lists.squeakfoundation.org" rel="noreferrer" target="_blank">squeak-dev-bounces@lists.squeakfoundation.org</a>> im Auftrag von Eliot Miranda
<<a href="mailto:eliot.miranda@gmail.com" rel="noreferrer" target="_blank">eliot.miranda@gmail.com</a>><br>
<b>Gesendet:</b> Montag, 4. April 2022 20:54:59<br>
<b>An:</b> The general-purpose Squeak developers list<br>
<b>Betreff:</b> Re: [squeak-dev] Unicode</span>
<div> </div>
</div>
<div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
<blockquote type="cite" style="min-width:500px">On Apr 4, 2022, at 11:20 AM, <a href="mailto:Christoph.Thiede@student.hpi.uni-potsdam.de" rel="noreferrer" target="_blank">
Christoph.Thiede@student.hpi.uni-potsdam.de</a> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite" style="min-width:500px">
<div dir="ltr">Merged via Multilingual-ct.271, Multilingual-ct.272, MultilingualTests-ct.41, and ReleaseBuilder-ct.231.<br>
<br>
Please run the following in your image to install the new Unicode data (and to uncover any regressions I may have missed :D):<br>
<br>
<span style="color:rgb(0,0,0)">Unicode</span><span style="color:rgb(0,0,0)"> </span>
<span style="color:rgb(0,0,128)">reinitializeData</span><span style="color:rgb(0,0,0)">.</span><br>
</div>
</blockquote>
<div><br>
</div>
If this is essential then it *must* be added as a post load script to one (or more) of the relevant packages. Asking “did you run <span style="color:rgb(0,0,0)">Unicode</span><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(0,0,128)">reinitializeData?”
when someone reports a strange bug isn’t acceptable.</span>
<div><br>
<blockquote type="cite" style="min-width:500px">
<div dir="ltr"><br>
Best,<br>
Christoph<br>
<br>
<span style="color:rgb(128,128,128)">---<br>
</span><span style="color:rgb(128,128,128)"><i>Sent from </i></span><span style="color:rgb(128,128,128)"><i><a href="https://github.com/hpi-swa-lab/squeak-inbox-talk" rel="noreferrer" target="_blank"><u><span style="color:rgb(128,128,128)">Squeak Inbox Talk</span></u></a></i></span></div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
<br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<br>
</blockquote></div>