Hi Jakob,

thanks for sharing!

I did also spent some time and effort in processing pipermail/mailman:
https://github.com/hpi-swa/squeak-history [https://github.com/hpi-swa/squeak-history]


For Tweak, there is also manual scraping of (web.archive.org) HTML pages.  See #downloadTweak.

Best,
Marcel
Am 15.04.2021 00:38:48 schrieb Jakob Reschke <jakres+squeak at gmail.com>:
Here is my workspace for getting all inbox messages for a start. It downloads the whole archive, which is probably not a good idea for everyone to do it, and could be optimized simply because the inbox has not existed as long as the list. The expressions themselves are also not particularly optimized. The squeak-dev-snapshot.dat file is around 500 MB. My Squeak process takes 2 GB of RAM now though, so I will clear the workspace bindings before saving this image, which I normally use for something else...

wc := WebClient new.
base := 'http://lists.squeakfoundation.org/pipermail/squeak-dev/ [http://lists.squeakfoundation.org/pipermail/squeak-dev/]'.
r := wc httpGet: base.
c := r content.
m := '"([^"]+)"' asRegex.
files := c regex: '\<href="(([^.]+)\.txt(\.gz)?)"' matchesCollect: [:each | (m search: each) ifTrue: [m subexpression: 2]].
filedata := files collect: [:each | f := wc httpGet: base, each. f content].
deflated := Array new: filedata size streamContents: [:str | i := 0. filedata do: [:each | (each beginsWith: 'From') ifTrue: [str nextPut: each] ifFalse: [str nextPut: (GZipReadStream on: each) contents]] displayingProgress: [:each | i := i + 1. files at: i]].
start := String lf, 'From '. 'Splitting messages... ' displaySequentialProgress: [messages := deflated gather: [:eachFile | | which | which := deflated indexOf: eachFile. ProgressNotification signal: which / deflated size extra: (files at: which).  Array streamContents: [:str | | rstr | rstr := ReadStream on: eachFile. [rstr atEnd] whileFalse: [str nextPut: (rstr upToAll: start)]]]]. messages.
messages saveOnFileNamed: 'squeak-dev-snapshot.dat'.
"messages := (FileStream oldFileNamed: 'squeak-dev-messages.dat' do: [:str | str fileInObjectAndCode])."
filedata saveOnFileNamed: 'squeak-dev-files.dat'.
"filedata := (FileStream oldFileNamed: 'squeak-dev-files.dat' do: [:str | str fileInObjectAndCode])."
submissions := messages select: [:each | (each includesSubstring: String lf, 'Subject: [squeak-dev] The Inbox:')].


6304 submissions found this way. This number also includes replies, so it is not the number of Monticello versions ever submitted to the Inbox.
To do: to find threads, read the Message-ID: headers, follow the In-Reply-To:, References: headers. Group by original submission email. Or download the thread view from the web frontend, but must merge across month boundaries nevertheless. Correlate with Monticello versions. Display in a tool. Come up with a saner way of downloading from the mailing list and not doing everything eagerly.

Am Mi., 14. Apr. 2021 um 10:03 Uhr schrieb Marcel Taeumel <marcel.taeumel at hpi.de [mailto:marcel.taeumel at hpi.de]>:

Hi Nicolas.

> I may try to answer a mail that is about 6 month old, but IMO, it does
> not make much sense.
> This mail is not in my client anymore, so it means going thru some
> forum API with extra login etc...
> It's just not acceptable.

I find both ways manageable. It doesn't take too much time for me. Yet, I see the benefit of a platform with integrated tools such as GitHub.

It should be possible to write a small tool (or extension to Monticello) to make a comment (via e-mail?) on an inbox version using the mailing list and the matching entry there.  No need to move all the data to a new platform where the tools are --- just add the missing tools. IMO the better cost-value ratio. ... Any takers? Christoph? :-D

Best,
Marcel 
Am 13.04.2021 21:08:52 schrieb Nicolas Cellier <nicolas.cellier.aka.nice at gmail.com [mailto:nicolas.cellier.aka.nice at gmail.com]>:
Hi Christoph,
yes, I saw that later,
but then decided that your first attempt was OK, some kind of
integrator privilege ;)
If someone has a strong opinion about it, we can change it again, but
IMO it's not worth it.

The problem with our dev model is that it's been designed with very
short cycles in mind.
Once the contributions are rotting for more than a month or two, it's
very inconvenient to retrieve the messages and discussion from the
mailing list, while reviewing the changes in the image. I personally
skip this step because the process is already too demanding.
Unlike pull/merge requests in github/gitlab, we have no way to let a
review process span multiple months, multiple commits, multiple steps
(understand separate steps for loading into a live image like we do
with MCM update maps), etc...
Answering a mail in a cold thread (2 month later or more) does not
make sense,see an example below...

We also have to use a 3rd tool for moving the package: the
source.squak.org [http://source.squak.org] web interface.
I can guarantee that I have to re-login at each package review (logins
are short lived).
The way our site URL works, it means that I have to retrieve the page
holding the exact version at each re-login.
If 1st package letter is > M, it can be 10 clicks away...
Plus the fact that I'm pretty sure that in the past, some packages
that I previously moved to trunk or treated inbox did reappear in the
inbox, that's somehow discouraging, just put a picture of me and
Marcel faces here
https://en.wikipedia.org/wiki/Dana%C3%AFdes#/media/File:Danaides_by_John_William_Waterhouse,_1903.jpg [https://en.wikipedia.org/wiki/Dana%C3%AFdes#/media/File:Danaides_by_John_William_Waterhouse,_1903.jpg]
I'm generally satisfied with our lightweight model, except for the
review and integration of inbox contributions.

As example, yesterday, I wanted to reject
https://source.squeak.org/inbox/Kernel_ct.1366.diff [https://source.squeak.org/inbox/Kernel_ct.1366.diff]
This is because (result := Compiler evaluate:... ifFail: [^nil]) is
not the same as your proposal:
there is a local return that exit the method in the former case, and
an assignment of result with nil in the later case.
I can see the diff on our site, but cannot add any annotation unlike
github/gitlab.
I may try to answer a mail that is about 6 month old, but IMO, it does
not make much sense.
This mail is not in my client anymore, so it means going thru some
forum API with extra login etc...
It's just not acceptable.
Rejecting without a comment is not acceptable either.
It's not fair for contributors, and not fair to ourselves: the reason
why we reject is more important than the rejection itself: it is also
information reusable for future contributions.
If we really want to stick to our lghtweight model, i suggest that we
add one more commit message when we move some package to treated,
possibly with an automated link to original commit message in some
mailing list. This way, I will have a chance to give some additional
reason for rejection...

I stop the rant here. Above all, I don't want to restrain the will to
contribute.
Having enthusiastic users overwhelming the inbox with fixes, ideas and
enhancements is not the problem, it's a chance!
Having too few people to perform the review with poor tools is the
problem. We have to do something about it.
In my eyes, the reviews done in the mailing list a few month ago are
dead, they are not easily findable/usable/amendable.
Thanks for bringing the subject, and please, continue to chanllenge us!
Ideas for improving the process are wecome too.

Le mar. 13 avr. 2021 à 19:52, Christoph Thiede
a écrit :
>
> Hi Nicolas,
>
> actually, ST80-ct.255 should have been moved into the treated inbox in favor
> of ST80-ct.256 as discussed in the thread. Nevertheless, thanks for
> integrating all these patches! :-)
>
> Best,
> Christoph
>
>
>
> -----
> Carpe Squeak!
> --
> Sent from: http://forum.world.st/Squeak-Dev-f45488.html [http://forum.world.st/Squeak-Dev-f45488.html]
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20210415/4ce72603/attachment.html>