On Mar 25, 2008, at 10:46 AM, Ties Stuij wrote:
To start with the known, I've been looking into the project loading times, and have managed to cut them in half by removing the gzipping on the project files and just using the .pr file in stead of the zip which claims to be a .pr. Now this might of course not be the perfect solution for all, I did uncover some practices on which I'd like either the opinion of Squeak people in general or that of the Etoys people in particular.
When I was building the Sophie storage subsystem for OLPC I noticed the OLPC "hard disk" uses a compressed file system. There is some technical notes on this suggesting developers do NOT zip files because this is pointless: Compress the file, then store on a compressed file system. On read then you use cpu time to uncompress the compressed file, then uncompress the file. Waste of CPU time.
As a compromise Sophie documents depending on the platform contain a zip file which contains uncompressed images and compressed text, or uncompressed images and text if we know the platform target is OLPC.
However I was not sure of the network impact of then passing uncompressed files around.
First, I noticed (together with Luke, who showed me the debugging ropes) that in the method asUnzippedStream, on ReadWriteStream, the method upToEndWithProgressBar uses '/' an awful lot to give the the right approximation to the progress bar. It sucked about 20 percent of the total loading time of an activity. So I changed upToEndWithProgressBar for upToEnd. Which seems like a more reasonable default 'cause I'd guess that usually one won't actually use the progress bar functionality.
The progress bars interfere with the actual time needed to perform the action, we had loading progress bars for Sophie documents but found for small documents the act of showing the progress bar and updating it could take 80% of the load time.
But thats not the end of it Etoys-wise. 'Cause it turns out an Etoys project file is gzipped twice. First the individual file in 'writeForExportWithSources:inDirectory:' and then the containing function writeForExportWithSources:inDirectory: as part of the .pr bundle. And this for a file that already has lots of it's contents compressed, seeing that most of the project file is taken up by jpegs. And this would ideally also be the case for sound files, but more on that later. The size decrease due to zipping in general is not more than 10%. So my question on this topic is: is there an argument for retaining the double zipping? And would there be a general need for a 'just save an uncompressed .pr file, instead of a bundle' option/patch?
See comment above about not zipping JPEGS/image data that is included in the ZIP file. We choose to store media (sound/video) in a separate media folder because it was pointless to unpack from a zip file on need. However this might not be possible when you want all the data in the zip file, still you could alter the code so that sound/video files aren't compressed by the zip logic when they are added to the zip file.
In Sophie we also have a ZIP file subclass because we found the ZIP storage class was rather inefficient for ZIP files with many members. For example when you lookup and manipulate a zip file member it scans an ordered list looking for the member by name. We altered that to use a dictionary.
So my concrete question is: how can one for once and for all delete a project instance. I was hoping to at least find some generic deleteInstance method in the image, but I couldn't find one (I'm a bit of a Squeak newbie).
There are programming tools in the image that allow you to see who is holding onto a project, I'll let someone else comment.
Sound format
Another strategy to trimming down the image size is using a compressed sound format. At the moment we use wav for sound right? If I'm wrong, excuse me, but I haven't had time to investigate the matter to thourougly. Anyways, since a number of activities have quite a few samples in them, ranging from short utterings to long sentences, it would be very worth while if we could use a compressed format. From the mailinglist I read rumours about Ogg plugin and a gstreamer plugin... Is it already possible to use Ogg files in Etoys? If so how? I'd already be happy with general directions.
http://tinlizzie.org/olpc/OggPlugin/
I am currently working on a gstreamer plugin for OLPC via funding from Viewpoints Research Institute, and have a target completion date for the end of the month. This plugin will let you build gstreamer pipeline chains and run them, right now you can build a pipleline to decode audio/video ogg files for example and send the video to a X11 window and the audio to the hardware sound player.
I also hope to have 'sinks' which let you import audio and video into Squeak.
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================