Brad,

   I understand your interest and frustration with sound but I think we need
to go further than just audio and video. Why? Because a lot of formats are
reused by various audio, video and still image formats. If we come up with a
framework which fits all these in, then we don't have to keep reinventing or
hacking existing classes to make them do what we want. 

 

I realize I am being vague so here is an example:

OGG files are made up of a container (OGG format) which usually contains one
or more Vorbis streams. However there is a small push to use OGM containers
which are OGG containers with AVI video and Vorbis audio streams. The
metadata of OGG or OGM files can have JPEG files in them. JPEG has metadata
which can be EXIF data showing camera information/who took the photo/etc.
AVI streams can be based on MPEG which is based on JPEG.

So you can see that once we have a basic framework, we can re-use classes to
fully support the variations that seem to pop-up all the time.

Now I see 2 parts to this, the "sinks" that play sound/video and the ability
to read, write and transform various formats into the native Squeak ones.
Then the tools you talk about sit on top of those fundamental classes.

 

Now one of my biggest concerns is that we should try to develop EVERYTHING
in Squeak with as few plugins as possible. I wrote the OGG decoder in "pure"
Smalltalk so it is possible. Of course by converting a small part of the
decoder to a plugin I sped everything up 10 fold but that came later ;) 

I think this is important to remove platform dependencies or dependencies on
external libraries. We may end up using them purely for performance but I
think we should have a goal of not using them in the first part.

Obviously the "sinks" would have to be plugins as they are very platform
specific.

 

Russell

 

  _____  

From: squeak-dev-bounces at lists.squeakfoundation.org
[mailto:squeak-dev-bounces at lists.squeakfoundation.org] On Behalf Of Brad
Fuller
Sent: Sunday, 20 February 2005 5:51 AM
To: The general-purpose Squeak developers list
Subject: Re: Improving Squeak's Mutlimedia

 

Thanks for the reply and interest, Martin. See below

Martin Kuball wrote: 

Am Friday 18 February 2005 21:09 schrieb Brad Fuller:
  

Is anyone interested in improving the multimedia facilities in
Squeak?
<snip>
    

I'm definitely interested. Unfortunately I'm not very good in writing 
long Mails. I will try to sum up my plans and ideas in the next 
couple of days. But here are some remarks  that come to my mind when 
reading your EMail.
 
What is Multimedia? Certainly it includes Audio and Video. But I think 
there is more.
  

Yeah, I didn't really explain, kinda vague.
I'm referring to only realtime full-motion video and sound/music only. I
wasn't refering to static graphics, text, etc.  The rest seems to be
addressed well by others. To me, both audio and video are in need of more
attention.



 
There are different levels or types of multimedia software. One type 
is concerned with the creation of new content. Another type deals 
with the manipulation of existing content. I think my AudioVideoLib 
ist an example for the latter type. Playback another.
  

Right. I think we could view this in layers:
* Low-level: Most of what is in the VM that addresses the platform-specific.
* Mid-level: Fundamental classes that connect to the VM and provide
fundamental 
   facilities for upper level "apps". This could probably be divided into
two.
* Upper-level: Applications for users - creation of content (like vegas,
premiere, 
   sound forge, protools) playback of content (like windows media player,
xmms) that make
   use of the fundamental classes.

(Perhaps it is not as clean a line as described here. But, if I remember
correctly, the sound recorder and sound playback are intertwined a bit and
several things are hard-coded (like sample-rate) that would be better as a
setting for the user. The fundamental recording/playback should be separate
from the user audio application. A while back, I've pulled these apart and
also made the AIF/WAV file access classes more general. But, it was more for
just my use and would like to make it more general for the image. Don't hold
me to this explanatin as the current state as I'm writing this from memory
of 6 months ago -- written here purely for explanation.)

Approaching the problems could be addressed in layers, as well. Perhaps
there are volunteers that would like to work on the VM some that work on the
fundamental classes, etc.




 
At the moment I'm trying to finish some missing parts of the AVI stuff 
and make everything more robust. Another important point is to really 
put it to use and find out the limits of what you can do with it 
(speed wise I mean) and to improve the API.
  

that could be helpful for many.



 
By the way I'm really excited about the things going on here at the 
moment. I hope we can move squeak to a new level in the next few 
month.
 
Martin
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/squeak-dev/attachments/20050220/2a9e470a/attachment.htm