Possible Uses of Magma

[Daniel Salama]

Chris,

Thanks for the response. I guess I could always create some sort of  
hashing algorithm to store the mp3 files in a multi-level directory  
structure (instead of a single directory as it is now) and then I  
could simply store the metadata information in Magma. This would give  
me two benefits:

1) The application that creates the mp3 files could work  
independently of Squeak/Magma and simply use the same hashing  
algorithm to store the media file in the OS file system directly and  
log somewhere the new medial file created. This would allow that part  
of the system to work independently and guarantee the media files  
will always be there regardless of the status of the Squeak VM and/or  
application
2) This would also allow me to overcome the stream transfer  
limitation of Magma as well as the 2GB file limit (if any) of Squeak.  
Then I could simply serve the media file using HTTP protocol when  
requested.

I know that in general OODBs are targeted for both complex object  
structures as well as large objects. I would hope the latter is  
something Magma can improve in the future.

In the meantime, I'm happy with Magma and will continue using it.  
Hopefully, I can put it all in production very soon.

BTW, any comments on the other issue from my original post?

Thanks,
Daniel

On Aug 15, 2005, at 2:00 PM, Chris Muller wrote:

>
> Hi Daniel,
>
>
>> ...
>> When a call comes in or is made, the system makes an entry in this
>> table and stores an MP3 file in a directory of the system. What I was
>> wondering if it would be applicable and make sense, is to store all
>> this information in a Magma database. Meaning, store the call meta
>> information as well as the actual audio content. At first, I don't
>> think there would be a problem. However, I was looking at the current
>> system and they have approximately 200GB of audio content stored in
>> over 100,000 files (in a period of 6 months).
>>
>> As additional requirements, the system should allow for the "easy"
>> search of call records by either date, agent, or phone number. Also,
>> the actual call "object" should be stored in the customer profile so
>> that when reviewing the customer record, historical call recordings
>> can be pulled.
>> ...
>>
>
> I'm glad you brought this up because I've contemplated this very  
> question
> before.  While Magma's ability to build and provide access to an  
> arbitrarily
> large object model includes large-and-flat collections of objects,  
> it does not
> include large streams of bytes.  This severely limits Magma's  
> ability to work
> with large multimedia files.
>
> While you could, theoretically, commit and retrieve up to a 16-meg  
> byte object,
> it was never really intended to do "files" like this.  A streaming  
> protocol
> directly to the Magma server would be needed, i.e., something that  
> could
> nextPut: bytes chunk-by-chunk *during the call* rather than  
> requiring the
> client to buffer the whole thing and dump it all at once on the  
> network to the
> server as soon as you hung up..
>
>
>> However, I was looking at the current
>> system and they have approximately 200GB of audio content stored in
>> over 100,000 files (in a period of 6 months).
>>
>
> Squeaks maximum file-size addressability is currently 2GB I think.   
> So we see
> here that any potential implementation to support this would need  
> to keep each
> file separate, rather than embedding it into the objects file  
> (which contains
> the serialized bytes for every object in the database).
>
> Perhaps Magma could emulate a "file-system" and provide access to  
> certain
> objects as files too?  i.e., what if the repository could be seen  
> as a "drive"
> and you could create "directories" and save arbitrary files in  
> them.  The VTOC
> tree would be kept in the system-area of the repository..
>
> On the other hand, I am hard-pressed to believe you would want to  
> use this over
> the (surely faster) standard file-system provided by the OS..  I  
> could add in a
> bunch of code to support this and then it might end up not meeting  
> your needs
> exactly.  For example, perhaps the mp3 files would need to be  
> accessible by
> another application in the company?  So it would be accessing the  
> files inside
> the directory where the Magma database resides, which could cause  
> contention..
>
> So from that argument, it seems appropriate for each application to  
> set up
> their own location in the real file-system and write their program  
> to access
> the files from their own application based and referenced by  
> location-objects
> from their own app..
>
> I hope others on list will share comments..
>
>  - Chris
>