Chris,
Thanks for the response. I guess I could always create some sort of hashing algorithm to store the mp3 files in a multi-level directory structure (instead of a single directory as it is now) and then I could simply store the metadata information in Magma. This would give me two benefits:
1) The application that creates the mp3 files could work independently of Squeak/Magma and simply use the same hashing algorithm to store the media file in the OS file system directly and log somewhere the new medial file created. This would allow that part of the system to work independently and guarantee the media files will always be there regardless of the status of the Squeak VM and/or application 2) This would also allow me to overcome the stream transfer limitation of Magma as well as the 2GB file limit (if any) of Squeak. Then I could simply serve the media file using HTTP protocol when requested.
I know that in general OODBs are targeted for both complex object structures as well as large objects. I would hope the latter is something Magma can improve in the future.
In the meantime, I'm happy with Magma and will continue using it. Hopefully, I can put it all in production very soon.
BTW, any comments on the other issue from my original post?
Thanks, Daniel
On Aug 15, 2005, at 2:00 PM, Chris Muller wrote:
Hi Daniel,
... When a call comes in or is made, the system makes an entry in this table and stores an MP3 file in a directory of the system. What I was wondering if it would be applicable and make sense, is to store all this information in a Magma database. Meaning, store the call meta information as well as the actual audio content. At first, I don't think there would be a problem. However, I was looking at the current system and they have approximately 200GB of audio content stored in over 100,000 files (in a period of 6 months).
As additional requirements, the system should allow for the "easy" search of call records by either date, agent, or phone number. Also, the actual call "object" should be stored in the customer profile so that when reviewing the customer record, historical call recordings can be pulled. ...
I'm glad you brought this up because I've contemplated this very question before. While Magma's ability to build and provide access to an arbitrarily large object model includes large-and-flat collections of objects, it does not include large streams of bytes. This severely limits Magma's ability to work with large multimedia files.
While you could, theoretically, commit and retrieve up to a 16-meg byte object, it was never really intended to do "files" like this. A streaming protocol directly to the Magma server would be needed, i.e., something that could nextPut: bytes chunk-by-chunk *during the call* rather than requiring the client to buffer the whole thing and dump it all at once on the network to the server as soon as you hung up..
However, I was looking at the current system and they have approximately 200GB of audio content stored in over 100,000 files (in a period of 6 months).
Squeaks maximum file-size addressability is currently 2GB I think. So we see here that any potential implementation to support this would need to keep each file separate, rather than embedding it into the objects file (which contains the serialized bytes for every object in the database).
Perhaps Magma could emulate a "file-system" and provide access to certain objects as files too? i.e., what if the repository could be seen as a "drive" and you could create "directories" and save arbitrary files in them. The VTOC tree would be kept in the system-area of the repository..
On the other hand, I am hard-pressed to believe you would want to use this over the (surely faster) standard file-system provided by the OS.. I could add in a bunch of code to support this and then it might end up not meeting your needs exactly. For example, perhaps the mp3 files would need to be accessible by another application in the company? So it would be accessing the files inside the directory where the Magma database resides, which could cause contention..
So from that argument, it seems appropriate for each application to set up their own location in the real file-system and write their program to access the files from their own application based and referenced by location-objects from their own app..
I hope others on list will share comments..
- Chris