Hi all,
Recently I've developed Sqnappy - Squeak/Pharo binding of the snappy compressor library. https://github.com/mumez/sqnappy
About snappy: https://code.google.com/p/snappy/
Sqnappy is easy to use. You can just send #compress:, #uncompress to SnappyCore: compressed := SnappyCore compress: data. uncompressed := SnappyCore uncompress: compressed.
With a simple test, Sqnappy was 10.8x faster than the existing GZipWriteStream/GZipReadStream. Additionally, Sqnappy implements snappy framing format, so that it can treat big data with a small memory allocation.
About framing format: https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
Tested from a workspace, 1.3 GB pg_dump file was compressed in around 10 seconds and decompressed in 6.5 seconds . No annoying GCs. It was comfortable.
Enjoy!
Very impressive! Especially if not creating additional garbage during compress or decompress (although I suppose the original input string would be garbage right? It doesn't output the compressed into the original String/ByteArray or does it?). I'm curious whether I could use this to reduce the sizes of Magma DB's -- if the decompression is fast enough then it shouldn't appreciably affect read performance.
Only downside is the need for the additional VM plug-in.
On Thu, Apr 25, 2013 at 9:49 PM, Masashi UMEZAWA masashi.umezawa@gmail.com wrote:
Hi all,
Recently I've developed Sqnappy - Squeak/Pharo binding of the snappy compressor library. https://github.com/mumez/sqnappy
About snappy: https://code.google.com/p/snappy/
Sqnappy is easy to use. You can just send #compress:, #uncompress to SnappyCore: compressed := SnappyCore compress: data. uncompressed := SnappyCore uncompress: compressed.
With a simple test, Sqnappy was 10.8x faster than the existing GZipWriteStream/GZipReadStream.
Additionally, Sqnappy implements snappy framing format, so that it can treat big data with a small memory allocation.
About framing format: https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
Tested from a workspace, 1.3 GB pg_dump file was compressed in around 10 seconds and decompressed in 6.5 seconds . No annoying GCs. It was comfortable.
Enjoy!
[:masashi | ^umezawa]
Hi,
Basically Compression/uncompression results are copied from the original, so there are some overheads. But if you use SnappyFraming (Stream API), copies are minimized until you retrieve the final result from the stream. Compression/uncompression are done in chunks and the same small buffer is reused while processing. So we can expect pretty linear performance.
LevelDB already utilizes snappy library for reducing its database size, so I think Magma with Sqnappy is a great idea.
Best regards,
2013/4/27 Chris Muller asqueaker@gmail.com:
Very impressive! Especially if not creating additional garbage during compress or decompress (although I suppose the original input string would be garbage right? It doesn't output the compressed into the original String/ByteArray or does it?). I'm curious whether I could use this to reduce the sizes of Magma DB's -- if the decompression is fast enough then it shouldn't appreciably affect read performance.
Only downside is the need for the additional VM plug-in.
On Thu, Apr 25, 2013 at 9:49 PM, Masashi UMEZAWA masashi.umezawa@gmail.com wrote:
Hi all,
Recently I've developed Sqnappy - Squeak/Pharo binding of the snappy compressor library. https://github.com/mumez/sqnappy
About snappy: https://code.google.com/p/snappy/
Sqnappy is easy to use. You can just send #compress:, #uncompress to SnappyCore: compressed := SnappyCore compress: data. uncompressed := SnappyCore uncompress: compressed.
With a simple test, Sqnappy was 10.8x faster than the existing GZipWriteStream/GZipReadStream.
Additionally, Sqnappy implements snappy framing format, so that it can treat big data with a small memory allocation.
About framing format: https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
Tested from a workspace, 1.3 GB pg_dump file was compressed in around 10 seconds and decompressed in 6.5 seconds . No annoying GCs. It was comfortable.
Enjoy!
[:masashi | ^umezawa]
On Thu, Apr 25, 2013 at 7:49 PM, Masashi UMEZAWA masashi.umezawa@gmail.comwrote:
Hi all,
Recently I've developed Sqnappy - Squeak/Pharo binding of the snappy compressor library. https://github.com/mumez/sqnappy
Bravo! Thanks for sharing!
Colin
squeak-dev@lists.squeakfoundation.org