Hi guys. I have the following scenario. I have a buffer (ByteArray) that I pass by FFI to a function of size N. This function puts data in the array and answers me the M number of bytes that it put. M <= N. Finally, I need to copy the array of size N to the accuare size M. To do that, I am using #copyFrom:to:. If the ByteArray is large (which could be the case), this function takes significant time because it needs to allocate space for the new large resulting array. So...is there a destructive primitive where I can "crop" the existing array, modify its size field and mark the remaining bytes as "free space for the heap".
Do we have a primitive for that?
Thanks,
-- Mariano http://marianopeck.wordpress.com
On 2012-11-08, at 16:22, Mariano Martinez Peck marianopeck@gmail.com wrote:
Hi guys. I have the following scenario. I have a buffer (ByteArray) that I pass by FFI to a function of size N. This function puts data in the array and answers me the M number of bytes that it put. M <= N. Finally, I need to copy the array of size N to the accuare size M. To do that, I am using #copyFrom:to:. If the ByteArray is large (which could be the case), this function takes significant time because it needs to allocate space for the new large resulting array. So...is there a destructive primitive where I can "crop" the existing array, modify its size field and mark the remaining bytes as "free space for the heap".
Do we have a primitive for that?
We do not have any primitive that changes an object's size.
However, if your problem is indeed the time for allocating the new array, then maybe there should be a primitive for that? E.g. one that copies a portion of one array to a new object. This would avoid having to initialize the memory of the new array - this is what's taking time, otherwise allocation is normally constant in time.
OTOH it seems unusual that you would have to use extremely large buffers where initialization time matters. E.g. if you were to use this for reading from a file or stream then it might make more sense to use many smaller buffers rather than a single huge one, no?
- Bert -
On Thu, Nov 8, 2012 at 4:38 PM, Bert Freudenberg bert@freudenbergs.de wrote:
On 2012-11-08, at 16:22, Mariano Martinez Peck marianopeck@gmail.com wrote:
Hi guys. I have the following scenario. I have a buffer (ByteArray) that I pass by FFI to a function of size N. This function puts data in the array and answers me the M number of bytes that it put. M <= N. Finally, I need to copy the array of size N to the accuare size M. To do that, I am using #copyFrom:to:. If the ByteArray is large (which could be the case), this function takes significant time because it needs to allocate space for the new large resulting array. So...is there a destructive primitive where I can "crop" the existing array, modify its size field and mark the remaining bytes as "free space for the heap".
Do we have a primitive for that?
We do not have any primitive that changes an object's size.
:(
However, if your problem is indeed the time for allocating the new array, then maybe there should be a primitive for that? E.g. one that copies a portion of one array to a new object. This would avoid having to initialize the memory of the new array - this is what's taking time, otherwise allocation is normally constant in time.
OTOH it seems unusual that you would have to use extremely large buffers where initialization time matters. E.g. if you were to use this for reading from a file or stream then it might make more sense to use many smaller buffers rather than a single huge one, no?
Hi Bert. I should have explained better. The library I am wrapping is LZ4, a fast compressor: http://code.google.com/p/lz4/ The function to compress looks like this:
int LZ4_compress (const char* source, char* dest, int isize);
/* LZ4_compress() : Compresses 'isize' bytes from 'source' into 'dest'. Destination buffer must be already allocated, and must be sized to handle worst cases situations (input data not compressible) Worst case size evaluation is provided by macro LZ4_compressBound()
isize : is the input size. Max supported value is ~1.9GB return : the number of bytes written in buffer dest
So, from the image side, I am doing (summarized):
dest := ByteArray new: (self funcCompressBound: aByteArray size) + 4. bytesCompressedOrError := self funcCompress: aByteArray byteArrayDestination: dest isize: aByteArray size maxOutputSize: maxOutputSize . compressed := dest copyFrom: 1 to: bytesCompressedOrError + 4.
But a normal scenario of compression is when the bytearray is indeed quite large. The function requires that "dest" is already allocated. And then I need to do the #copyFrom:to:, and that is what I was trying to avoid.
- Bert -
-- Mariano http://marianopeck.wordpress.com
On Thu, Nov 8, 2012 at 4:48 PM, Mariano Martinez Peck marianopeck@gmail.com wrote:
On Thu, Nov 8, 2012 at 4:38 PM, Bert Freudenberg bert@freudenbergs.de wrote:
On 2012-11-08, at 16:22, Mariano Martinez Peck marianopeck@gmail.com wrote:
Hi guys. I have the following scenario. I have a buffer (ByteArray) that I pass by FFI to a function of size N. This function puts data in the array and answers me the M number of bytes that it put. M <= N. Finally, I need to copy the array of size N to the accuare size M. To do that, I am using #copyFrom:to:. If the ByteArray is large (which could be the case), this function takes significant time because it needs to allocate space for the new large resulting array. So...is there a destructive primitive where I can "crop" the existing array, modify its size field and mark the remaining bytes as "free space for the heap".
Do we have a primitive for that?
We do not have any primitive that changes an object's size.
:(
However, if your problem is indeed the time for allocating the new array, then maybe there should be a primitive for that? E.g. one that copies a portion of one array to a new object. This would avoid having to initialize the memory of the new array - this is what's taking time, otherwise allocation is normally constant in time.
OTOH it seems unusual that you would have to use extremely large buffers where initialization time matters. E.g. if you were to use this for reading from a file or stream then it might make more sense to use many smaller buffers rather than a single huge one, no?
Hi Bert. I should have explained better. The library I am wrapping is LZ4, a fast compressor: http://code.google.com/p/lz4/ The function to compress looks like this:
int LZ4_compress (const char* source, char* dest, int isize);
/* LZ4_compress() : Compresses 'isize' bytes from 'source' into 'dest'. Destination buffer must be already allocated, and must be sized to handle worst cases situations (input data not compressible) Worst case size evaluation is provided by macro LZ4_compressBound()
isize : is the input size. Max supported value is ~1.9GB return : the number of bytes written in buffer destSo, from the image side, I am doing (summarized):
dest := ByteArray new: (self funcCompressBound: aByteArray size) + 4. bytesCompressedOrError := self funcCompress: aByteArray byteArrayDestination: dest isize: aByteArray size maxOutputSize: maxOutputSize . compressed := dest copyFrom: 1 to: bytesCompressedOrError + 4.
But a normal scenario of compression is when the bytearray is indeed quite large. The function requires that "dest" is already allocated. And then I need to do the #copyFrom:to:, and that is what I was trying to avoid.
maybe there is something faster than #copyFrom:to: for this case?
- Bert -
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On 08.11.2012, at 22:42, Mariano Martinez Peck marianopeck@gmail.com wrote:
On Thu, Nov 8, 2012 at 4:48 PM, Mariano Martinez Peck marianopeck@gmail.com wrote:
On Thu, Nov 8, 2012 at 4:38 PM, Bert Freudenberg bert@freudenbergs.de wrote:
On 2012-11-08, at 16:22, Mariano Martinez Peck marianopeck@gmail.com wrote:
Hi guys. I have the following scenario. I have a buffer (ByteArray) that I pass by FFI to a function of size N. This function puts data in the array and answers me the M number of bytes that it put. M <= N. Finally, I need to copy the array of size N to the accuare size M. To do that, I am using #copyFrom:to:. If the ByteArray is large (which could be the case), this function takes significant time because it needs to allocate space for the new large resulting array. So...is there a destructive primitive where I can "crop" the existing array, modify its size field and mark the remaining bytes as "free space for the heap".
Do we have a primitive for that?
We do not have any primitive that changes an object's size.
:(
However, if your problem is indeed the time for allocating the new array, then maybe there should be a primitive for that? E.g. one that copies a portion of one array to a new object. This would avoid having to initialize the memory of the new array - this is what's taking time, otherwise allocation is normally constant in time.
OTOH it seems unusual that you would have to use extremely large buffers where initialization time matters. E.g. if you were to use this for reading from a file or stream then it might make more sense to use many smaller buffers rather than a single huge one, no?
Hi Bert. I should have explained better. The library I am wrapping is LZ4, a fast compressor: http://code.google.com/p/lz4/ The function to compress looks like this:
int LZ4_compress (const char* source, char* dest, int isize);
/* LZ4_compress() : Compresses 'isize' bytes from 'source' into 'dest'. Destination buffer must be already allocated, and must be sized to handle worst cases situations (input data not compressible) Worst case size evaluation is provided by macro LZ4_compressBound()
isize : is the input size. Max supported value is ~1.9GB return : the number of bytes written in buffer dest
So, from the image side, I am doing (summarized):
dest := ByteArray new: (self funcCompressBound: aByteArray size) + 4. bytesCompressedOrError := self funcCompress: aByteArray byteArrayDestination: dest isize: aByteArray size maxOutputSize: maxOutputSize . compressed := dest copyFrom: 1 to: bytesCompressedOrError + 4.
But a normal scenario of compression is when the bytearray is indeed quite large. The function requires that "dest" is already allocated. And then I need to do the #copyFrom:to:, and that is what I was trying to avoid.
maybe there is something faster than #copyFrom:to: for this case?
Not yet. There was a discussion some time ago to add an "uninitialized allocation" primitive (i.e. a ByteArray/WordArray not initialized to 0) but I think we didn't follow up on that.
Are you actually keeping that huge array in memory? If not, e.g you are sending it to disk or network, then just keeping a count around (like OrderedCollection), or putting it into a read stream) would suffice.
- Bert -
On 8 November 2012 19:51, Bert Freudenberg bert@freudenbergs.de wrote:
On 08.11.2012, at 22:42, Mariano Martinez Peck marianopeck@gmail.com wrote:
On Thu, Nov 8, 2012 at 4:48 PM, Mariano Martinez Peck marianopeck@gmail.com wrote:
On Thu, Nov 8, 2012 at 4:38 PM, Bert Freudenberg bert@freudenbergs.de wrote:
On 2012-11-08, at 16:22, Mariano Martinez Peck marianopeck@gmail.com wrote:
Hi guys. I have the following scenario. I have a buffer (ByteArray) that I pass by FFI to a function of size N. This function puts data in the array and answers me the M number of bytes that it put. M <= N. Finally, I need to copy the array of size N to the accuare size M. To do that, I am using #copyFrom:to:. If the ByteArray is large (which could be the case), this function takes significant time because it needs to allocate space for the new large resulting array. So...is there a destructive primitive where I can "crop" the existing array, modify its size field and mark the remaining bytes as "free space for the heap".
Do we have a primitive for that?
We do not have any primitive that changes an object's size.
:(
However, if your problem is indeed the time for allocating the new array, then maybe there should be a primitive for that? E.g. one that copies a portion of one array to a new object. This would avoid having to initialize the memory of the new array - this is what's taking time, otherwise allocation is normally constant in time.
OTOH it seems unusual that you would have to use extremely large buffers where initialization time matters. E.g. if you were to use this for reading from a file or stream then it might make more sense to use many smaller buffers rather than a single huge one, no?
Hi Bert. I should have explained better. The library I am wrapping is LZ4, a fast compressor: http://code.google.com/p/lz4/ The function to compress looks like this:
int LZ4_compress (const char* source, char* dest, int isize);
/* LZ4_compress() : Compresses 'isize' bytes from 'source' into 'dest'. Destination buffer must be already allocated, and must be sized to handle worst cases situations (input data not compressible) Worst case size evaluation is provided by macro LZ4_compressBound()
isize : is the input size. Max supported value is ~1.9GB return : the number of bytes written in buffer dest
So, from the image side, I am doing (summarized):
dest := ByteArray new: (self funcCompressBound: aByteArray size) + 4. bytesCompressedOrError := self funcCompress: aByteArray byteArrayDestination: dest isize: aByteArray size maxOutputSize: maxOutputSize . compressed := dest copyFrom: 1 to: bytesCompressedOrError + 4.
But a normal scenario of compression is when the bytearray is indeed quite large. The function requires that "dest" is already allocated. And then I need to do the #copyFrom:to:, and that is what I was trying to avoid.
maybe there is something faster than #copyFrom:to: for this case?
Not yet. There was a discussion some time ago to add an "uninitialized allocation" primitive (i.e. a ByteArray/WordArray not initialized to 0) but I think we didn't follow up on that.
Are you actually keeping that huge array in memory? If not, e.g you are sending it to disk or network, then just keeping a count around (like OrderedCollection), or putting it into a read stream) would suffice.
right, i also said that to Mariano: usually you split data on smaller chunks and compress them. And the compressed output you pipe to output stream. So, yes, you can't avoid copying things around (because of piping to stream).. but you cannot squeeze out of it much at this point. And since you using stream, you don't really need to crop anything.
- Bert -
Igor Stasenko wrote
right, i also said that to Mariano: usually you split data on smaller chunks and compress them. And the compressed output you pipe to output stream. So, yes, you can't avoid copying things around (because of piping to stream).. but you cannot squeeze out of it much at this point. And since you using stream, you don't really need to crop anything.
Plus, unless the estimate is wildly wrong you can just wrap your result into a stream; say:
compressed := ReadWriteStream on: dest from: 1 to: bytesCompressedOrError + 4.
Cheers, - Andreas
-- View this message in context: http://forum.world.st/Primitive-to-crop-a-ByteArray-tp4654477p4654591.html Sent from the Squeak VM mailing list archive at Nabble.com.
On Thu, 8 Nov 2012, Mariano Martinez Peck wrote:
On Thu, Nov 8, 2012 at 4:48 PM, Mariano Martinez Peck marianopeck@gmail.com wrote:
On Thu, Nov 8, 2012 at 4:38 PM, Bert Freudenberg bert@freudenbergs.de wrote:
On 2012-11-08, at 16:22, Mariano Martinez Peck marianopeck@gmail.com wrote:
Hi guys. I have the following scenario. I have a buffer (ByteArray) that I pass by FFI to a function of size N. This function puts data in the array and answers me the M number of bytes that it put. M <= N. Finally, I need to copy the array of size N to the accuare size M. To do that, I am using #copyFrom:to:. If the ByteArray is large (which could be the case), this function takes significant time because it needs to allocate space for the new large resulting array. So...is there a destructive primitive where I can "crop" the existing array, modify its size field and mark the remaining bytes as "free space for the heap".
Do we have a primitive for that?
We do not have any primitive that changes an object's size.
:(
However, if your problem is indeed the time for allocating the new array, then maybe there should be a primitive for that? E.g. one that copies a portion of one array to a new object. This would avoid having to initialize the memory of the new array - this is what's taking time, otherwise allocation is normally constant in time.
OTOH it seems unusual that you would have to use extremely large buffers where initialization time matters. E.g. if you were to use this for reading from a file or stream then it might make more sense to use many smaller buffers rather than a single huge one, no?
Hi Bert. I should have explained better. The library I am wrapping is LZ4, a fast compressor: http://code.google.com/p/lz4/ The function to compress looks like this:
int LZ4_compress (const char* source, char* dest, int isize);
/* LZ4_compress() : Compresses 'isize' bytes from 'source' into 'dest'. Destination buffer must be already allocated, and must be sized to handle worst cases situations (input data not compressible) Worst case size evaluation is provided by macro LZ4_compressBound()
isize : is the input size. Max supported value is ~1.9GB return : the number of bytes written in buffer destSo, from the image side, I am doing (summarized):
dest := ByteArray new: (self funcCompressBound: aByteArray size) + 4. bytesCompressedOrError := self funcCompress: aByteArray byteArrayDestination: dest isize: aByteArray size maxOutputSize: maxOutputSize . compressed := dest copyFrom: 1 to: bytesCompressedOrError + 4.
But a normal scenario of compression is when the bytearray is indeed quite large. The function requires that "dest" is already allocated. And then I need to do the #copyFrom:to:, and that is what I was trying to avoid.
maybe there is something faster than #copyFrom:to: for this case?
Do you think #copyFrom:to: is slow because it shows up in the profiler? If yes, then try running the same code without calling the compression function to see how "slow" #copyFrom:to: really is. Also, calling #funcCompressBound: is just silly. Reimplement it in Smalltalk, it's easy and probably more efficient. If it turns out that it's really #copyFrom:to: what's slow, then consider compressing your data in smaller chunks (at most a few MiB each). In this case you can even precalculate the upper bound.
Levente
P.S.: And please don't propose VM changes for such a marginal issue. P.P.S.: I still doubt that using LZ4 will give you any benefit.
- Bert -
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
vm-dev@lists.squeakfoundation.org
-
Andreas.Raab -
Bert Freudenberg -
Igor Stasenko -
Levente Uzonyi -
Mariano Martinez Peck