Hey Levente,
I am trying to primitize the #findErrorLocations: & #findErrorMagnitudes:errorLocations: methods. I thought I had it done, but I am getting a segfault when running the RSFECReedSolomonTest>>#testDataMatrix test. I tried to dig into it and find out the issue. I think one of the params to the primitive is nil. The strange thing I observed is a debug stack in Squeak, where the argument to the prior stack frame is an instance, but the argument in the top frame is nil. I do not understand how it can go from type -> nil.
If you wish to help me figure this out, the current RSFEC code is loaded with:
Installer ss project: 'Cryptography'; install: 'ProCrypto-1-1-1'; install: 'ProCryptoTests-1-1-1'.
Then the Decoder primitives to RSFECPlugin is loaded from:
CryptographyRSFECPluginsDecoder-segfault-rww.2
--- Kindly, Robert
On 6/2/21 10:40 AM, Levente Uzonyi wrote:
Hi Robert,
On Tue, 1 Jun 2021, Robert Withers wrote:
Hey ya, Levente,
Woot!!! I was able to get all the primitives working with ByteArrays. Everything is passing green! (except for several GF fields: Aztec and Maxicode and QRCode) I think I will primitize one of the codingLoops on the RSErasure side. Here are the numbers. Take note of the tallies and time: 50 seconds versus 114 seconds. I updated the squeak wiki Cryptography page ( http://wiki.squeak.org/squeak/5776 ) with d/l links to the plugins. Here are those
Brilliant. ByteArray is indeed the optimal choice there.
Let me answer your question in your previous email here: AFAIK, on all the currently supported platforms and C compilers, unsigned int is 32-bits, so it matches WordArray in Squeak. DoubleWordArray is 64-bits, and in C that matches the type unsigned long long (on the currently supported platforms and compilers).
I think you can shave off a few more seconds there by adding those <inline: #always> pragmas to the methods the primitive methods send either directly or indirectly.
And, I think the logTable and expTable variables can be declared as constant as well as unsigned char arrays instead of unsigned int arrays. That might give more opportunities to the C compiler to optimize the code.
I prefer to use #var:type:array: instead of #var:declareC: when possible. E.g.
cg var: #logTable declareC: 'unsigned int logTable[256] = { 0, 0, 1, 240, 2, 225, 241, 53, 3, 38, 226, 133, 242, 43, 54, 210, 4, 195, 39, 114, 227, 106, 134, 28, 243, 140, 44, 23, 55, 118, 211, 234, 5, 219, 196, 96, 40, 222, 115, 103, 228, 78, 107, 125, 135, 8, 29, 162, 244, 186, 141, 180, 45, 99, 24, 49, 56, 13, 119, 153, 212, 199, 235, 91, 6, 76, 220, 217, 197, 11, 97, 184, 41, 36, 223, 253, 116, 138, 104, 193, 229, 86, 79, 171, 108, 165, 126, 145, 136, 34, 9, 74, 30, 32, 163, 84, 245, 173, 187, 204, 142, 81, 181, 190, 46, 88, 100, 159, 25, 231, 50, 207, 57, 147, 14, 67, 120, 128, 154, 248, 213, 167, 200, 63, 236, 110, 92, 176, 7, 161, 77, 124, 221, 102, 218, 95, 198, 90, 12, 152, 98, 48, 185, 179, 42, 209, 37, 132, 224, 52, 254, 239, 117, 233, 139, 22, 105, 27, 194, 113, 230, 206, 87, 158, 80, 189, 172, 203, 109, 175, 166, 62, 127, 247, 146, 66, 137, 192, 35, 252, 10, 183, 75, 216, 31, 83, 33, 73, 164, 144, 85, 170, 246, 65, 174, 61, 188, 202, 205, 157, 143, 169, 82, 72, 182, 215, 191, 251, 47, 178, 89, 151, 101, 94, 160, 123, 26, 112, 232, 21, 51, 238, 208, 131, 58, 69, 148, 18, 15, 16, 68, 17, 121, 149, 129, 19, 155, 59, 249, 70, 214, 250, 168, 71, 201, 156, 64, 60, 237, 130, 111, 20, 93, 122, 177, 150 }';
could be written as
cg var: #logTable type: #'const unsigned char' array: #[0 0 1 240 2 225 241 53 3 38 226 133 242 43 54 210 4 195 39 114 227 106 134 28 243 140 44 23 55 118 211 234 5 219 196 96 40 222 115 103 228 78 107 125 135 8 29 162 244 186 141 180 45 99 24 49 56 13 119 153 212 199 235 91 6 76 220 217 197 11 97 184 41 36 223 253 116 138 104 193 229 86 79 171 108 165 126 145 136 34 9 74 30 32 163 84 245 173 187 204 142 81 181 190 46 88 100 159 25 231 50 207 57 147 14 67 120 128 154 248 213 167 200 63 236 110 92 176 7 161 77 124 221 102 218 95 198 90 12 152 98 48 185 179 42 209 37 132 224 52 254 239 117 233 139 22 105 27 194 113 230 206 87 158 80 189 172 203 109 175 166 62 127 247 146 66 137 192 35 252 10 183 75 216 31 83 33 73 164 144 85 170 246 65 174 61 188 202 205 157 143 169 82 72 182 215 191 251 47 178 89 151 101 94 160 123 26 112 232 21 51 238 208 131 58 69 148 18 15 16 68 17 121 149 129 19 155 59 249 70 214 250 168 71 201 156 64 60 237 130 111 20 93 122 177 150]
Levente
Levente,
Some more info. I saved another version of the segfault package. I was able to run the test resulting in a Squeak debugger. I am listing two snapshots, one is on the #decode:twoS: stack frame and the other is the next frame #findErrorLocations: As you can see the sigma method variable is a GFPolynomial, while the errorLocator in the subsequent frame is nil, even though the sigma is passed as that argument. How did *that* happen? Here is the DoIt to load everything needed:
Installer ss project: 'Cryptography'; install: 'ProCrypto-1-1-1'; install: 'ProCryptoTests-1-1-1'; install: 'CryptographyRSFECPlugins'; install: 'TraceMonitor'; install: 'CryptographyRSFECPluginsDecoder-segfault'.
I am also attaching the etrace.log with some debug info I logged. I am confused. When I have gotten a crash.dmp, one of the lines in the c stack is a call to isBytes.
/usr/local/bin/../lib/squeak/5.0-202010010729/squeak(isBytes+0x9)[0x44d289]
Grazie! Robert
--- Kindly, Robert
On 6/2/21 2:49 PM, Robert Withers wrote:
Hey Levente,
I am trying to primitize the #findErrorLocations: & #findErrorMagnitudes:errorLocations: methods. I thought I had it done, but I am getting a segfault when running the RSFECReedSolomonTest>>#testDataMatrix test. I tried to dig into it and find out the issue. I think one of the params to the primitive is nil. The strange thing I observed is a debug stack in Squeak, where the argument to the prior stack frame is an instance, but the argument in the top frame is nil. I do not understand how it can go from type -> nil.
If you wish to help me figure this out, the current RSFEC code is loaded with:
Installer ss project: 'Cryptography'; install: 'ProCrypto-1-1-1'; install: 'ProCryptoTests-1-1-1'.
Then the Decoder primitives to RSFECPlugin is loaded from:
CryptographyRSFECPluginsDecoder-segfault-rww.2
Kindly, Robert
On 6/2/21 10:40 AM, Levente Uzonyi wrote:
Hi Robert,
On Tue, 1 Jun 2021, Robert Withers wrote:
Hey ya, Levente,
Woot!!! I was able to get all the primitives working with ByteArrays. Everything is passing green! (except for several GF fields: Aztec and Maxicode and QRCode) I think I will primitize one of the codingLoops on the RSErasure side. Here are the numbers. Take note of the tallies and time: 50 seconds versus 114 seconds. I updated the squeak wiki Cryptography page ( http://wiki.squeak.org/squeak/5776 ) with d/l links to the plugins. Here are those
Brilliant. ByteArray is indeed the optimal choice there.
Let me answer your question in your previous email here: AFAIK, on all the currently supported platforms and C compilers, unsigned int is 32-bits, so it matches WordArray in Squeak. DoubleWordArray is 64-bits, and in C that matches the type unsigned long long (on the currently supported platforms and compilers).
I think you can shave off a few more seconds there by adding those <inline: #always> pragmas to the methods the primitive methods send either directly or indirectly.
And, I think the logTable and expTable variables can be declared as constant as well as unsigned char arrays instead of unsigned int arrays. That might give more opportunities to the C compiler to optimize the code.
I prefer to use #var:type:array: instead of #var:declareC: when possible. E.g.
cg var: #logTable declareC: 'unsigned int logTable[256] = { 0, 0, 1, 240, 2, 225, 241, 53, 3, 38, 226, 133, 242, 43, 54, 210, 4, 195, 39, 114, 227, 106, 134, 28, 243, 140, 44, 23, 55, 118, 211, 234, 5, 219, 196, 96, 40, 222, 115, 103, 228, 78, 107, 125, 135, 8, 29, 162, 244, 186, 141, 180, 45, 99, 24, 49, 56, 13, 119, 153, 212, 199, 235, 91, 6, 76, 220, 217, 197, 11, 97, 184, 41, 36, 223, 253, 116, 138, 104, 193, 229, 86, 79, 171, 108, 165, 126, 145, 136, 34, 9, 74, 30, 32, 163, 84, 245, 173, 187, 204, 142, 81, 181, 190, 46, 88, 100, 159, 25, 231, 50, 207, 57, 147, 14, 67, 120, 128, 154, 248, 213, 167, 200, 63, 236, 110, 92, 176, 7, 161, 77, 124, 221, 102, 218, 95, 198, 90, 12, 152, 98, 48, 185, 179, 42, 209, 37, 132, 224, 52, 254, 239, 117, 233, 139, 22, 105, 27, 194, 113, 230, 206, 87, 158, 80, 189, 172, 203, 109, 175, 166, 62, 127, 247, 146, 66, 137, 192, 35, 252, 10, 183, 75, 216, 31, 83, 33, 73, 164, 144, 85, 170, 246, 65, 174, 61, 188, 202, 205, 157, 143, 169, 82, 72, 182, 215, 191, 251, 47, 178, 89, 151, 101, 94, 160, 123, 26, 112, 232, 21, 51, 238, 208, 131, 58, 69, 148, 18, 15, 16, 68, 17, 121, 149, 129, 19, 155, 59, 249, 70, 214, 250, 168, 71, 201, 156, 64, 60, 237, 130, 111, 20, 93, 122, 177, 150 }';
could be written as
cg var: #logTable type: #'const unsigned char' array: #[0 0 1 240 2 225 241 53 3 38 226 133 242 43 54 210 4 195 39 114 227 106 134 28 243 140 44 23 55 118 211 234 5 219 196 96 40 222 115 103 228 78 107 125 135 8 29 162 244 186 141 180 45 99 24 49 56 13 119 153 212 199 235 91 6 76 220 217 197 11 97 184 41 36 223 253 116 138 104 193 229 86 79 171 108 165 126 145 136 34 9 74 30 32 163 84 245 173 187 204 142 81 181 190 46 88 100 159 25 231 50 207 57 147 14 67 120 128 154 248 213 167 200 63 236 110 92 176 7 161 77 124 221 102 218 95 198 90 12 152 98 48 185 179 42 209 37 132 224 52 254 239 117 233 139 22 105 27 194 113 230 206 87 158 80 189 172 203 109 175 166 62 127 247 146 66 137 192 35 252 10 183 75 216 31 83 33 73 164 144 85 170 246 65 174 61 188 202 205 157 143 169 82 72 182 215 191 251 47 178 89 151 101 94 160 123 26 112 232 21 51 238 208 131 58 69 148 18 15 16 68 17 121 149 129 19 155 59 249 70 214 250 168 71 201 156 64 60 237 130 111 20 93 122 177 150]
Levente
Oops, email size too big. I zipped everything up. The SqueakDebug.log has some good info in it. Here is the evidence:
https://www.dropbox.com/s/fcinfptyvv4xb42/Decoder-primitiveFailed.zip?dl=0
Levente,
Some more info. I saved another version of the segfault package. I was able to run the test resulting in a Squeak debugger. I am listing two snapshots, one is on the #decode:twoS: stack frame and the other is the next frame #findErrorLocations: As you can see the sigma method variable is a GFPolynomial, while the errorLocator in the subsequent frame is nil, even though the sigma is passed as that argument. How did *that* happen? Here is the DoIt to load everything needed:
Installer ss project: 'Cryptography'; install: 'ProCrypto-1-1-1'; install: 'ProCryptoTests-1-1-1'; install: 'CryptographyRSFECPlugins'; install: 'TraceMonitor'; install: 'CryptographyRSFECPluginsDecoder-segfault'.
I am also attaching the etrace.log with some debug info I logged. I am confused. When I have gotten a crash.dmp, one of the lines in the c stack is a call to isBytes.
/usr/local/bin/../lib/squeak/5.0-202010010729/squeak(isBytes+0x9)[0x44d289]
Grazie! Robert --- Kindly, Robert
On 6/2/21 2:49 PM, Robert Withers wrote:
Hey Levente,
I am trying to primitize the #findErrorLocations: & #findErrorMagnitudes:errorLocations: methods. I thought I had it done, but I am getting a segfault when running the RSFECReedSolomonTest>>#testDataMatrix test. I tried to dig into it and find out the issue. I think one of the params to the primitive is nil. The strange thing I observed is a debug stack in Squeak, where the argument to the prior stack frame is an instance, but the argument in the top frame is nil. I do not understand how it can go from type -> nil.
If you wish to help me figure this out, the current RSFEC code is loaded with:
Installer ss project: 'Cryptography'; install: 'ProCrypto-1-1-1'; install: 'ProCryptoTests-1-1-1'.
Then the Decoder primitives to RSFECPlugin is loaded from:
CryptographyRSFECPluginsDecoder-segfault-rww.2
Kindly, Robert
On 6/2/21 10:40 AM, Levente Uzonyi wrote:
Hi Robert,
On Tue, 1 Jun 2021, Robert Withers wrote:
Hey ya, Levente,
Woot!!! I was able to get all the primitives working with ByteArrays. Everything is passing green! (except for several GF fields: Aztec and Maxicode and QRCode) I think I will primitize one of the codingLoops on the RSErasure side. Here are the numbers. Take note of the tallies and time: 50 seconds versus 114 seconds. I updated the squeak wiki Cryptography page ( http://wiki.squeak.org/squeak/5776 ) with d/l links to the plugins. Here are those
Brilliant. ByteArray is indeed the optimal choice there.
Let me answer your question in your previous email here: AFAIK, on all the currently supported platforms and C compilers, unsigned int is 32-bits, so it matches WordArray in Squeak. DoubleWordArray is 64-bits, and in C that matches the type unsigned long long (on the currently supported platforms and compilers).
I think you can shave off a few more seconds there by adding those <inline: #always> pragmas to the methods the primitive methods send either directly or indirectly.
And, I think the logTable and expTable variables can be declared as constant as well as unsigned char arrays instead of unsigned int arrays. That might give more opportunities to the C compiler to optimize the code.
I prefer to use #var:type:array: instead of #var:declareC: when possible. E.g.
cg var: #logTable declareC: 'unsigned int logTable[256] = { 0, 0, 1, 240, 2, 225, 241, 53, 3, 38, 226, 133, 242, 43, 54, 210, 4, 195, 39, 114, 227, 106, 134, 28, 243, 140, 44, 23, 55, 118, 211, 234, 5, 219, 196, 96, 40, 222, 115, 103, 228, 78, 107, 125, 135, 8, 29, 162, 244, 186, 141, 180, 45, 99, 24, 49, 56, 13, 119, 153, 212, 199, 235, 91, 6, 76, 220, 217, 197, 11, 97, 184, 41, 36, 223, 253, 116, 138, 104, 193, 229, 86, 79, 171, 108, 165, 126, 145, 136, 34, 9, 74, 30, 32, 163, 84, 245, 173, 187, 204, 142, 81, 181, 190, 46, 88, 100, 159, 25, 231, 50, 207, 57, 147, 14, 67, 120, 128, 154, 248, 213, 167, 200, 63, 236, 110, 92, 176, 7, 161, 77, 124, 221, 102, 218, 95, 198, 90, 12, 152, 98, 48, 185, 179, 42, 209, 37, 132, 224, 52, 254, 239, 117, 233, 139, 22, 105, 27, 194, 113, 230, 206, 87, 158, 80, 189, 172, 203, 109, 175, 166, 62, 127, 247, 146, 66, 137, 192, 35, 252, 10, 183, 75, 216, 31, 83, 33, 73, 164, 144, 85, 170, 246, 65, 174, 61, 188, 202, 205, 157, 143, 169, 82, 72, 182, 215, 191, 251, 47, 178, 89, 151, 101, 94, 160, 123, 26, 112, 232, 21, 51, 238, 208, 131, 58, 69, 148, 18, 15, 16, 68, 17, 121, 149, 129, 19, 155, 59, 249, 70, 214, 250, 168, 71, 201, 156, 64, 60, 237, 130, 111, 20, 93, 122, 177, 150 }';
could be written as
cg var: #logTable type: #'const unsigned char' array: #[0 0 1 240 2 225 241 53 3 38 226 133 242 43 54 210 4 195 39 114 227 106 134 28 243 140 44 23 55 118 211 234 5 219 196 96 40 222 115 103 228 78 107 125 135 8 29 162 244 186 141 180 45 99 24 49 56 13 119 153 212 199 235 91 6 76 220 217 197 11 97 184 41 36 223 253 116 138 104 193 229 86 79 171 108 165 126 145 136 34 9 74 30 32 163 84 245 173 187 204 142 81 181 190 46 88 100 159 25 231 50 207 57 147 14 67 120 128 154 248 213 167 200 63 236 110 92 176 7 161 77 124 221 102 218 95 198 90 12 152 98 48 185 179 42 209 37 132 224 52 254 239 117 233 139 22 105 27 194 113 230 206 87 158 80 189 172 203 109 175 166 62 127 247 146 66 137 192 35 252 10 183 75 216 31 83 33 73 164 144 85 170 246 65 174 61 188 202 205 157 143 169 82 72 182 215 191 251 47 178 89 151 101 94 160 123 26 112 232 21 51 238 208 131 58 69 148 18 15 16 68 17 121 149 129 19 155 59 249 70 214 250 168 71 201 156 64 60 237 130 111 20 93 122 177 150]
Levente
cryptography@lists.squeakfoundation.org
-
Robert Withers