We have postgresql database using UTF-8 charset and I had problems doing queries against the database using umlaut characters. It looked like there was no character conversion done to the statements going to opendbx api. Or is it that I just did not find it?
However I implemented a little fix that encodes strings going and coming from OpenDBX and mechanism that queries the encoding database uses and attaches this information to the connection. Unfortunately the charset query is currently only implemented for postgresql (show client_encoding). Where should I post these changes? Directly to squeaksource?
Also is there a connection pooling facility for SqueakDBX or Glorp?
On Tue, Aug 24, 2010 at 9:55 AM, Panu Suominen panu.j.m.suominen@gmail.comwrote:
We have postgresql database using UTF-8 charset and I had problems doing queries against the database using umlaut characters. It looked like there was no character conversion done to the statements going to opendbx api. Or is it that I just did not find it?
Hi and welcome!
I think there is no conversion in OpenDBX nor in SqueakDBX. Anyway, I attach Norbert (OpenDBX author). Maybe he can help you with this.
However I implemented a little fix that encodes strings going and coming from OpenDBX and mechanism that queries the encoding database uses and attaches this information to the connection. Unfortunately the charset query is currently only implemented for postgresql (show client_encoding). Where should I post these changes? Directly to squeaksource?
Yes. So that I can see it. Do you know if it breaks compatibility with other databases or OS? because SqueakDBX classes like DBXConnection should be cross-backend.
If you want to can use another new package with your commit.
Also is there a connection pooling facility for SqueakDBX or Glorp?
Yes, several.
-Check GlorpDBX-ConnectionPool-JohnToohey.4 in SqueakDBX repo. I am not sure if he could have time to document it...but if he did I can include it in the website.
-GlorpSeaside is another package but I don't remember were is it.
-Ramon Leon did: http://onsmalltalk.com/making-a-connection-pool-for-glorp-in-seaside,
- Esteban Lorenzano said My Pool implementation can be download from here:
http://www.squeaksource.com/Officious (package name: Pool)
it is a very small, very simple (and probably buggy) implementation. Here are some example of it's use:
pool := (Pool creator: [ self createSession ]) maxObjects: 10; maxRetries: 3; waitTimeInMilliseconds: 50; yourself.
session := pool borrow. ... your glorp code... pool release: session.
Cheers
Mariano
-- Panu _______________________________________________ SqueakDBX mailing list SqueakDBX@lists.squeakfoundation.org http://lists.squeakfoundation.org/mailman/listinfo/squeakdbx
2010/8/24 Mariano Martinez Peck marianopeck@gmail.com:
Yes. So that I can see it. Do you know if it breaks compatibility with other databases or OS? because SqueakDBX classes like DBXConnection should be cross-backend.
I don't know how cross-platform my fix is. Database part should be fine. Postgresql can now detect the encoding. Others database platforms just don't know anything about encoding (they use default utf-8). If other smalltalk dialects have TextConverter then it should be fine. If they haven't then I can implement another abstraction layer to provide compatibility between dialects.
If you want to can use another new package with your commit.
I am sorry, but I did not quite understand what you meant...
Unfortunately I am a little busy at work right now so I did not had time to learn how to use monticello properly to create version only containing required changes. For this reason I attached change set file to this mail.
Also is there a connection pooling facility for SqueakDBX or Glorp?
Yes, several.
Thank you.
On Wed, Aug 25, 2010 at 7:20 AM, Panu Suominen panu.j.m.suominen@gmail.comwrote:
2010/8/24 Mariano Martinez Peck marianopeck@gmail.com:
Yes. So that I can see it. Do you know if it breaks compatibility with
other
databases or OS? because SqueakDBX classes like DBXConnection should be cross-backend.
I don't know how cross-platform my fix is. Database part should be fine. Postgresql can now detect the encoding.
Thanks a lot for the changeset. Now I will test with different OS and databases and I will let you know.
now a couple of questions:
- I guess a lot of people doesn't need UTF (or other) encoding. And I guess that the transformation brings some overhead in the query result mosty...(I should run the benchmarks and see if there is difference). But I was thinking if we could make this optional. I like the code and design you did. I think that a good idea would be to add some kind of NullEncoder which is set in:
DBXConnection >> initialize
instead of TextConverter newForEncoding: #utf8.
Then, for postgres, you will be using the one from TextConverter newForEncoding: (r rows first valueAt: 1).
Even more, I would like to make it optional. Maybe we can create DBXCOnnection class >> platform: aPlatform settings: aConnectionSettings encode: anEncoder and that will do:
platform: aPlatform settings: aConnectionSettings encoder: anEncoder ^self new platform: aPlatform; settings: aConnectionSettings; encoder: anEncoder; yourself
Finally, the cleint code (your app code) creates the connection with that message :)
or something like that....but the idea of making it optional not to have overhead if not required, and being able to be enabled from the app code.
- What happens is the database supports and specific encoding that Pharo doesn't support? TextConverter newForEncoding: (r rows first valueAt: 1).
will return nil, causing UndefinedObject dnu something.
maybe we can just set again the NullEncoder in that case? or raise an error ? what do you think?
Others database platforms just don't know anything about encoding
(they use default utf-8). If other smalltalk dialects have TextConverter then it should be fine. If they haven't then I can implement another abstraction layer to provide compatibility between dialects.
If you want to can use another new package with your commit.
I am sorry, but I did not quite understand what you meant...
Unfortunately I am a little busy at work right now so I did not had time to learn how to use monticello properly to create version only containing required changes. For this reason I attached change set file to this mail.
Don't worry :)
thanks!
Also is there a connection pooling facility for SqueakDBX or Glorp?
Yes, several.
Thank you.
-- Panu
On Thu, Aug 26, 2010 at 9:43 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Wed, Aug 25, 2010 at 7:20 AM, Panu Suominen < panu.j.m.suominen@gmail.com> wrote:
2010/8/24 Mariano Martinez Peck marianopeck@gmail.com:
Yes. So that I can see it. Do you know if it breaks compatibility with
other
databases or OS? because SqueakDBX classes like DBXConnection should be cross-backend.
I don't know how cross-platform my fix is. Database part should be fine. Postgresql can now detect the encoding.
Thanks a lot for the changeset. Now I will test with different OS and databases and I will let you know.
now a couple of questions:
- I guess a lot of people doesn't need UTF (or other) encoding. And I guess
that the transformation brings some overhead in the query result mosty...(I should run the benchmarks and see if there is difference). But I was thinking if we could make this optional. I like the code and design you did. I think that a good idea would be to add some kind of NullEncoder which is set in:
DBXConnection >> initialize
instead of TextConverter newForEncoding: #utf8.
Then, for postgres, you will be using the one from TextConverter newForEncoding: (r rows first valueAt: 1).
Even more, I would like to make it optional. Maybe we can create DBXCOnnection class >> platform: aPlatform settings: aConnectionSettings encode: anEncoder and that will do:
platform: aPlatform settings: aConnectionSettings encoder: anEncoder ^self new platform: aPlatform; settings: aConnectionSettings; encoder: anEncoder; yourself
Finally, the cleint code (your app code) creates the connection with that message :)
or something like that....but the idea of making it optional not to have overhead if not required, and being able to be enabled from the app code.
- What happens is the database supports and specific encoding that Pharo
doesn't support? TextConverter newForEncoding: (r rows first valueAt: 1).
will return nil, causing UndefinedObject dnu something.
maybe we can just set again the NullEncoder in that case? or raise an error ? what do you think?
maybe encoder should be a class variable, not intance variable. And in addition, you can set it from outside ;)
Others database platforms just don't know anything about encoding
(they use default utf-8). If other smalltalk dialects have TextConverter then it should be fine. If they haven't then I can implement another abstraction layer to provide compatibility between dialects.
If you want to can use another new package with your commit.
I am sorry, but I did not quite understand what you meant...
Unfortunately I am a little busy at work right now so I did not had time to learn how to use monticello properly to create version only containing required changes. For this reason I attached change set file to this mail.
Don't worry :)
thanks!
Also is there a connection pooling facility for SqueakDBX or Glorp?
Yes, several.
Thank you.
-- Panu
On Thu, Aug 26, 2010 at 9:43 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Wed, Aug 25, 2010 at 7:20 AM, Panu Suominen < panu.j.m.suominen@gmail.com> wrote:
2010/8/24 Mariano Martinez Peck marianopeck@gmail.com:
Yes. So that I can see it. Do you know if it breaks compatibility with
other
databases or OS? because SqueakDBX classes like DBXConnection should be cross-backend.
I don't know how cross-platform my fix is. Database part should be fine. Postgresql can now detect the encoding.
Thanks a lot for the changeset. Now I will test with different OS and databases and I will let you know.
now a couple of questions:
- I guess a lot of people doesn't need UTF (or other) encoding. And I guess
that the transformation brings some overhead in the query result mosty...(I should run the benchmarks and see if there is difference). But I was thinking if we could make this optional. I like the code and design you did. I think that a good idea would be to add some kind of NullEncoder which is set in:
Ok.... I run the benchmarks and I was rigth ;)
Before:
Benchmarking: DBXTinyBenchmarks (10 runs) benchmarkInsert: 521.4 AVG benchmarkInsertConverted: 795.9 AVG benchmarkSelect10: 2.5 AVG benchmarkSelect100: 17.5 AVG benchmarkSelect1000: 167.4 AVG benchmarkSelect10000: 1691.9 AVG
After:
Benchmarking: DBXTinyBenchmarks (10 runs) benchmarkInsert: 643.2 AVG benchmarkInsertConverted: 905.6 AVG benchmarkSelect10: 3.8 AVG benchmarkSelect100: 25.1 AVG benchmarkSelect1000: 246.4 AVG benchmarkSelect10000: 2577.6 AVG
In addition, several SqueakDBX tests were failing with your changes...In order to have all green tests, I had to do:
fieldRawValue: anIndex on: aResultSet "Returns the value of the column at anIndex of the current row from aResultSet" | raw | raw := (OpenDBX current apiQueryFieldValue: (aResultSet handle) index: (anIndex - 1)). raw isNil ifFalse: [^aResultSet connection encoder convertFromSystemString: raw]. ^ nil
Panu, I really appreciate your help. I am happy to integrate this change when we can make it option :)
DBXConnection >> initialize
instead of TextConverter newForEncoding: #utf8.
Then, for postgres, you will be using the one from TextConverter newForEncoding: (r rows first valueAt: 1).
Even more, I would like to make it optional. Maybe we can create DBXCOnnection class >> platform: aPlatform settings: aConnectionSettings encode: anEncoder and that will do:
platform: aPlatform settings: aConnectionSettings encoder: anEncoder ^self new platform: aPlatform; settings: aConnectionSettings; encoder: anEncoder; yourself
Finally, the cleint code (your app code) creates the connection with that message :)
or something like that....but the idea of making it optional not to have overhead if not required, and being able to be enabled from the app code.
- What happens is the database supports and specific encoding that Pharo
doesn't support? TextConverter newForEncoding: (r rows first valueAt: 1).
will return nil, causing UndefinedObject dnu something.
maybe we can just set again the NullEncoder in that case? or raise an error ? what do you think?
Others database platforms just don't know anything about encoding
(they use default utf-8). If other smalltalk dialects have TextConverter then it should be fine. If they haven't then I can implement another abstraction layer to provide compatibility between dialects.
If you want to can use another new package with your commit.
I am sorry, but I did not quite understand what you meant...
Unfortunately I am a little busy at work right now so I did not had time to learn how to use monticello properly to create version only containing required changes. For this reason I attached change set file to this mail.
Don't worry :)
thanks!
Also is there a connection pooling facility for SqueakDBX or Glorp?
Yes, several.
Thank you.
-- Panu
On Thu, Aug 26, 2010 at 9:43 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Wed, Aug 25, 2010 at 7:20 AM, Panu Suominen < panu.j.m.suominen@gmail.com> wrote:
2010/8/24 Mariano Martinez Peck marianopeck@gmail.com:
Yes. So that I can see it. Do you know if it breaks compatibility with
other
databases or OS? because SqueakDBX classes like DBXConnection should be cross-backend.
I don't know how cross-platform my fix is. Database part should be fine. Postgresql can now detect the encoding.
Thanks a lot for the changeset. Now I will test with different OS and databases and I will let you know.
now a couple of questions:
- I guess a lot of people doesn't need UTF (or other) encoding. And I guess
that the transformation brings some overhead in the query result mosty...(I should run the benchmarks and see if there is difference). But I was thinking if we could make this optional. I like the code and design you did. I think that a good idea would be to add some kind of NullEncoder which is set in:
DBXConnection >> initialize
instead of TextConverter newForEncoding: #utf8.
In addition, if someone has its own database using a particular encoding, and doesn't implement openConnection (that does the query, like you did in postgres) because he doesn't knwo, this will be a problem beasue the database will have an encoding X and you will be encoding always with UTF8 ;)
Then, for postgres, you will be using the one from TextConverter newForEncoding: (r rows first valueAt: 1).
Even more, I would like to make it optional. Maybe we can create DBXCOnnection class >> platform: aPlatform settings: aConnectionSettings encode: anEncoder and that will do:
platform: aPlatform settings: aConnectionSettings encoder: anEncoder ^self new platform: aPlatform; settings: aConnectionSettings; encoder: anEncoder; yourself
Finally, the cleint code (your app code) creates the connection with that message :)
or something like that....but the idea of making it optional not to have overhead if not required, and being able to be enabled from the app code.
- What happens is the database supports and specific encoding that Pharo
doesn't support? TextConverter newForEncoding: (r rows first valueAt: 1).
will return nil, causing UndefinedObject dnu something.
maybe we can just set again the NullEncoder in that case? or raise an error ? what do you think?
Others database platforms just don't know anything about encoding
(they use default utf-8). If other smalltalk dialects have TextConverter then it should be fine. If they haven't then I can implement another abstraction layer to provide compatibility between dialects.
If you want to can use another new package with your commit.
I am sorry, but I did not quite understand what you meant...
Unfortunately I am a little busy at work right now so I did not had time to learn how to use monticello properly to create version only containing required changes. For this reason I attached change set file to this mail.
Don't worry :)
thanks!
Also is there a connection pooling facility for SqueakDBX or Glorp?
Yes, several.
Thank you.
-- Panu
squeakdbx@lists.squeakfoundation.org
-
Mariano Martinez Peck -
Panu Suominen