[Seaside] Zinc-Seaside: Problem with POSTing UTF-8 JSON
Tomas Kukol
tomas.kukol at gmail.com
Wed Jun 26 13:35:26 UTC 2013
Hi Sven,
thanks for the answers. My REST services are really simple, for example
there is a list of users:
GPRestfulHandler>>getUsers
<get>
<path: 'user/list'>
<produces: 'application/json; charset=utf-8'>
| instance container items |
instance := GPUser new.
container := instance descriptionId , instance descriptionLogin.
items := self database getUsers.
^ (items collect: [ :item | item asDictionaryFrom: container ]) asJson
There is a JSON result:
[{"Id":"688a7ymp9zn60qczvvyniprwq","Login":"tomas"},{"Id":"av0t1s9u4ddu747hd2xyfi58","Login":"petr"}]
Detail of user can be read by this method:
GPRestfulHandler>>getUserByLogin: aLogin
<get>
<path: 'user/detail/by-login/{aLogin}'>
<produces: 'application/json; charset=utf-8'>
| user |
user := self database getUserByLogin: aLogin.
user isNil
ifFalse: [ ^ user asJson ]
ifTrue: [ self respondAsUserWasNotFindByLogin: aLogin ]
Result is here. I had to fix the failing ByteArray method to get GoalName
with Czech characters:
{"Id":"688a7ymp9zn60qczvvyniprwq","Login":"tomas","CurrentAccountBalance":12000,"SavingsAccountReserve":0,"SavingsAccountHeap":0,"InterestRate":1,"PaymentDay":15,"SimulationDate":"29
May 2013","SavingsAccountInterestReserve":0,"SavingsAccountInterestHeap":0,"Goals":[{"GoalName":"ěščřžýáíé-ĚŠČŘŽÝÁÍÉ","GoalAmount":8000,"MaturityDate":"29
November 2013"}],"Plans":[{"PaymentDate":"29 May
2013","PaymentAmount":1140.410292918764},{"PaymentDate":"29 June
2013","PaymentAmount":1140.410292918764},{"PaymentDate":"29 July
2013","PaymentAmount":1140.410292918764},{"PaymentDate":"29 August
2013","PaymentAmount":1140.410292918764},{"PaymentDate":"29 September
2013","PaymentAmount":1140.410292918764},{"PaymentDate":"29 October
2013","PaymentAmount":1140.410292918764},{"PaymentDate":"29 November
2013","PaymentAmount":1140.410292918764}]}
Can you see any problem with the REST method?
Tomas
On Wed, Jun 26, 2013 at 2:00 PM, Sven Van Caekenberghe <sven at stfx.eu> wrote:
> Hi Tomas,
>
> On 26 Jun 2013, at 13:20, Tomas Kukol <tomas.kukol at gmail.com> wrote:
>
> > Hi Sven,
> >
> > you are right. Seaside needs it as a string. But I've found another
> problem with output (maybe because Seaside uses string???) which leads to
> error 'Improper store into indexable object'.
> >
> > I have JSON on output. It is again string and there is this problem. In
> ZnZincServerAdaptor>>responseFrom: we are asking for contents.
>
> I don't know much about Seaside-REST but I would guess, from my general
> understanding how Seaside works, that if you generate output, like JSON,
> your output has to pass through an encoder before it is stored as bytes
> (well actually a ByteString, but that is another story) inside
> RWBinaryOrTextStream's contents of the WABufferedResponse. The encoder will
> make sure WideCharacters get encoded properly and then the
> #replaceFrom:to:with:startingAt: will work (and magically bridge the
> ByteArray/ByteString gap).
>
> So my guess is that you either have not set the proper encoder on the
> response when writing output to it, or you write the output the wrong way.
>
> There has to be a codec stream in between. See
>
> WAServerAdaptor>>#responseFor:
> GRPharoUtf8Codec>>#encoderFor:
>
> HTH,
>
> Sven
>
> > ZnZincServerAdaptor>>responseFrom: aRequestContext
> > | partialHeaders cookies fullHeaders seasideResponse contents
> entity contentType |
> > seasideResponse := aRequestContext response.
> > partialHeaders := seasideResponse headers.
> > cookies := seasideResponse cookies.
> > fullHeaders := ZnHeaders defaultResponseHeaders.
> > partialHeaders keysAndValuesDo: [ :key :value |
> > fullHeaders at: key put: value ].
> > cookies do: [ :each |
> > fullHeaders at: 'Set-Cookie' add: each oldNetscapeString.
> > fullHeaders at: 'Set-Cookie2' add: each rfc2965String ].
> > contentType := seasideResponse contentType greaseString.
> > contents := seasideResponse contents. "<------------------------
> HERE - we are asking for contents"
> > entity := (ZnEntity bytes: contents) contentType: contentType;
> yourself.
> > ^ ZnResponse new
> > statusLine: (ZnStatusLine code: seasideResponse status);
> > headers: fullHeaders;
> > entity: entity;
> > yourself
> >
> > It calls:
> >
> > WABufferedResponse>>contents
> > ^ contentsStream contents "<------------------------ HERE -
> normally, asking underlying stream for contents"
> >
> > An the error is coming here:
> >
> > RWBinaryOrTextStream>>contents
> > "Answer with a copy of my collection from 1 to readLimit."
> >
> > | newArray |
> > isBinary ifFalse: [^ super contents]. "String"
> > readLimit := readLimit max: position.
> > newArray := ByteArray new: readLimit. "<------------------------
> HERE - we are creating array of bytes (SmallInteger 0 - 255)"
> > ^ newArray replaceFrom: 1
> > to: readLimit
> > with: collection "<------------------------ HERE - but we
> are having WideString in collection - characters"
> > startingAt: 1.
> >
> > Now we are calling primitive 105:
> >
> > ByteArray>>replaceFrom: start to: stop with: replacement startingAt:
> repStart
> > "Primitive. This destructively replaces elements from start to
> stop in the receiver starting at index, repStart, in the collection,
> replacement. Answer the receiver. Range checks are performed in the
> primitive only. Optional. See Object documentation whatIsAPrimitive."
> > <primitive: 105>
> > super replaceFrom: start to: stop with: replacement startingAt:
> repStart "<------------------------ HERE - it fails and continues here"
> >
> > It calls loop in SequenceableCollection:
> >
> > ByteArray(SequenceableCollection)>>replaceFrom: start to: stop with:
> replacement startingAt: repStart
> > "This destructively replaces elements from start to stop in the
> receiver
> > starting at index, repStart, in the sequenceable collection,
> > replacementCollection. Answer the receiver. No range checks are
> > performed."
> >
> > | index repOff |
> > repOff := repStart - start.
> > index := start - 1.
> > [(index := index + 1) <= stop]
> > whileTrue: [self at: index put: (replacement at: repOff +
> index)] "<------------------------ HERE - here we are putting character
> into ByteArray"
> >
> > Stops here at Object at primitive 61:
> >
> > ByteArray(Object)>>at: index put: value
> > "Primitive. Assumes receiver is indexable. Store the argument
> value in
> > the indexable element of the receiver indicated by index. Fail if
> the
> > index is not an Integer or is out of bounds. Or fail if the value
> is not of
> > the right type for this kind of collection. Answer the value that
> was
> > stored. Essential. See Object documentation whatIsAPrimitive."
> >
> > <primitive: 61>
> > index isInteger ifTrue:
> > [self class isVariable
> > ifTrue: [(index >= 1 and: [index <= self size])
> > ifTrue: [self errorImproperStore]
> "<------------------------ HERE - ends here"
> > ifFalse: [self
> errorSubscriptBounds: index]]
> > ifFalse: [self errorNotIndexable]].
> > index isNumber
> > ifTrue: [^self at: index asInteger put: value]
> > ifFalse: [self errorNonIntegerIndex]
> >
> > I've seen this problem mostly with Seaside-REST, but also in some other
> posts from Seaside.
> >
> > Any idea how to fix it?
> >
> > Regards,
> > Tomas
> >
> >
> > On Wed, Jun 26, 2013 at 12:17 AM, Sven Van Caekenberghe <sven at stfx.eu>
> wrote:
> > Hi Tomas,
> >
> > Thanks for the report: this is an important area indeed.
> >
> > However, the comment says it: this is intentional, and as far as I know,
> it is correct. You see, Zinc will correctly decode/encode any HTTP payload,
> using the supplied mime-type and/or charsets. Seaside is written such that
> it does not want this: it insists on doing this on its own. That is why
> ZnZincServerAdaptor is _not_ using the normal request reading code of Zn,
> but uses a special option to read everything binary. The stupid thing is
> that even though Seaside needs bytes, it wants them as a String. That is
> the reason for the otherwise brain dead #asString (and the implicit copy is
> inefficient as well).
> >
> > Of course, you will see your special characters there if you do UTF8
> decoding, but that is because you already know what is inside.
> >
> > What normally happens, is that later on in the processing, Seaside will
> access the WARequest payload using proper decoding, using its own framework
> (much like what Zn would do). AFAIK this whole process works. You can
> actually test this using some of the functional tests.
> >
> > I am not sure that Seaside-REST is doing the right thing (there were
> some issues with SmalltalkHub as well), but I would think so.
> >
> > Are you sure you have set the correct encoding on the adaptor ?
> >
> > Are you sure you are posting as application/json;charset=utf-8 and if
> you do not set the charset, are you sure utf-8 is the default ?
> >
> > Are you sure your REST handler and/or JSON parser does the right thing ?
> >
> > It is too late right now, but if we want to get further with this, I
> will need a failing unit test - if these exist in Seaside-REST, but I would
> assume so. I have no experience running Seaside-REST, I am using Zinc-REST
> myself, but I would like to learn.
> >
> > Regards,
> >
> > Sven
> >
> > On 25 Jun 2013, at 23:24, Tomas Kukol <tomas.kukol at gmail.com> wrote:
> >
> > > Hi Sven.
> > >
> > > I've had a problem when POSTing non-ascii UTF-8 characters in JSON to
> Seaside REST service. I've located the problem in the method
> ZnZincServerAdaptor>>requestBodyFor: where the body of ZnRequest is
> translated to body of WARequest. I use Pharo 1.4 with Seaside 3.0.8 and
> Zinc-Seaside-SvenVanCaekenberghe.40.
> > >
> > > When the POSTed JSON contains non-ascii UTF-8 characters (Czech
> characters), they are corrupted. The problem is on the "MARKED" line, where
> the array of bytes changed to string by asString.
> > >
> > > "Problematic" code:
> > >
> > > ZnZincServerAdaptor>>requestBodyFor: aZincRequest
> > > ^ (aZincRequest method ~= #TRACE
> > > and: [ aZincRequest hasEntity
> > > and: [ aZincRequest entity isEmpty not
> > > and: [ (aZincRequest entity contentType
> matches: ZnMimeType applicationFormUrlEncoded) not
> > > and: [ (aZincRequest entity
> contentType matches: ZnMimeType multiPartFormData) not ] ] ] ])
> > > ifTrue: [
> > > "Seaside wants to do its own text
> conversions"
> > > aZincRequest entity bytes asString
> "MARKED" ]
> > > ifFalse: [
> > > String new ]
> > >
> > > I did a quick correction, which is not nice, but works for me:
> > >
> > > ZnZincServerAdaptor>>requestBodyFor: aZincRequest
> > > ^ (aZincRequest method ~= #TRACE
> > > and: [ aZincRequest hasEntity
> > > and: [ aZincRequest entity isEmpty not
> > > and: [ (aZincRequest entity contentType
> matches: ZnMimeType applicationFormUrlEncoded) not
> > > and: [ (aZincRequest entity
> contentType matches: ZnMimeType multiPartFormData) not ] ] ] ])
> > > ifTrue: [
> > > "Seaside wants to do its own text
> conversions"
> > > ZnUTF8Encoder new decodeBytes:
> aZincRequest entity bytes "CORRECTED" ]
> > > ifFalse: [
> > > String new ]
> > >
> > > My correction tries to decode byte array with ZnUTF8Encoder and the
> result is OK.
> > >
> > > Maybe I would recommend to use GRPharoUtf8Codec (although I like
> ZnUTF8Encoder more) or even better self codec (self = ZnZincServerAdaptor)
> to try to decode the bytes.
> > >
> > > Regards,
> > > Tomas Kukol
> > > _______________________________________________
> > > seaside mailing list
> > > seaside at lists.squeakfoundation.org
> > > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> >
> >
> >
> > --
> > Sven Van Caekenberghe
> > http://stfx.eu
> > Smalltalk is the Red Pill
> >
> > _______________________________________________
> > seaside mailing list
> > seaside at lists.squeakfoundation.org
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> >
> > _______________________________________________
> > seaside mailing list
> > seaside at lists.squeakfoundation.org
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> seaside at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/seaside/attachments/20130626/5693f88b/attachment-0001.htm
More information about the seaside
mailing list