Upload large, was: Re: [Seaside] ZINC - Kom ...

[Robert Kuszinger]

Sven,

thanks for the demo. Zn without Seaside is just fine if it could work. A
one-field form with only the uploaded file could work also. Some javascript
addition on client side is acceptable - I'll see then. I understand that a
simple file upload is also "composite" data with filename and binary
content...

*Usage*: An office need to receive large map / digital survey data files
from clients. Now they post it on CD or DVD disks, however the typical
amount is 100-200 Mb in one or two or more files (depending one who has
heard about ZIP and who hasn't :) - really! ). So we are trying to create
an upload portal where they could login and then upload files to folders
where folder name contains their ID and date. That's it.

No, SSH/SFTP or FTP with OS auth is not acceptable. They want pure browser
upload as clients know this from their everyday life. And they could also
add metadata about their uploads.

Login, auth to existing client database is done in Seaside/Pharo in a few
hours, works nicely.

I would be great to create the upload receiving part also with Pharo at
least.

All this stuff is behind and IIS/ARR - tested for large uploads, worked
well when extending timeout limitations is IIS (with Kom but eating memory,
maybe not so much as Zinc now, but it had the codepage problem I wanted
debug earlier). OS is Windows Server 2008 R2Datacenter Edition, IIS 7.5.

I'm developing on Linux and testing on Windows Server 2008 configured to
the same setup (IIS, ARR, etc.)

This is the scenario.

Robert



Sven Van Caekenberghe <sven at stfx.eu> ezt írta (időpont: 2016. jan. 18., H,
16:30):

> Robert,
>
> This is not such an easy problem, you have to really understand HTTP.
>
> BTW, such huge uploads don't seem a very good idea anyway, you will get
> annoying timeouts as well. I am curious, what is in those files ?
>
> Now, here is the key idea (pure Zn, no Seaside, quick hack):
>
> (ZnServer startOn: 1701)
>   reader: [ :stream | ZnRequest readStreamingFrom: stream ];
>   maximumEntitySize: 100*1024*1024;
>   onRequestRespond: [ :req |
>     '/tmp/upload.bin' asFileReference writeStreamDo: [ :out |
>        out binary.
>        ZnUtils streamFrom: req entity stream to: out ].
>     ZnResponse ok: (ZnEntity text: 'done') ];
>   yourself.
>
> You would use it like this:
>
> $ echo one two three > data.bin
> $ curl -X POST -d @data.bin http://localhost:1701
> $ cat /tmp/upload.bin
> one two three
>
> With a 1Mb data file generated from Pharo:
>
> '/tmp/data.txt' asFileReference writeStreamDo: [ :out |
>   1 * 1024 timesRepeat: [
>     1 to: 32 do: [ :each |
>       out << Character alphabet << (each printStringPadded: 5); lf ] ] ]
>
> $ curl -v -X POST --data-binary @data2.bin http://localhost:1701
> * Rebuilt URL to: http://localhost:1701/
> *   Trying ::1...
> * connect to ::1 port 1701 failed: Connection refused
> *   Trying 127.0.0.1...
> * Connected to localhost (127.0.0.1) port 1701 (#0)
> > POST / HTTP/1.1
> > Host: localhost:1701
> > User-Agent: curl/7.43.0
> > Accept: */*
> > Content-Length: 1048576
> > Content-Type: application/x-www-form-urlencoded
> > Expect: 100-continue
> >
> * Done waiting for 100-continue
> * We are completely uploaded and fine
> < HTTP/1.1 200 OK
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 4
> < Date: Mon, 18 Jan 2016 14:56:53 GMT
> < Server: Zinc HTTP Components 1.0
> <
> * Connection #0 to host localhost left intact
> done
>
> $ diff data2.bin /tmp/upload.bin
>
> This code is totally incomplete, you need lots of error handling.
> Furthermore, working with streaming requests is dangerous, because you are
> responsible for reading the bodies correctly.
>
> Also, if you want an upload in a form, you will have to parse that form
> (see ZnApplicationFormUrlEncodedEntity and ZnMultiPartFormDataEntity), you
> will then again take it in memory. These things are normally done for you
> by Zn and/or Seaside.
>
> I also tried with 100Mb, it worked, but it took several minutes, like 10
> to 15. The #streamFrom:to: above uses a 16Kb buffer, which is probably too
> small for this use case. Maybe curl doesn't upload very aggressively.
> Performance is another issue.
>
> That is why I asked what is in the files, what you eventually want to do
> with it. Is the next processing step in Pharo too ?
>
> Maybe all that is needed is giving Pharo more memory. What platform are
> you on ?
>
> Sven
>
> > On 18 Jan 2016, at 13:39, Robert Kuszinger <kuszinger at giscom.hu> wrote:
> >
> > Sven,
> >
> > thanks for the comments. I understand all.
> >
> > Could you please clarify this:
> >
> > "Technically, it would be possible to write a Zn handler that can accept
> a large upload in a streaming fashion (and save it to a file for example),
> but I don't think that will happen with Seaside - so you will pull that all
> in memory."
> >
> > Is there a chance to create a streaming solution? Is it documented
> somewhere? Why do you think it won't happen with Seaside? Is there a
> Seaside set or design limitation?
> >
> >
> > Answering on how it goes:
> >
> > 20 - 40 - 10MB upload in seconds. Now it seems to stuck on a ~ 120 MB
> upload. Pharo memory (windows OS) seemed to grow ~ 441 MB. "Space is low"
> warning window appeared. I've clicked on "Proceed" just for curiosity but
> no reaction in the Pharo gui... hmmm...
> >
> >
> > thanks
> > Robert
> >
> >
> >
> >
> > 2016-01-18 13:27 GMT+01:00 Sven Van Caekenberghe <sven at stfx.eu>:
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/seaside/attachments/20160118/d687dd4b/attachment-0001.htm