Upload large, was: Re: [Seaside] ZINC - Kom ...

Robert Kuszinger kuszinger at giscom.hu
Mon Jan 18 18:24:37 UTC 2016 

Hmmm. Putting upload one step out in the frontline..

I ask my office also about nginx and also test install for myself.

Anyway, a pure pharo streaming solution is still interesting for me.

R
2016.01.18. 19:20 ezt írta ("Tobias Pape" <Das.Linux at gmx.de>):

> Hi Johan
> On 18.01.2016, at 19:18, Johan Brichau <johan at inceptive.be> wrote:
>
> > Hi Tobias,
> >
> > This is what we do since years :)
> > There was a blog post online describing all the details but I don’t find
> it anymore.
> > The only think I can find is Nick Ager’s reply when we had a little
> trouble setting it up [1]
> >
> > I might try to separate off the code to spare you some time.
> > It’s quite simple, actually.
>
> Oh that would be just great. My students would rejoice to no longer see
> the spurious "503 gateway timed out" messaged ;)
>
> Best regards
>         -Tobias
>
>
> >
> > [1] http://forum.world.st/Using-nginx-file-upload-module-td3591666.html
> >
> > Johan
> >
> >> On 18 Jan 2016, at 18:56, Tobias Pape <Das.Linux at gmx.de> wrote:
> >>
> >> Hey all
> >>
> >> just my 2ct while skimming the thread.
> >>
> >> I have upload problems with my seaside app
> >> and plan to tackle them by utilizing the reverse proxy.
> >> In my scenario, that is nginx, wich ships an "upload module"
> >>      https://www.nginx.com/resources/wiki/modules/upload/
> >>
> >> Given that, the upload is handled by the reverse proxy and only when
> >> the file is already on the file system, the backend (seaside in this
> case)
> >> would get a notification request.
> >>
> >> I plan to implement this within the next 6 weeks, so if I get going
> something
> >> usable, I'll probably hand it back to the seaside community :)
> >> Remind me if I forget ;)
> >>
> >> best regards
> >>      -Tobias
> >>
> >>
> >> On 18.01.2016, at 18:00, Robert Kuszinger <kuszinger at giscom.hu> wrote:
> >>
> >>>
> >>> Sven,
> >>>
> >>> thanks for the demo. Zn without Seaside is just fine if it could work.
> A one-field form with only the uploaded file could work also. Some
> javascript addition on client side is acceptable - I'll see then. I
> understand that a simple file upload is also "composite" data with filename
> and binary content...
> >>>
> >>> Usage: An office need to receive large map / digital survey data files
> from clients. Now they post it on CD or DVD disks, however the typical
> amount is 100-200 Mb in one or two or more files (depending one who has
> heard about ZIP and who hasn't :) - really! ). So we are trying to create
> an upload portal where they could login and then upload files to folders
> where folder name contains their ID and date. That's it.
> >>>
> >>> No, SSH/SFTP or FTP with OS auth is not acceptable. They want pure
> browser upload as clients know this from their everyday life. And they
> could also add metadata about their uploads.
> >>>
> >>> Login, auth to existing client database is done in Seaside/Pharo in a
> few hours, works nicely.
> >>>
> >>> I would be great to create the upload receiving part also with Pharo
> at least.
> >>>
> >>> All this stuff is behind and IIS/ARR - tested for large uploads,
> worked well when extending timeout limitations is IIS (with Kom but eating
> memory, maybe not so much as Zinc now, but it had the codepage problem I
> wanted debug earlier). OS is Windows Server 2008 R2Datacenter Edition, IIS
> 7.5.
> >>>
> >>> I'm developing on Linux and testing on Windows Server 2008 configured
> to the same setup (IIS, ARR, etc.)
> >>>
> >>> This is the scenario.
> >>>
> >>> Robert
> >>>
> >>>
> >>>
> >>> Sven Van Caekenberghe <sven at stfx.eu> ezt írta (időpont: 2016. jan.
> 18., H, 16:30):
> >>> Robert,
> >>>
> >>> This is not such an easy problem, you have to really understand HTTP.
> >>>
> >>> BTW, such huge uploads don't seem a very good idea anyway, you will
> get annoying timeouts as well. I am curious, what is in those files ?
> >>>
> >>> Now, here is the key idea (pure Zn, no Seaside, quick hack):
> >>>
> >>> (ZnServer startOn: 1701)
> >>> reader: [ :stream | ZnRequest readStreamingFrom: stream ];
> >>> maximumEntitySize: 100*1024*1024;
> >>> onRequestRespond: [ :req |
> >>>  '/tmp/upload.bin' asFileReference writeStreamDo: [ :out |
> >>>     out binary.
> >>>     ZnUtils streamFrom: req entity stream to: out ].
> >>>  ZnResponse ok: (ZnEntity text: 'done') ];
> >>> yourself.
> >>>
> >>> You would use it like this:
> >>>
> >>> $ echo one two three > data.bin
> >>> $ curl -X POST -d @data.bin http://localhost:1701
> >>> $ cat /tmp/upload.bin
> >>> one two three
> >>>
> >>> With a 1Mb data file generated from Pharo:
> >>>
> >>> '/tmp/data.txt' asFileReference writeStreamDo: [ :out |
> >>> 1 * 1024 timesRepeat: [
> >>>  1 to: 32 do: [ :each |
> >>>    out << Character alphabet << (each printStringPadded: 5); lf ] ] ]
> >>>
> >>> $ curl -v -X POST --data-binary @data2.bin http://localhost:1701
> >>> * Rebuilt URL to: http://localhost:1701/
> >>> *   Trying ::1...
> >>> * connect to ::1 port 1701 failed: Connection refused
> >>> *   Trying 127.0.0.1...
> >>> * Connected to localhost (127.0.0.1) port 1701 (#0)
> >>>> POST / HTTP/1.1
> >>>> Host: localhost:1701
> >>>> User-Agent: curl/7.43.0
> >>>> Accept: */*
> >>>> Content-Length: 1048576
> >>>> Content-Type: application/x-www-form-urlencoded
> >>>> Expect: 100-continue
> >>>>
> >>> * Done waiting for 100-continue
> >>> * We are completely uploaded and fine
> >>> < HTTP/1.1 200 OK
> >>> < Content-Type: text/plain;charset=utf-8
> >>> < Content-Length: 4
> >>> < Date: Mon, 18 Jan 2016 14:56:53 GMT
> >>> < Server: Zinc HTTP Components 1.0
> >>> <
> >>> * Connection #0 to host localhost left intact
> >>> done
> >>>
> >>> $ diff data2.bin /tmp/upload.bin
> >>>
> >>> This code is totally incomplete, you need lots of error handling.
> Furthermore, working with streaming requests is dangerous, because you are
> responsible for reading the bodies correctly.
> >>>
> >>> Also, if you want an upload in a form, you will have to parse that
> form (see ZnApplicationFormUrlEncodedEntity and ZnMultiPartFormDataEntity),
> you will then again take it in memory. These things are normally done for
> you by Zn and/or Seaside.
> >>>
> >>> I also tried with 100Mb, it worked, but it took several minutes, like
> 10 to 15. The #streamFrom:to: above uses a 16Kb buffer, which is probably
> too small for this use case. Maybe curl doesn't upload very aggressively.
> Performance is another issue.
> >>>
> >>> That is why I asked what is in the files, what you eventually want to
> do with it. Is the next processing step in Pharo too ?
> >>>
> >>> Maybe all that is needed is giving Pharo more memory. What platform
> are you on ?
> >>>
> >>> Sven
> >>>
> >>>> On 18 Jan 2016, at 13:39, Robert Kuszinger <kuszinger at giscom.hu>
> wrote:
> >>>>
> >>>> Sven,
> >>>>
> >>>> thanks for the comments. I understand all.
> >>>>
> >>>> Could you please clarify this:
> >>>>
> >>>> "Technically, it would be possible to write a Zn handler that can
> accept a large upload in a streaming fashion (and save it to a file for
> example), but I don't think that will happen with Seaside - so you will
> pull that all in memory."
> >>>>
> >>>> Is there a chance to create a streaming solution? Is it documented
> somewhere? Why do you think it won't happen with Seaside? Is there a
> Seaside set or design limitation?
> >>>>
> >>>>
> >>>> Answering on how it goes:
> >>>>
> >>>> 20 - 40 - 10MB upload in seconds. Now it seems to stuck on a ~ 120 MB
> upload. Pharo memory (windows OS) seemed to grow ~ 441 MB. "Space is low"
> warning window appeared. I've clicked on "Proceed" just for curiosity but
> no reaction in the Pharo gui... hmmm...
> >>>>
> >>>>
> >>>> thanks
> >>>> Robert
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> 2016-01-18 13:27 GMT+01:00 Sven Van Caekenberghe <sven at stfx.eu>:
> >>>>
> >>> _______________________________________________
> >>> seaside mailing list
> >>> seaside at lists.squeakfoundation.org
> >>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> >>
> >> _______________________________________________
> >> seaside mailing list
> >> seaside at lists.squeakfoundation.org
> >> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> >
> > _______________________________________________
> > seaside mailing list
> > seaside at lists.squeakfoundation.org
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
> _______________________________________________
> seaside mailing list
> seaside at lists.squeakfoundation.org
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.squeakfoundation.org/pipermail/seaside/attachments/20160118/495dda87/attachment-0001.htm

More information about the seaside mailing list