Prev: Merging two dictionaries
Next: run subprocesses in parallel
From: Sanjeeb on 2 Aug 2010 02:52
Hi,
I have a web client which send a file to a server as multipart form
data, the sending of data is from
http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.

I dont want to open the whole file to memory(at cliend end) and then
send, i just want to send part by part, say chunk of 1024 bytes to the
server and then assemble at the server end.

Could some one suggest what would be the best way to do this?

Regards
-Sanjeeb
From: Sanjeeb on 3 Aug 2010 01:03
On Aug 2, 11:52 am, Sanjeeb <sanjee...(a)gmail.com> wrote:
> Hi,
> I have a web client which send a file to a server as multipart form
> data, the sending of data is fromhttp://code.activestate.com/recipes/146306-http-client-to-post-using-....
>
> I dont want to open the whole file to memory(at cliend end) and then
> send, i just want to send part by part, say chunk of 1024 bytes to the
> server and then assemble at the server end.
>
> Could some one suggest what would be the best way to do this?
>
> Regards
> -Sanjeeb

Hey any taker for this question???
From: Kushal Kumaran on 3 Aug 2010 01:15
On Mon, Aug 2, 2010 at 12:22 PM, Sanjeeb <sanjeeb25(a)gmail.com> wrote:
> Hi,
> I have a web client which send a file to a server as multipart form
> data, the sending of data is from
> http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.
>
> I dont want to open the whole file to memory(at cliend end) and then
> send, i just want to send part by part, say chunk of 1024 bytes to the
> server and then assemble at the server end.
>
> Could some one suggest what would be the best way to do this?
>

There's no reason why sending the whole file implies reading the whole
file into memory at one time. You can just read your desired chunk
size and send it, then read the next chunk, and so on. You might have
to first find the total size to calculate what to set Content-Length
to.

ISTR questions of this nature have been asked in the past on this list.

--
regards,
kushal
From: Ryan Kelly on 3 Aug 2010 01:52
On Tue, 2010-08-03 at 10:45 +0530, Kushal Kumaran wrote:
> On Mon, Aug 2, 2010 at 12:22 PM, Sanjeeb <sanjeeb25(a)gmail.com> wrote:
> > Hi,
> > I have a web client which send a file to a server as multipart form
> > data, the sending of data is from
> > http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.
> >
> > I dont want to open the whole file to memory(at cliend end) and then
> > send, i just want to send part by part, say chunk of 1024 bytes to the
> > server and then assemble at the server end.
> >
> > Could some one suggest what would be the best way to do this?
> >
>
> There's no reason why sending the whole file implies reading the whole
> file into memory at one time. You can just read your desired chunk
> size and send it, then read the next chunk, and so on. You might have
> to first find the total size to calculate what to set Content-Length
> to.

More concretely, you would restructure the encode_multipart_formdata()
function as a generator, yielding chunks of data to send one at a time.
Something like this:

def post_multipart(host,selector,fields,files):
...snip...
h.endheaders()
for chunk in encode_multipart_formdata_chunks(fields,files):
h.send(chunk)
errcode, errmsg, headers = h.getreply()
return h.file.read()


def encode_multipart_formdata(fields,files):
BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
CRLF = '\r\n'
for (key, value) in fields:
yield '--' + BOUNDARY
yield 'Content-Disposition: form-data; name="%s"' % key
yield ''
yield value
for (key, filename, value) in files:
yield '--' + BOUNDARY
yield 'Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename)
...etc...
...etc...

There are many improvements to make, but this should get you started.
For example, you'll need to calculate the total content-length rather
than just calling len(body) to obtain it. That's left as an exercise to
the reader.


Ryan


--
Ryan Kelly
http://www.rfk.id.au | This message is digitally signed. Please visit
ryan(a)rfk.id.au | http://www.rfk.id.au/ramblings/gpg/ for details

 | 
Pages: 1
Prev: Merging two dictionaries
Next: run subprocesses in parallel