how to upload large file ( bigger than 1G) with PHP
in [General]
|
Prev: PHP MySQL Insert Statements
Next: headers help
From: Michael Shadle on 12 Mar 2010 14:37 On Fri, Mar 12, 2010 at 4:41 AM, Ashley Sheridan <ash(a)ashleysheridan.co.uk> wrote: > I've noticed that large uploads over http seem to behave a little > unpredictably at times, and aren't something I'd rely on. FTP is > definitely the way to go, and there are plenty of Java applets that > allow you to do this. FTP is not a realistic option, for a multitude of reasons. a) mapping an HTTP request and user -> FTP account / "pick this file up" b) firewall issues c) additional services having to be enabled and routed to on the server side While I do agree FTP is FILE transfer protocol, it still isn't the right solution IMHO. Ideally, HTML5 will provide a more industry standard method (IIRC, a coworker already pointed out something in the spec for it, but I forget) A very workable solution we've came up with has been using Google Gears + PHP. Re-using the browser and HTTP conversations provides us multiple benefits: a) Cookie support - to identify the user b) supports HTTP and HTTPS c) Firewalls are not an issue - reuses the same proxy settings The difference between standard file upload using a single POST vs. our method is key - it's chunking the file. Google Gears has this support, Java can too; send up portions of the file at a time, and either glue it together on the fly on the server, or take all the chunks and merge them all at once at the end. By doing it in a chunked format, it allows us to also re-transmit failed chunks and treat files of any size in "bite size chunks" - with a little bit of Javascript, PHP and Gears, we can support files of any size (within filesystem and OS limits) and it does not require -any- tweaking of the webserver. It is chunks of data sent to the server using standard POSTs and small enough to fit under even small PHP and webserver memory limits (and could always be configurable) - no more suhosin.memory_limit, memory_limit, post_max_size, upload_max_filesize to fuss with. It's a shame that Google had to decide to stop developing and maintaining Gears. It was a lightweight, perfect solution. We're working on a Java-based version instead now. Lightest footprint we can possibly get in Java, but it's the only applet language that has all the support we need for chunking, cross-browser, cross-platform, etc. I believe our plan is to release it out to the public so people can enhance it, use it, do whatever... For now though, Gears works pretty awesome for us, a handful of our users have complained though Gears won't install for them (not sure why) and there is no support for Snow Leopard, I believe. So we're starting to hit the point where it isn't our magical solution anymore.
From: Ashley Sheridan on 12 Mar 2010 14:51 On Fri, 2010-03-12 at 11:37 -0800, Michael Shadle wrote: > On Fri, Mar 12, 2010 at 4:41 AM, Ashley Sheridan > <ash(a)ashleysheridan.co.uk> wrote: > > > I've noticed that large uploads over http seem to behave a little > > unpredictably at times, and aren't something I'd rely on. FTP is > > definitely the way to go, and there are plenty of Java applets that > > allow you to do this. > > FTP is not a realistic option, for a multitude of reasons. > > a) mapping an HTTP request and user -> FTP account / "pick this file up" > b) firewall issues > c) additional services having to be enabled and routed to on the server side > > While I do agree FTP is FILE transfer protocol, it still isn't the > right solution IMHO. Ideally, HTML5 will provide a more industry > standard method (IIRC, a coworker already pointed out something in the > spec for it, but I forget) > > A very workable solution we've came up with has been using Google Gears + PHP. > > Re-using the browser and HTTP conversations provides us multiple benefits: > a) Cookie support - to identify the user > b) supports HTTP and HTTPS > c) Firewalls are not an issue - reuses the same proxy settings > > The difference between standard file upload using a single POST vs. > our method is key - it's chunking the file. Google Gears has this > support, Java can too; send up portions of the file at a time, and > either glue it together on the fly on the server, or take all the > chunks and merge them all at once at the end. By doing it in a chunked > format, it allows us to also re-transmit failed chunks and treat files > of any size in "bite size chunks" - with a little bit of Javascript, > PHP and Gears, we can support files of any size (within filesystem and > OS limits) and it does not require -any- tweaking of the webserver. It > is chunks of data sent to the server using standard POSTs and small > enough to fit under even small PHP and webserver memory limits (and > could always be configurable) - no more suhosin.memory_limit, > memory_limit, post_max_size, upload_max_filesize to fuss with. > > It's a shame that Google had to decide to stop developing and > maintaining Gears. It was a lightweight, perfect solution. > > We're working on a Java-based version instead now. Lightest footprint > we can possibly get in Java, but it's the only applet language that > has all the support we need for chunking, cross-browser, > cross-platform, etc. > > I believe our plan is to release it out to the public so people can > enhance it, use it, do whatever... > > For now though, Gears works pretty awesome for us, a handful of our > users have complained though Gears won't install for them (not sure > why) and there is no support for Snow Leopard, I believe. So we're > starting to hit the point where it isn't our magical solution anymore. > It's not much trouble to map the FTP to a file and have the right permissions, and FTP is a doddle to set up on a server. I'd say a darn sight less work than rolling your own mechanism in Java. Lastly, I don't think firewalls are that big an issue, as most firewalls I've seen will allow outgoing FTP connections from a users computer by default. Thanks, Ash http://www.ashleysheridan.co.uk
From: Michael Shadle on 12 Mar 2010 16:18
On Fri, Mar 12, 2010 at 11:51 AM, Ashley Sheridan <ash(a)ashleysheridan.co.uk> wrote: > It's not much trouble to map the FTP to a file and have the right permissions, and FTP is a doddle to set up on a server. I'd say a darn sight less work than rolling your own mechanism in Java. Well, mechanisms already exist. We're just trying to create a cleaner one that works with the browser's DOM so it looks native in the browser, has the chunk support, etc. FTP servers can be "easy" to setup but mapping who uploaded what and keeping that secure is a pain. Do you use one generic account, or one random account per user? If one generic account, how do you keep others from downloading someone else's content? etc? Lots of questions come to mind. But lunch is more important... :) > Lastly, I don't think firewalls are that big an issue, as most firewalls I've seen will allow outgoing FTP connections from a users computer by default. Must not deal with that many corporate firewalls :) |