fcopy -size limit
in [TCL]
|
From: Frédéric Bonnet on 19 Mar 2010 16:35 Howdy folks, I've just encountered an unfortunate "feature" of fcopy: it seems that the optional -size argument only accepts integer values, not wide ints. This means that it cannot be used reliably with large files; they will need several successive fcopy with -size under the integer limit, until the desired value is reached. Not really difficult to do thanks to -command callbacks and event handling (Tcl rocks!), but nevertheless surprising, as I've found no mention of this limitation in the manpage or through the intertubes. And the source code gives no clue on the reason behind this lack of support for wide ints. So, bug or feature? In the latter case I think that the doc should be updated accordingly.
From: tom.rmadilo on 19 Mar 2010 21:30 On Mar 19, 1:35 pm, Frédéric Bonnet <fredericbon...(a)free.fr> wrote: > Howdy folks, > > I've just encountered an unfortunate "feature" of fcopy: it seems that the > optional -size argument only accepts integer values, not wide ints. This means > that it cannot be used reliably with large files; they will need several > successive fcopy with -size under the integer limit, until the desired value is > reached. Not really difficult to do thanks to -command callbacks and event > handling (Tcl rocks!), but nevertheless surprising, as I've found no mention of > this limitation in the manpage or through the intertubes. And the source code > gives no clue on the reason behind this lack of support for wide ints. > > So, bug or feature? In the latter case I think that the doc should be updated > accordingly. A very good feature. The -size limit is somewhat bogus. Internally [fcopy] does one buffer size copy and then reschedules itself so that it shares time with other Tcl events, essentially it automatically does what the -size feature is supposed to do. There are numerous document bugs related to this, but since buffer sizes are not meaningful (they can expand beyond control), who knows what needs changing. But [fcopy] itself seems to work as expected...any error would have shown up very quickly. Personally I wouldn't try to copy 2^32+ bits of data at once, but if you do, maybe [fcopy] isn't the best choice. Hopefully [fcopy] has an even lower limit. The -buffersize limit is 1M, so my guess is that 1M is the max transfer per event.
From: Tcl Bliss on 20 Mar 2010 01:37 > This means > that it cannot be used reliably with large files; they will need several > successive fcopy with -size under the integer limit, until the desired value is > reached. Actually, this is exactly the point of using -size. If you want want a single transfer, just use fcopy without the "-size" option. "-size" is meant to be used to break the transfer into smaller chunks, so you can do something else with it, like transfer monitoring, progress report, speed control, bandwidth limiting etc.
From: Alexandre Ferrieux on 20 Mar 2010 06:12 On Mar 20, 2:30 am, "tom.rmadilo" <tom.rmad...(a)gmail.com> wrote: > > Personally I wouldn't try to copy 2^32+ bits of data at once, but if > you do, maybe [fcopy] isn't the best choice. Hopefully [fcopy] has an > even lower limit. The -buffersize limit is 1M, so my guess is that 1M > is the max transfer per event. Per 'event', yes, in the sense of the internal 'fileevents' involved in async fcopy, but this has nothing to do with the -size limit (fcopy will keep going until the -size is reached). -Alex
From: Frédéric Bonnet on 20 Mar 2010 04:23
Tcl Bliss wrote: >> This means >> that it cannot be used reliably with large files; they will need several >> successive fcopy with -size under the integer limit, until the desired value is >> reached. > > Actually, this is exactly the point of using -size. If you want want a > single transfer, just use fcopy without the "-size" option. "-size" is > meant to be used to break the transfer into smaller chunks, so you can > do something else with it, like transfer monitoring, progress report, > speed control, bandwidth limiting etc. Sure, but there are other perfectly good reasons to use -size. I've encountered this "feature" with a HTTP Media Server of mine: the client may send a "Range:" header with arbitrary values, for example when playing back videos from a given position, so it makes sense to issue a single fcopy with the range length given to -size. And internally fcopy chunks the output anyway using the channel -buffersize. The whole point of fcopy here is to fire-and-forget the channel copy and let Tcl manage the background work. FYI fcopy works fine with -size values of 2GB, and I've worked around this limitation without much work. But my point is: if the -size limit makes sense (which I doubt), it should be documented. I guess that's the best option anyway, as there are certainly other priorities than fixing this arbitrary limitation. |