Prev: Textfile with unwanted spaces in words
Next: compress all *.c files in a hierarchy?
From: marc on 15 Dec 2009 08:55
Hello,

I gave a big zipped file (foo.gz) and I would want to split it into n
small files, but in memory, because I haven't enough space on my FS.
I tried with mkfifo, gunzip and split, but it splits first all the
files on disk and it stops logically with
"No space left on device" error.
Is there another way ?

Thanks.
From: Stephane Chazelas on 15 Dec 2009 15:24
2009-12-15, 05:55(-08), marc:
[...]
> I gave a big zipped file (foo.gz) and I would want to split it into n
> small files, but in memory, because I haven't enough space on my FS.
> I tried with mkfifo, gunzip and split, but it splits first all the
> files on disk and it stops logically with
> "No space left on device" error.
> Is there another way ?
[...]

If you mean load the file in memory, then remove it and then
create the split files, then with zsh (other shells can't cope
with binary data).:

print -rn -- "$(cat foo.gz && rm foo.gz") | split -b 10m - foo.gz.

However note that if foo.gz ends in newline characters, they
will be removed. It's probably not gonna be very efficient

You could use perl instead:

perl -0777 -pe 'unlink $ARGV' foo.gz | split -b10m - foo.gz.

One of the other things you could do that would be less resource
intensive if you have space for the whole file and one chunk is
to process the file from the end backward: start creating the
last chunk, then truncate the original big file, and proceed
backward like this.


--
St�phane
From: bb on 18 Dec 2009 11:21
On 2009-12-15 14:55, marc wrote:
> Hello,
>
> I gave a big zipped file (foo.gz) and I would want to split it into n
> small files, but in memory, because I haven't enough space on my FS.
> I tried with mkfifo, gunzip and split, but it splits first all the
> files on disk and it stops logically with
> "No space left on device" error.
> Is there another way ?
>
> Thanks.

move it to /tmp

/bb
From: Kaz Kylheku on 18 Dec 2009 14:18
On 2009-12-15, marc <marc.tessis(a)caramail.com> wrote:
> Hello,
>
> I gave a big zipped file (foo.gz) and I would want to split it into n
> small files, but in memory, because I haven't enough space on my FS.

Stick an inexpensive storage device into your USB port.
 | 
Pages: 1
Prev: Textfile with unwanted spaces in words
Next: compress all *.c files in a hierarchy?