Prev: Replacing NaN Values...?
Next: SQL type of data operators needed in Matlab
From: Luna Moon on 11 Jun 2010 08:04
Hi all,

In my project, I have to access gigantic (GB size) data.

To handle those in Matlab,

I divided them into small chunks.

For now I do:

for i=1:TotalNumOfTrunks

fetch (downloading from Internet) that trunk of data;

process this trunk;

end;

-------------------------------------

The problem is that the "fetch" part takes a lot of time.

If I save the downloaded data trunk onto local disk, "trunk1.mat",
"trunk2.mat", etc...

it seems to me that Matlab's "save" "load" commands are slow too...
(only a bit faster than downloading from Internet)...

Too sloooooow...

What's my way out here?

Any smart solutions here?

Thanks a lot!
From: Jan Simon on 11 Jun 2010 12:08
Dear Luna Moon!

> In my project, I have to access gigantic (GB size) data.

Compressed or uncompressed?
Try SAVE with the -v6 option to avoid compression. Then the files are larger, but the processing may be much faster - depending on the data.

Good luck, Jan
 | 
Pages: 1
Prev: Replacing NaN Values...?
Next: SQL type of data operators needed in Matlab