Export
in [Mathematica]
|
Prev: difficulty in calculating first two terms of an
Next: discretized Laplacian or linear inverse problem with extremely simple
From: David Bailey on 10 Jun 2010 08:11 On 09/06/10 12:21, Clint wrote: > Hi Bill, > > Thank you for youre response. I just read this post and in the meantime I found this work around: > > read a file and skip so many records down.. (here i skip so many iterations...) > > strm = OpenRead[ToFileName[{NotebookDirectory[]}, "1laserGrid.txt"]]; > If[iter != 1, Skip[strm, Record, (iter - 1)*Nz]]; > EL = ReadList[strm, Expression]; > Close[strm]; > > append to a file > strm = OpenAppend[ToFileName[{NotebookDirectory[]}, "1laserGrid.txt"], > FormatType -> OutputForm]; > WriteString[strm, > "\n"<> StringDrop[ > StringDrop[ > StringReplace[ToString[ELNewNew, InputForm], "," -> "\n"], > 1], -1]]; > Close[strm]; > > Both your solution and this one seem to give me the same problem which I will describe below: > > The reason for using export and import is that I max out the 32 GB RAM on my PC. So to keep RAM down I use file I/O. > > While the "write" stream takes no time at all and doesn't depend on the current file size the read command gets bogged down as the file gets larger and larger... I naively thought that using "skip" in the read would prevent the processor from reading in the entire file which starts to take a very long time as the file sizes approch 100,000 KB's... > > This is making simulations virtually imposibble to run since they are taking 10 hours primarily because I keep looping through this read command everytime I propogate my PDE ...and each one is taking a minute or so... > > I'm at a loss on what to do here..... > > I have no choice but to use file I/O do to RAM limitations but i don't see a way around my problem there either :( > > One way I thought of was to maintain two data files.. "one where i store all my data" and the second one "where I just store the last 2 iterations of data since that is all I need to propogate my pde forward in time anyway".. However, I thought maybe I would not have to do that if I could use non-sequential data pulling from a file in Mathematica but I guess that isn't possible? > The problem is that Skip can't just skip N records, because it doesn't 'know' how long each record is without looking! The best answer is probably to perform binary I/O which will let you write records of a fixed number of bytes, and then use SetStreamPosition to move directly to the point of interest. Real's for example take 8 bytes (when written in binary). I am assuming you are working at machine precision. I am also not clear why you are using ReadList, because that will read the rest of your file into store - which is not what you want, I presume. What you want to do certainly sounds do-able! David Bailey www.dbaileyconsultancy.co.uk |