Export
in [Mathematica]
|
Prev: difficulty in calculating first two terms of an
Next: discretized Laplacian or linear inverse problem with extremely simple
From: Clint on 3 Jun 2010 05:47 Hi, How can I accomplish the following: mylist = {1,2,3,4}; I would like to "export" my list to a data file so that inside the data file it would look like: 1,2,3,4 on the first line... I then need to be able to read this back in using import.. I also need the ability to "Append" to this data file another line of data (ie mylist2 = {2,4,6,8}) I would like to "write" this second line of data to the same file so that the new file would look like: 1,2,3,4 on first line... 2,4,6,8 on second line... etc.. Then also how can I access a specific line of data? Thanks for the guidance. -Clint
From: Bill Rowe on 4 Jun 2010 08:02 On 6/3/10 at 5:38 AM, clint.zeringue(a)kirtland.af.mil (Clint) wrote: >How can I accomplish the following: >mylist = {1,2,3,4}; >I would like to "export" my list to a data file so that inside the >data file it would look like: >1,2,3,4 on the first line... This is easily done using Export. That is Export[filename, {{1,2,3,4}},"CSV"] will create the desired file. Note, the double brackets are needed here. Without them, Export will create 4 rows. >I then need to be able to read this back in using import.. This is done by doing Import[filename,"CSV"] >I also need the ability to "Append" to this data file another line >of data (ie mylist2 = {2,4,6,8}) >I would like to "write" this second line of data to the same file so >that the new file would look like: >1,2,3,4 on first line... >2,4,6,8 on second line... etc.. I would do this as follows: read in the contents of the existing file using Import[filename, "CSV"] to a conveniently variable. append the new data to that variable export entire data set back out to the file using Export[filename, variablename, "CSV"] If there currently is an option to append new data to "CSV" files I am not aware of it. It would be possible to create a means to append to "CSV" files if that is needed due to say a very large file size it could be done as: str=OpenAppend[filename]; WriteString[str, "\n",StringTake[ToString@{2,4,6,8}],{2,-2}]; Close[str]; A couple of things here. I've assumed an existing file (filename) created using Export as above. The last line of a "CVS" file created by Export won't have a newline and WriteString does not write a new line. That is why the newline is written first. WriteString will convert the data to a string before writing it out. But with a list that means an set of curly braces surrounding the output string. That is why the StringTake[ToString@ part is included. Finally, this will be a comma separated value format but it will also have spaces between the values. This won't matter to Import. But it might for some other purpose. >Then also how can I access a specific line of data? This last can be done as Import[filename, {"Data",n}] where n is the line number you want to access
From: Clint on 9 Jun 2010 07:21 Hi Bill, Thank you for youre response. I just read this post and in the meantime I found this work around: read a file and skip so many records down.. (here i skip so many iterations...) strm = OpenRead[ToFileName[{NotebookDirectory[]}, "1laserGrid.txt"]]; If[iter != 1, Skip[strm, Record, (iter - 1)*Nz]]; EL = ReadList[strm, Expression]; Close[strm]; append to a file strm = OpenAppend[ToFileName[{NotebookDirectory[]}, "1laserGrid.txt"], FormatType -> OutputForm]; WriteString[strm, "\n" <> StringDrop[ StringDrop[ StringReplace[ToString[ELNewNew, InputForm], "," -> "\n"], 1], -1]]; Close[strm]; Both your solution and this one seem to give me the same problem which I will describe below: The reason for using export and import is that I max out the 32 GB RAM on my PC. So to keep RAM down I use file I/O. While the "write" stream takes no time at all and doesn't depend on the current file size the read command gets bogged down as the file gets larger and larger... I naively thought that using "skip" in the read would prevent the processor from reading in the entire file which starts to take a very long time as the file sizes approch 100,000 KB's... This is making simulations virtually imposibble to run since they are taking 10 hours primarily because I keep looping through this read command everytime I propogate my PDE ...and each one is taking a minute or so... I'm at a loss on what to do here..... I have no choice but to use file I/O do to RAM limitations but i don't see a way around my problem there either :( One way I thought of was to maintain two data files.. "one where i store all my data" and the second one "where I just store the last 2 iterations of data since that is all I need to propogate my pde forward in time anyway".. However, I thought maybe I would not have to do that if I could use non-sequential data pulling from a file in Mathematica but I guess that isn't possible?
From: Mitch Stonehocker on 10 Jun 2010 08:08 I make the following suggestion not knowing exactly what you're after and how you need/want to process. When you read data Mathematica consumes memory to retain the data. When you write data Mathematica doesn't release any memory therefore when you have a situation where you are looping through a process like - read data, manipulate, add to the original data, write new data - Mathematica will be consuming ever more amounts of memory each time data is read. You may be able control exactly how memory grows and avoid reading and writing files using a technique exemplified here: In[1]= data[0] = a; data[1] = b; ?data Out[2]= Global`data data[0]=a data[1]=b In[3]= data[0, 0] = c; data[0, 1] = d; ?data Out[4]= Global`data data[0]=a data[1]=b data[0,0]=c data[0,1]=d I hope that helps. Cheers, Mitch -----Original Message----- From: Clint [mailto:clint.zeringue(a)kirtland.af.mil] Sent: Wednesday, June 09, 2010 7:22 AM Subject: Re: Export Hi Bill, Thank you for youre response. I just read this post and in the meantime I found this work around: read a file and skip so many records down.. (here i skip so many iterations...) strm = OpenRead[ToFileName[{NotebookDirectory[]}, "1laserGrid.txt"]]; If[iter != 1, Skip[strm, Record, (iter - 1)*Nz]]; EL = ReadList[strm, Expression]; Close[strm]; append to a file strm = OpenAppend[ToFileName[{NotebookDirectory[]}, "1laserGrid.txt"], FormatType -> OutputForm]; WriteString[strm, "\n" <> StringDrop[ StringDrop[ StringReplace[ToString[ELNewNew, InputForm], "," -> "\n"], 1], -1]]; Close[strm]; Both your solution and this one seem to give me the same problem which I will describe below: The reason for using export and import is that I max out the 32 GB RAM on my PC. So to keep RAM down I use file I/O. While the "write" stream takes no time at all and doesn't depend on the current file size the read command gets bogged down as the file gets larger and larger... I naively thought that using "skip" in the read would prevent the processor from reading in the entire file which starts to take a very long time as the file sizes approch 100,000 KB's... This is making simulations virtually imposibble to run since they are taking 10 hours primarily because I keep looping through this read command everytime I propogate my PDE ...and each one is taking a minute or so... I'm at a loss on what to do here..... I have no choice but to use file I/O do to RAM limitations but i don't see a way around my problem there either :( One way I thought of was to maintain two data files.. "one where i store all my data" and the second one "where I just store the last 2 iterations of data since that is all I need to propogate my pde forward in time anyway".. However, I thought maybe I would not have to do that if I could use non-sequential data pulling from a file in Mathematica but I guess that isn't possible?
From: Albert Retey on 10 Jun 2010 08:08
Hi, > > Thank you for youre response. I just read this post and in the > meantime I found this work around: > > read a file and skip so many records down.. (here i skip so many > iterations...) > > strm = OpenRead[ToFileName[{NotebookDirectory[]}, > "1laserGrid.txt"]]; If[iter != 1, Skip[strm, Record, (iter - > 1)*Nz]]; EL = ReadList[strm, Expression]; Close[strm]; > > append to a file strm = OpenAppend[ToFileName[{NotebookDirectory[]}, > "1laserGrid.txt"], FormatType -> OutputForm]; WriteString[strm, "\n" > <> StringDrop[ StringDrop[ StringReplace[ToString[ELNewNew, > InputForm], "," -> "\n"], 1], -1]]; Close[strm]; > > Both your solution and this one seem to give me the same problem > which I will describe below: > > The reason for using export and import is that I max out the 32 GB > RAM on my PC. So to keep RAM down I use file I/O. > > While the "write" stream takes no time at all and doesn't depend on > the current file size the read command gets bogged down as the file > gets larger and larger... I naively thought that using "skip" in the > read would prevent the processor from reading in the entire file > which starts to take a very long time as the file sizes approch > 100,000 KB's... I think that your expectation was wrong, since when you want to use something like Skip, then of course the program needs to read the file to analyze where Records start and end. If you want to avoid a complete reread, you will need to explicitly tell the program where to read and write by giving it a position in the file, as you would have to in other programming languages. There are StreamPosition and SetStreamPosition for that purpose and I think that these should make it possible to read and write to a large file without too much (or any) overhead. One thing is that now you have to remember the position of where in the file your records are. Of course doing so has a certain risk of destroying the data in the file, if you write to wrong positions, so you'd better test that code well before using it in production. > This is making simulations virtually imposibble to run since they are > taking 10 hours primarily because I keep looping through this read > command everytime I propogate my PDE ...and each one is taking a > minute or so... > > I'm at a loss on what to do here..... > > I have no choice but to use file I/O do to RAM limitations but i > don't see a way around my problem there either :( > > One way I thought of was to maintain two data files.. "one where i > store all my data" and the second one "where I just store the last 2 > iterations of data since that is all I need to propogate my pde > forward in time anyway".. However, I thought maybe I would not have > to do that if I could use non-sequential data pulling from a file in > Mathematica but I guess that isn't possible? If I understand your problem correctly, you only will need to remember the last two write positions, so that shouldn't be a large burden... Basically I think your idea with 2 files isn't too bad, it will probably be simpler and just as fast as the 1 file version with StreamPositions. There is a third option to handle things like this: you could store the data in a database and read and write to the database using DatabaseLink. I have done that with good success, but of course it needs a little knowledge about databases and some experimentation with the DatabaseLink`-package. hth, albert |