Prev: error in my program
Next: Batch of experiments
From: Henrik on 31 May 2010 03:14
Hi,

I am using a matlab script for data acquisition from an instrument.
I open a .txt file

w_file = fopen(['C:\log\',fileName],'a');

and save the data to the file using a while-loop and

fprintf(w_file,my_data);

This works fine, and I can see the .txt file growing with time. When I want to end the script, I have a button to press that issues the break; command and gets me out of the loop. The

fclose(w_file);

command is then executed.

This works perfectly when I only acquire for short time (file sizes < 1MB). When I keep my acquisition running for several days (file sizes > 15MB) the program hangs for several minutes while executing the fclose() command.

Any suggestions as to why this happens? Is there a better way to do this?

Regards
Henrik
From: Rune Allnor on 31 May 2010 05:21
On 31 Mai, 09:14, "Henrik " <heol9...(a)student.uu.se> wrote:
> Hi,
>
> I am using a matlab script for data acquisition from an instrument.
> I open a .txt file
>
> w_file = fopen(['C:\log\',fileName],'a');
>
> and save the data to the file using a while-loop and
>
> fprintf(w_file,my_data);
>
> This works fine, and I can see the .txt file growing with time. When I want to end the script, I have a button to press that issues the break; command and gets me out of the loop. The
>
> fclose(w_file);
>
> command is then executed.
>
> This works perfectly when I only acquire for short time (file sizes < 1MB).  When I keep my acquisition running for several days (file sizes > 15MB) the program hangs for several minutes while executing the fclose() command.
>
> Any suggestions as to why this happens? Is there a better way to do this?

First of all, accessing files is slow. It has a lot to do with
slow response times in hard disks, low capacity across data buses
or networks and so on.

If you save data on text format you have additional delays associated
with converting the numbers from binary to text format. These kinds
of
delays might accumulate, so the baseline delay would be something
like
1 minute per 100 MBytes of data.

Any delays beyond that start to become suspicious.

File IO is far too complicated to start messing with details, so
I suggest you break the logs down in batches. The obvious first
attempt is to write many small log files instead of one big file.
You talk about your system running for days. If so, I'd suggest you
start a new log file for, say, eevery 3 or 6 hrs.

Rune
From: Walter Roberson on 31 May 2010 10:48
Henrik wrote:

> I am using a matlab script for data acquisition from an instrument.
> I open a .txt file
>
> w_file = fopen(['C:\log\',fileName],'a');

I see from the file name that you are using Windows. On Windows, in
order to work with text files, you need to include 't' in the access
permissions code, fopen(Filename, 'at'); the default is to treat it as a
binary file. This might perhaps have some bearing on the problem, but it
seems more likely to be an independent issue.

>
> and save the data to the file using a while-loop and
>
> fprintf(w_file,my_data);

The default format for fprintf() is "%s\n". I will assume that you
showed the above statement as an abbreviation for your actual statement.


> This works fine, and I can see the .txt file growing with time. When I
> want to end the script, I have a button to press that issues the break;
> command and gets me out of the loop. The
> fclose(w_file);

> command is then executed.

> This works perfectly when I only acquire for short time (file sizes <
> 1MB). When I keep my acquisition running for several days (file sizes >
> 15MB) the program hangs for several minutes while executing the fclose()
> command.

That is a bit odd. Does your file increase in size more than one cluster
between the time you interrupt and the time the close finishes? If so
then it could be a buffering issue: with modern systems it would not be
uncommon for several tens of megabytes to be buffered in memory.
However, modern hard disks can easily sustain writes of more than 50 MB
per second,
http://www.tomshardware.com/reviews/seagate-momentus-xt-hybrid-hard-drive-ssd,2638-6.html
so 15 MB should be peanuts. Conversion of binary to text would take
place at computer speeds rather than hard-disk speeds and should be much
much faster than 15 MB/s.

On a test system, I just wrote 18 MB of formatted data in 2 seconds:

>> foo = rand(2000,1000);
>> size(foo)
ans =
2000 1000
>> whos foo
Name Size Bytes Class Attributes

foo 2000x1000 16000000 double

>> f= fopen('/tmp/junk.dat','wt')
f =
3
>> tic;fprintf(f,'%g\n',foo);fclose(f);toc
Elapsed time is 2.047559 seconds.
>> !ls -l /tmp/junk.dat
-rw-rw-r-- 1 roberson roberson 18000005 2010-05-31 09:42 /tmp/junk.dat

I suggest you try a similar test and so establish a baseline for file
I/O on your system. You could also benchmark your system by timing the
copy of an existing file.

What kind of disk interface and hard disk are you using? Or more
correctly, how fast are they?
From: Henrik on 3 Jun 2010 05:37
Sorry for a bit late response to this.

I will try the test you propose and get back with the results.

When the script is running, I do see a constant increase in the size of the file that is being written. I can also copy (or open in notepad) that file while the macro is running, and all the data will be there.

For some reason, when the macro has been running for a very long time, matlab becomes unresponsive when the macro is stopped.

I will now try a solution where I close (fclose) and reopen the file (with append) every 10,000nd iteration. We will see if this solves the problem
From: Henrik on 3 Jun 2010 05:47
I just tried this:

foo = rand(2000,1000);
size(foo)
whos foo
file = fopen('c:\speedtest.txt','at');
for i=1:10
tic
fprintf(file,'%g\n',foo);
toc
end
fclose(file);

with this result:

Elapsed time is 2.481803 seconds.
Elapsed time is 3.411456 seconds.
Elapsed time is 4.328087 seconds.
Elapsed time is 3.772671 seconds.
Elapsed time is 3.816619 seconds.
Elapsed time is 2.211746 seconds.
Elapsed time is 2.214240 seconds.
Elapsed time is 2.247375 seconds.
Elapsed time is 2.254755 seconds.
Elapsed time is 2.222354 seconds.


total filesize almost 200MB.

It is a bit strange indeed. I am using the macro to measure a potential from an Agilent multimeter connected through GPIB. Perhaps it is something completely different creating the lag
 |  Next  |  Last
Pages: 1 2
Prev: error in my program
Next: Batch of experiments