Prev: "Vectorize" code with nested for-loops
Next: remove a specific frequency from emg signal
From: bogfrog on 9 Aug 2010 08:09
Hello,

I have a loop that always crashes with java memory errors after about 10 hours. It is a for-loop. I increased my max java heap to 756MB using the java.opts file, and this only gave me a couple hours more of runtime.

I can't post all my code here, but I would just like to know what kind of operations use the java heap?

I put some java.lang.Runtime.getRuntime.freeMemory at different points in my code to try and see what's going on. I don't really see a correspondence with any actual parts of my code. It seems random. Also, free memory decreases in a stepwise pattern, but then after a while suddenly decreases in huge bursts. It does this for a while, then finally it completely crashes.

I stored the value of java.lang.Runtime.getRuntime.freeMemory to file for each loop iteration until it crashed. Here is a plot of these values right up until the crash (the x-axis is loop iterations):

http://img809.imageshack.us/img809/2893/javafreememplot.jpg

What kind of things can cause this? I really don't even know what to look for.

I know you can't figure out what's wrong specifically in my case, since I can't post my code. But if anyone knows the kinds of things that use the java heap, that would be great.

I am not using any add-on toolboxes. I am using calllib() for .dll calls, but as far as I understand, that does not use the java heap. So what could it be?
From: Steven_Lord on 9 Aug 2010 12:47


"bogfrog" <aj00mcgraw(a)gmail.com> wrote in message
news:736646707.81966.1281370224836.JavaMail.root(a)gallium.mathforum.org...
> Hello,
>
> I have a loop that always crashes with java memory errors after about 10
> hours. It is a for-loop. I increased my max java heap to 756MB using the
> java.opts file, and this only gave me a couple hours more of runtime.
>
> I can't post all my code here, but I would just like to know what kind of
> operations use the java heap?

The easy answer to that question is operations that involve Java. [I'm not
being facetious; that question is just way too general for a specific
answer.]

The more difficult answer to that question would depend on what you're
doing. For instance, the types of operations you're performing that could
use the heap would be different if you were creating a movie using figures
than if you were interfacing with an instrument or reading a database. A
general description of what your code is intended to do may allow people to
offer some suggestions even if you can't provide the actual code.

> I put some java.lang.Runtime.getRuntime.freeMemory at different points in
> my code to try and see what's going on. I don't really see a
> correspondence with any actual parts of my code. It seems random. Also,
> free memory decreases in a stepwise pattern, but then after a while
> suddenly decreases in huge bursts. It does this for a while, then finally
> it completely crashes.
>
> I stored the value of java.lang.Runtime.getRuntime.freeMemory to file for
> each loop iteration until it crashed. Here is a plot of these values
> right up until the crash (the x-axis is loop iterations):
>
> http://img809.imageshack.us/img809/2893/javafreememplot.jpg

One potential debugging step would be to look around iteration 2000 (just
guessing) where there's a large drop in free memory. If that behavior is
consistent when you try running this several times, what's going on in the
iteration where that drop occurs that's not taking place in the earlier or
later iterations? Perhaps that will shed some light on what's consuming
memory. You should be able to check if that dip is consistent across the
first few thousand iterations in less than 10 hours.

> What kind of things can cause this? I really don't even know what to look
> for.
>
> I know you can't figure out what's wrong specifically in my case, since I
> can't post my code. But if anyone knows the kinds of things that use the
> java heap, that would be great.
>
> I am not using any add-on toolboxes. I am using calllib() for .dll calls,
> but as far as I understand, that does not use the java heap. So what
> could it be?

--
Steve Lord
slord(a)mathworks.com
comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

From: bogfrog on 9 Aug 2010 12:12
Thanks for the pointers. I ended up greatly simplifying my program, and a similar memory leak is _still_ happening. I've only run it for a couple hours, but here is a plot of the value of java.lang.Runtime.getRuntime.freeMemory that I have so far:

http://img836.imageshack.us/img836/7685/memoryhistory.jpg


The interesting thing is, this is ALL I'm doing:

----
N = 100000;

[dir, name, ext, versn] = fileparts(mfilename('fullpath'));

fid = fopen([dir '\dummy.txt'], 'w');
fid2 = fopen([dir '\dummy2.txt'], 'w');
fid3 = fopen([dir '\MemoryHistory.txt'],'w');

tic;

a = cell(0,0);

for i = 1:N
mK = java.lang.Runtime.getRuntime.freeMemory;
pause(0.2);
fprintf(fid3,'%d \n', mK);
end
----


That's it! Just a for loop with a pause and an fprintf!

Yes, I restarted Matlab, and ran nothing else before running this.

So what could possibly cause this memory leak? I can't understand this.

Thanks for any insight you can give!
From: Steven_Lord on 9 Aug 2010 17:07


"bogfrog" <aj00mcgraw(a)gmail.com> wrote in message
news:659568111.83319.1281384755594.JavaMail.root(a)gallium.mathforum.org...
> Thanks for the pointers. I ended up greatly simplifying my program, and a
> similar memory leak is _still_ happening. I've only run it for a couple
> hours, but here is a plot of the value of
> java.lang.Runtime.getRuntime.freeMemory that I have so far:
>
> http://img836.imageshack.us/img836/7685/memoryhistory.jpg
>
>
> The interesting thing is, this is ALL I'm doing:
>
> ----
> N = 100000;
>
> [dir, name, ext, versn] = fileparts(mfilename('fullpath'));
>
> fid = fopen([dir '\dummy.txt'], 'w');
> fid2 = fopen([dir '\dummy2.txt'], 'w');
> fid3 = fopen([dir '\MemoryHistory.txt'],'w');
>
> tic;
>
> a = cell(0,0);
>
> for i = 1:N
> mK = java.lang.Runtime.getRuntime.freeMemory;
> pause(0.2);
> fprintf(fid3,'%d \n', mK);
> end
> ----
>
>
> That's it! Just a for loop with a pause and an fprintf!
>
> Yes, I restarted Matlab, and ran nothing else before running this.
>
> So what could possibly cause this memory leak? I can't understand this.

Are you running MATLAB with the Desktop?

If so, what components of the Desktop are open when you run this? I'm
particularly curious about the Workspace Browser (in case it's updating
whenever the value of mK changes) and the Current Folder (in case it's
updating whenever the Last Modified date on MemoryHistory.txt changes)
windows. If those are open, try closing them and run this code again.

--
Steve Lord
slord(a)mathworks.com
comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
To contact Technical Support use the Contact Us link on
http://www.mathworks.com

From: us on 9 Aug 2010 17:38
bogfrog <aj00mcgraw(a)gmail.com> wrote in message <659568111.83319.1281384755594.JavaMail.root(a)gallium.mathforum.org>...
> Thanks for the pointers. I ended up greatly simplifying my program, and a similar memory leak is _still_ happening. I've only run it for a couple hours, but here is a plot of the value of java.lang.Runtime.getRuntime.freeMemory that I have so far:
>
> http://img836.imageshack.us/img836/7685/memoryhistory.jpg
>
>
> The interesting thing is, this is ALL I'm doing:
>
> ----
> N = 100000;
>
> [dir, name, ext, versn] = fileparts(mfilename('fullpath'));
>
> fid = fopen([dir '\dummy.txt'], 'w');
> fid2 = fopen([dir '\dummy2.txt'], 'w');
> fid3 = fopen([dir '\MemoryHistory.txt'],'w');
>
> tic;
>
> a = cell(0,0);
>
> for i = 1:N
> mK = java.lang.Runtime.getRuntime.freeMemory;
> pause(0.2);
> fprintf(fid3,'%d \n', mK);
> end
> ----
>
>
> That's it! Just a for loop with a pause and an fprintf!
>
> Yes, I restarted Matlab, and ran nothing else before running this.
>
> So what could possibly cause this memory leak? I can't understand this.
>
> Thanks for any insight you can give!

here: ic2/2*2.6mhz/4gb/winxp.sp3.32/r2010a
your code works fine...
note: we also get the same type of curve (with this simplification)

n=100000;
mk=zeros(n,1);
for i=1:n
mk(i,1)=java.lang.Runtime.getRuntime.freeMemory;
end
stem(mk);

us
 |  Next  |  Last
Pages: 1 2
Prev: "Vectorize" code with nested for-loops
Next: remove a specific frequency from emg signal