running code for 22 hours now...
in [Matlab]
|
From: matlab_learner on 3 May 2010 22:39 so i finally programmed and my code is working CORRECT. the instructor warned it could run for days depending on your scheme. Mine has run for at least 21 hrs now and I am fearing for my Mac Intel processor now. Looking at the rate of the decrease of my residuals (what i am solving for) I am just halfway there. What are the dangers of running the processor non stop (i have kept it at optimum temp so the fan doesn't come on very freq due to heat) and how can I allocate more memory to it to finish the code faster? I am basically a Windows guy. Thanks.
From: John D'Errico on 3 May 2010 23:52 matlab_learner <cibeji(a)gmail.com> wrote in message <19fb4769-ce75-46e5-ba59-052ad45a9b5d(a)34g2000prs.googlegroups.com>... > so i finally programmed and my code is working CORRECT. the instructor > warned it could run for days depending on your scheme. Mine has run > for at least 21 hrs now and I am fearing for my Mac Intel processor > now. Looking at the rate of the decrease of my residuals (what i am > solving for) I am just halfway there. What are the dangers of running > the processor non stop (i have kept it at optimum temp so the fan > doesn't come on very freq due to heat) and how can I allocate more > memory to it to finish the code faster? I am basically a Windows guy. > Thanks. You can't allocate more memory to a process. You can write your code to be more efficient. Learn to use the profile tool. You won't melt your CPU either. John
From: Walter Roberson on 4 May 2010 00:15 John D'Errico wrote: > matlab_learner <cibeji(a)gmail.com> wrote in message > <19fb4769-ce75-46e5-ba59-052ad45a9b5d(a)34g2000prs.googlegroups.com>... >> What are the dangers of running >> the processor non stop (i have kept it at optimum temp so the fan >> doesn't come on very freq due to heat) > You won't melt your CPU either. My CPU on my Windows machine didn't literally _melt_, but it did fail from heat. I wasn't even overclocking it. Investigation showed that a fair bit of dust inside, but also showed that the last time I had cleaned and put the cover back on, I had failed to extend the air vent cone all the way to the CPU, so the CPU heat was circulating around the chassis more than the chassis was designed for. A combination of circumstances. Anyhow, on Mac's, it is important that you give enough clearance around the device and not put items on top of the device, especially for the fanless Mac Mini's.
From: Walter Roberson on 4 May 2010 02:04 matlab_learner wrote: > Mac Intel processor > how can I allocate more > memory to it to finish the code faster? I am basically a Windows guy. John indicated that you cannot allocate more memory to a process (in MacOS). I'm not sure that is _completely_ true, but the circumstances that I can think of under which it would potentially be falsifiable are quite unlikely to apply to you. Unless you have a wiki:Bastard_Operator_from_Hell running your system, you should assume that the system is automatically managing physical memory to balance out the competing demands of different programs. You should assume that if the system detects that a block of memory from a user-level program has not been used "recently", that the system may write that a copy of that block of memory to disk and then release the physical memory to be used by some other more active program. This process of writing "old" memory to disk and reading it back from disk when it is needed, is called "swapping", and if you have much swapping going on then it can slow your system enormously. Thus, what you need to address first is the competing demands from other programs. First, you have to find out what those demands are. If you open the folder for the main hard disk, and go from there to the Applications folder, and open the Utilities folder inside the Applications folder, then you will find a tool named Activity Monitor . Double click on that to start it up. The window that it brings up by default will have to do with CPU activity. By default that window is sorted by Process Name. Click on the column heading 'CPU'; if you need to, click on it again so that the triangle points downward: that will sort in descending order by percentage of the CPU. If you see there a user-level process that is getting more CPU time than Matlab is getting, then unless Matlab is showing up with an unexpectedly low CPU time (e.g., 1%), you probably would prefer to terminate that user level process if it isn't being used. Unfortunately, as you are not very familiar with Mac OS X, you will probably not know which processes can be safely killed -- but you can be reasonably sure that you can terminate iTunes and iTunes Helper . Now, click on the column for Virtual Memory, again clicking until the triangle points downward. Virtual Memory is the amount of physical memory that your process would use if that memory was available, but it also includes in that total a number of shared libraries that are used by many different programs at the same time, so do not worry so much if you see quite a few programs using 600 Mb or so. However, if you have a process in which the Virtual Memory is larger than your physical memory, then some amount of swapping to disk will be taking place. Swapping of physical memory to disk is something you would prefer to avoid!! Do not be surprised if you find that Safari or Firefox or Thunderbird or Mail are taking up large amounts of virtual memory. For as long as it takes to finish your calculations, you should shut down your browsers and mail readers when-ever practical: that will probably free up a significant amount of memory that those programs will no longer be "stealing" out from underneath Matlab. If you now click on the heading for Real Memory, you will see the amount of physical memory being used by each program. To the degree practical given your knowledge of what various programs do, you would prefer to terminate the programs using the largest amounts of physical memory, so that that physical memory can be turned over to Matlab. I mentioned "swapping to disk". The situation is much the same as on Windows: if a block of physical memory has not been used "recently" and a program needs some physical memory, then the "inactive" block of memory might be written to hard disk, and then the physical memory made available for other uses; the operating system will read that block of memory back from the disk when the program next asks to access something in that block. If you have a bunch of processes that are pretty much idle, swapping to disk can be very effective -- but if you have several programs competing for memory, then they can end up taking memory from each other, only to have some of their own taken away in turn for use by one of the other processes. Something like that will degrade performance. If you have a program which is actively asking to use more physical memory than exists on your machine (and which is not reserved for the operating system), then you can run into very very nasty situations in which a program is continually stealing its own memory. When that happens, the program can end up spending nearly all of its time waiting for the hard disk. Programs that are that large can end up using only about 1% or 2% of the available CPU time, because the hard drive is roughly 50 to 100 times slower than physical memory. If your Matlab process is using enough memory to need to swap itself to disk, then chances are high that your program can be written more efficiently. John referenced the Matlab profiler, which is a good tool to use to figure out where your program is spending most of its time. Probably the most common Matlab efficiency mistake is failing to preallocate large arrays and having them grow as they are used: that can require copying large amounts of memory around to a place that has enough memory to hold the larger block, only to have to copy it all around again the next time through the loop when the block gets bigger again. Another common Matlab mistake is to primarily use row vectors to represent data that is used as a group. In Matlab, data is stored by columns rather than by rows. B(I,J) might be stored quite a distance away from B(I,J+1), but B(I+1,J) will be stored right beside B(I,J). If your routine asks Matlab to work with a row of a large matrix, then that might involve quite a few different "pages" of physical memory, and thus can result in Matlab "thrashing", needing to access the same set of blocks over and over again but not being able to hold them all in physical memory at the same time. When you access arrays by columns, the probability of thrashing is considerably reduced, likely only needing to access three or four different "pages" of physical memory simultaneously. It is possible to detect and monitor paging activity, but this message is already long. You need to start up a terminal window (program Terminal in the Utilities folder of the Applications folder of the main disk, and type the command top inside the terminal window.) The Mac OS Activity Monitor allows you to terminate processes (like the Windows monitor that you get by pressing control-alt-delete), but unlike the Windows monitor, the Mac OS Activity Monitor does not allow you to change the priority of running programs. It is possible to change the priority of programs in Mac OS, but I'm not going to detail that either: you need to use the 'renice' command from a terminal window to change a program's priority.
From: John D'Errico on 4 May 2010 06:05
Walter Roberson <roberson(a)hushmail.com> wrote in message <pNODn.105596$u62.105086(a)newsfe10.iad>... > matlab_learner wrote: > > Mac Intel processor > > > how can I allocate more > > memory to it to finish the code faster? I am basically a Windows guy. > > John indicated that you cannot allocate more memory to a process (in > MacOS). I'm not sure that is _completely_ true, but the circumstances > that I can think of under which it would potentially be falsifiable are > quite unlikely to apply to you. > > Unless you have a wiki:Bastard_Operator_from_Hell running your system, > you should assume that the system is automatically managing physical > memory to balance out the competing demands of different programs. You > should assume that if the system detects that a block of memory from a > user-level program has not been used "recently", that the system may > write that a copy of that block of memory to disk and then release the > physical memory to be used by some other more active program. This > process of writing "old" memory to disk and reading it back from disk > when it is needed, is called "swapping", and if you have much swapping > going on then it can slow your system enormously. > > Thus, what you need to address first is the competing demands from other > programs. First, you have to find out what those demands are. > > If you open the folder for the main hard disk, and go from there to the > Applications folder, and open the Utilities folder inside the > Applications folder, then you will find a tool named Activity Monitor . > Double click on that to start it up. > > The window that it brings up by default will have to do with CPU > activity. By default that window is sorted by Process Name. Click on the > column heading 'CPU'; if you need to, click on it again so that the > triangle points downward: that will sort in descending order by > percentage of the CPU. If you see there a user-level process that is > getting more CPU time than Matlab is getting, then unless Matlab is > showing up with an unexpectedly low CPU time (e.g., 1%), you probably > would prefer to terminate that user level process if it isn't being > used. Unfortunately, as you are not very familiar with Mac OS X, you > will probably not know which processes can be safely killed -- but you > can be reasonably sure that you can terminate iTunes and iTunes Helper . If you are not absolutely sure that you know what a process does and that killing it will not cause a problem, then it is not recommended to kill it off. > Now, click on the column for Virtual Memory, again clicking until the > triangle points downward. Virtual Memory is the amount of physical > memory that your process would use if that memory was available, but it > also includes in that total a number of shared libraries that are used > by many different programs at the same time, so do not worry so much if > you see quite a few programs using 600 Mb or so. However, if you have a > process in which the Virtual Memory is larger than your physical memory, > then some amount of swapping to disk will be taking place. Swapping of > physical memory to disk is something you would prefer to avoid!! > Walter is describing an operation on a specific (and apparently older) version of Mac OSX by the way. He really should indicate which version he is describing. I will point out that the most recent version of OSX would be somewhat different. > Do not be surprised if you find that Safari or Firefox or Thunderbird or > Mail are taking up large amounts of virtual memory. For as long as it > takes to finish your calculations, you should shut down your browsers > and mail readers when-ever practical: that will probably free up a > significant amount of memory that those programs will no longer be > "stealing" out from underneath Matlab. This is a good recommendation in general. One thing that you do not wish to do is to move between applications if your matlab session is using up a lot of memory. This can cause problems, depending upon the number of processors your system has available. It is also true that having a web browser open on the side can use up cpu cycles, even is you are doing nothing in it while MATLAB is running. Regardless, I will argue that the first and foremost thing you should do is NOT to play with system parameters. Do not go off willy nilly killing processes. Do not try to allocate more memory on a machine that is doing a good job of it without your help. Instead, learn to use the profile tool. It will help you to program in MATLAB more efficiently. Learn to find the bottlenecks in your code. This will gain you FAR more than will trying to tweak your system. John |