Allocatable versus automatic arrays
in [Fortran]
|
From: helvio on 2 Jun 2010 00:14 Hi all, I have a huge F90 code composed by several modules, with several module procedures each, and a main program. No external procedures are used. I've come across this situation: some modules have very large arrays declared in the global scope (with sizes of order ~50000 each), but some of these arrays are only used conditionally. They might be used elsewhere, but there's also the possibility that they might not. The situation is the following, schematically: MODULE mod integer, parameter :: N = 50000 real, allocatable, save :: U(:) real :: V(N) CONTAINS subroutine using_UV allocate(U(N)) !exec-statements using U and V end subroutine using_UV subroutine using_V !exec-statements using V end subroutine using_V subroutine kill_U deallocate(U) end subroutine kill_U END MODULE mod PROGRAM foo use mod, only: U, V, using_UV, using_V, kill_U logical :: L !(...) call using_V !calculate L: L = <something> if (L) then call using_UV !exec-statements using U and V call kill_U end if !(...) END PROGRAM foo The array U will only be used if L is .TRUE., but V is necessary to exist. Naively, I would think that when U is declared explicitly as an automatic array ( REAL :: U(N) ), it allocates a large chunk of memory that isn't used, hence that would make the program more inefficient than when U is given the attribute ALLOCATABLE. But I don't know if this will make any difference in speed or memory management after the program is compiled (especially if the arrays get even larger). Therefore my naive question is: is the attribute ALLOCATABLE for U wise in this case, or is it irrelevant? Best, --helvio
From: Richard Maine on 2 Jun 2010 00:52 helvio <helvio.vairinhos(a)googlemail.com> wrote: > some modules have very large > arrays declared in the global scope (with sizes of order ~50000 each), > but some of these arrays are only used conditionally.... > Naively, I would think that when U is declared explicitly as an > automatic array ( REAL :: U(N) ), it allocates a large chunk of memory > that isn't used, hence that would make the program more inefficient > than when U is given the attribute ALLOCATABLE. But I don't know if > this will make any difference in speed or memory management after the > program is compiled (especially if the arrays get even larger). > > Therefore my naive question is: is the attribute ALLOCATABLE for U > wise in this case, or is it irrelevant? I would say that your analysis seems accurate... except for one very important detail. There was a day when things were different, but today, unless your environment is extremely unusual, a size of 50,000 single precision reals (so 200,000 bytes with most compilers) does not count as "very large". Consider that your typical consumer computer today probably has around 10,000 times that much physical memory, and a virtual address space of also about 10,000 times that much. Even a old machine that isn't big enough to install a current OS version probably has physical memory of 1,000 times that. If you really had arrays large enough to have significant impact on memory availability, then I'd agree with your concern. But you are multiple orders of magnitude away from that area. On any reasonable current system, allocating 200kB of virtual memory that you don't reference is not likely to make a difference that you can measure. Neither is 10 times that much, or 100. At 1,000 times that much, you might have grounds to worry. So do it whichever way is most convenient for other reasons. I'd not worry about the performance difference. Do note that I am talking only about the difference between allocatable versus automatic. There can be more cost to declaring static arrays, even of relatively modest size (because they often end up bloating the size of the executable program file and need to be loaded from it into memory). -- Richard Maine | Good judgment comes from experience; email: last name at domain . net | experience comes from bad judgment. domain: summertriangle | -- Mark Twain
From: glen herrmannsfeldt on 2 Jun 2010 01:30 Richard Maine <nospam(a)see.signature> wrote: > helvio <helvio.vairinhos(a)googlemail.com> wrote: >> some modules have very large >> arrays declared in the global scope (with sizes of order ~50000 each), >> but some of these arrays are only used conditionally.... >> Naively, I would think that when U is declared explicitly as an >> automatic array ( REAL :: U(N) ), it allocates a large chunk of memory >> that isn't used, hence that would make the program more inefficient >> than when U is given the attribute ALLOCATABLE. But I don't know if >> this will make any difference in speed or memory management after the >> program is compiled (especially if the arrays get even larger). If it is in a module like that, I believe it will be static if not allocatable. On many machines, access to static arrays is sligthly more efficient than automatic, and automatic might be a little more efficient that allocatable. The differences are pretty small, and pipelining might even remove them. You probably shouldn't worry about them unless you actually have timing results that show that they are specifically a problem. Automatic arrays are often on the stack, which requires that enough (virtual memory for) stack be available. Watch for systems with stack limits. >> Therefore my naive question is: is the attribute ALLOCATABLE for U >> wise in this case, or is it irrelevant? > I would say that your analysis seems accurate... except for one very > important detail. There was a day when things were different, but today, > unless your environment is extremely unusual, a size of 50,000 single > precision reals (so 200,000 bytes with most compilers) does not count as > "very large". Consider that your typical consumer computer today > probably has around 10,000 times that much physical memory, and a > virtual address space of also about 10,000 times that much. Even a old > machine that isn't big enough to install a current OS version probably > has physical memory of 1,000 times that. Well, also that with a virtual memory system it should not even be in memory if not accessed. (With the usual 4K page size.) > If you really had arrays large enough to have significant impact on > memory availability, then I'd agree with your concern. But you are > multiple orders of magnitude away from that area. On any reasonable > current system, allocating 200kB of virtual memory that you don't > reference is not likely to make a difference that you can measure. > Neither is 10 times that much, or 100. At 1,000 times that much, you > might have grounds to worry. There are still some strange systems around. I had problems once on an Alpha system with 4GB memory trying to allocate 100K of static memory. At least with some systems and compilers there is a small limit for static allocation. > So do it whichever way is most convenient for other reasons. I'd not > worry about the performance difference. > Do note that I am talking only about the difference between allocatable > versus automatic. There can be more cost to declaring static arrays, > even of relatively modest size (because they often end up bloating the > size of the executable program file and need to be loaded from it into > memory). That should only happen for initialized static data. C requires that all static data be initialized (to zero if not otherwise specified), and some Fortran compilers also do that. It seems that many can efficiently initialize static zeros, but other than zeros are written to the executable file. (Even very large arrays of the same value.) -- glen
From: Richard Maine on 2 Jun 2010 01:49 glen herrmannsfeldt <gah(a)ugcs.caltech.edu> wrote: > Richard Maine <nospam(a)see.signature> wrote: > > helvio <helvio.vairinhos(a)googlemail.com> wrote: > > >> some modules have very large > >> arrays declared in the global scope (with sizes of order ~50000 each), > >> but some of these arrays are only used conditionally.... .... > If it is in a module like that, I believe it will be static if > not allocatable. Oops. I missed that because I didn't look at the code closely enough. I just took the OP's word for it that he was talking about automatic arrays, but that was inaccurate. An array dimensioned N is not automatic when N is a parameter, as in the example code. In a non-module scope, it might possibly be implemented in a way that looks like an automatic, but it is not automatic as defined by the Fortran standard. In a module scope, as you imply, you can't have an array dimensioned N if N isn't a parameter. > There are still some strange systems around. I had problems > once on an Alpha system with 4GB memory trying to allocate > 100K of static memory. Yes, there are strange systems around. But any one where you have trouble with 100k of static memory easily fits in the category I described as "unless your environment is extremely unusual." -- Richard Maine | Good judgment comes from experience; email: last name at domain . net | experience comes from bad judgment. domain: summertriangle | -- Mark Twain
From: helvio on 2 Jun 2010 08:46
On Jun 2, 6:30 am, glen herrmannsfeldt <g...(a)ugcs.caltech.edu> wrote: > Richard Maine <nos...(a)see.signature> wrote: > > helvio <helvio.vairin...(a)googlemail.com> wrote: > > If it is in a module like that, I believe it will be static if not allocatable. Yes, sorry, I meant static and not automatic. I have several instances of automatic arrays in my code, but this is not the case. My question comes from the fact that my code contains several instances of such modules as in my example, with many potentially non- usable arrays like "U", but also many "V" arrays. In fact, I have more "U"'s than "V"'s, in terms of the amount of allocated memory they require. I am not afraid of exceeding physical memory space, I think I am still far from that (at least for now), but I was worried that the efficiency in accessing the "V"'s in physical memory would be affected by the size of the total allocated memory. Naively, I thought that the less allocated memory I have, the more efficient it is to access it. And since I have the option of declaring the "U"'s as non-static, I imagined that by having *only* "V"'s allocated in the physical memory, their access would be more efficient especially if the sizes of "U" and "V" increase. In sum, I think my doubts reduce to the question of whether the efficiency of accessing the physical memory depends on the size of the allocated memory, or if it is independent of it. I also have the following related question: can the ALLOCATION / DEALLOCATION statements slow down the program if they are called multiple times, as compared with a single static declaration of "U"? e.g. by introducing a loop in my example above: do i=1,M call using_UV ! U is allocated here call kill_U ! U is deallocated here end do Thanks! --helvio |