Memory bug in Tcl 8.6
in [TCL]
|
Prev: kbs 0.4 binaries
Next: 16-bit greyscale photos in Tk
From: Robert Heller on 26 Jan 2010 10:08 At Tue, 26 Jan 2010 06:03:23 -0800 (PST) pmarin <pacogeek(a)gmail.com> wrote: > > On Jan 26, 2:46=A0pm, Robert Heller <hel...(a)deepsoft.com> wrote: > > At Tue, 26 Jan 2010 05:21:12 -0800 (PST) pmarin <pacog...(a)gmail.com> wrot= > e: > > > > > > > > > Hi all. > > > > > Tcl "never" release the memory. Try the following code several times > > > (source the script several times in the same shell), the first time It > > > takes the memory and the other times I will not take more memory but > > > It "never" will release the memory. > > > > > Tested in Linus =A02.6.28.4 > > >http://gist.github.com/286826 > > > > If you are looking at top (or similar process memory tool), this is > > *normal* for most C/UNIX programs: ALL UNIX-style C programs that use > > malloc() (or UNIX-style C++ programs that use operator new) maintain a > > heap: initially the heap is empty and the first time malloc (new) is > > called, the brk() system call is used to extend the process's memory > > and the heap is started. =A0When objects are free()ed (deleted), the > > memory is placed on the heap and future malloc()s (news) use the memory > > off the heap. =A0At no point is the memory returned to the O/S until > > process termination. > > > > -- > > Robert Heller =A0 =A0 =A0 =A0 =A0 =A0 -- 978-544-6933 > > Deepwoods Software =A0 =A0 =A0 =A0-- Download the Model Railroad Systemht= > tp://www.deepsoft.com/=A0-- Binaries for Linux and MS-Windows > > hel...(a)deepsoft.com =A0 =A0 =A0 --http://www.deepsoft.com/ModelRailroadSy= > stem/ > > Is there any reason for this behaviour? As George stated, this has been standard, common practice for UNIX C programms, since the stone age. The brk() system call is relativly costly (in terms of system overhead) and it is very difficult to undo the brk() call, since there is no predicting what memory object will be free()ed when. It is not possible to free up a 'hole' under UNIX when memory is allocated with brk(), so only memory allocated at the high end of the heap can be returned to the O/S when it is free()ed -- other memory blocks cannot be returned to the O/S, until ALL of the memory blocks above them have been free()ed (and returned to the O/S). It is faster and simplier to just hang only all of the memory allocated (in a heap list) and re-cycle it as needed (UNIX memory management has been 'green' for like 40+ years!). Actually, malloc() et al, use brk() to grab *large* chunks of memory on an as needed basis. Under Linux (and UNIX) process termination (for whatever reason) causes the kernel to reclaim ALL of the memory used by a process, so there is no need for a process to explicitly return any memory it allocates back to the O/S, since it all gets automagically reclaimed when the process terminates. > -- Robert Heller -- 978-544-6933 Deepwoods Software -- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows heller(a)deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/
From: Georgios Petasis on 26 Jan 2010 10:13 στις 26/1/2010 17:08, O/H Robert Heller έγραψε: > At Tue, 26 Jan 2010 06:03:23 -0800 (PST) pmarin<pacogeek(a)gmail.com> wrote: > >> >> On Jan 26, 2:46=A0pm, Robert Heller<hel...(a)deepsoft.com> wrote: >>> At Tue, 26 Jan 2010 05:21:12 -0800 (PST) pmarin<pacog...(a)gmail.com> wrot= >> e: >>> >>> >>> >>>> Hi all. >>> >>>> Tcl "never" release the memory. Try the following code several times >>>> (source the script several times in the same shell), the first time It >>>> takes the memory and the other times I will not take more memory but >>>> It "never" will release the memory. >>> >>>> Tested in Linus =A02.6.28.4 >>>> http://gist.github.com/286826 >>> >>> If you are looking at top (or similar process memory tool), this is >>> *normal* for most C/UNIX programs: ALL UNIX-style C programs that use >>> malloc() (or UNIX-style C++ programs that use operator new) maintain a >>> heap: initially the heap is empty and the first time malloc (new) is >>> called, the brk() system call is used to extend the process's memory >>> and the heap is started. =A0When objects are free()ed (deleted), the >>> memory is placed on the heap and future malloc()s (news) use the memory >>> off the heap. =A0At no point is the memory returned to the O/S until >>> process termination. >>> >>> -- >>> Robert Heller =A0 =A0 =A0 =A0 =A0 =A0 -- 978-544-6933 >>> Deepwoods Software =A0 =A0 =A0 =A0-- Download the Model Railroad Systemht= >> tp://www.deepsoft.com/=A0-- Binaries for Linux and MS-Windows >>> hel...(a)deepsoft.com =A0 =A0 =A0 --http://www.deepsoft.com/ModelRailroadSy= >> stem/ >> >> Is there any reason for this behaviour? > > As George stated, this has been standard, common practice for UNIX C > programms, since the stone age. The brk() system call is relativly > costly (in terms of system overhead) and it is very difficult to undo > the brk() call, since there is no predicting what memory object will be > free()ed when. It is not possible to free up a 'hole' under UNIX when > memory is allocated with brk(), so only memory allocated at the high > end of the heap can be returned to the O/S when it is free()ed -- other > memory blocks cannot be returned to the O/S, until ALL of the memory > blocks above them have been free()ed (and returned to the O/S). It is > faster and simplier to just hang only all of the memory allocated (in a > heap list) and re-cycle it as needed (UNIX memory management has been > 'green' for like 40+ years!). Actually, malloc() et al, use brk() to > grab *large* chunks of memory on an as needed basis. > > Under Linux (and UNIX) process termination (for whatever reason) causes > the kernel to reclaim ALL of the memory used by a process, so there is > no need for a process to explicitly return any memory it allocates back > to the O/S, since it all gets automagically reclaimed when the process > terminates. > >> > And also keep in mind that Tcl never frees a Tcl object. Instead it is placed in a pool of "unused" objects, to be returned when a new tcl object is required. The details are in generic/tclObj.c (global variable tclFreeObjList holds this list of unused objects). George
From: Robert Heller on 26 Jan 2010 10:59 At Tue, 26 Jan 2010 17:13:42 +0200 Georgios Petasis <petasis(a)iit.demokritos.gr> wrote: > > And also keep in mind that Tcl never frees a Tcl object. Instead it is > placed in a pool of "unused" objects, to be returned when a new tcl > object is required. The details are in generic/tclObj.c (global variable > tclFreeObjList holds this list of unused objects). This is a common 'trick' used by many C programs to optimize memory allocation. Keeping separate 'free lists' of some commonly used/reused data objects avoids searching through the heap for just the right sized piece of memory. Freeing is also quicker, since there is no need to search the heap for adjacent memory blocks to join the freed object with. > > George > -- Robert Heller -- 978-544-6933 Deepwoods Software -- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows heller(a)deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/
From: pmarin on 26 Jan 2010 11:10 On Jan 26, 4:59 pm, Robert Heller <hel...(a)deepsoft.com> wrote: > At Tue, 26 Jan 2010 17:13:42 +0200 Georgios Petasis <peta...(a)iit.demokritos.gr> wrote: > > > > > And also keep in mind that Tcl never frees a Tcl object. Instead it is > > placed in a pool of "unused" objects, to be returned when a new tcl > > object is required. The details are in generic/tclObj.c (global variable > > tclFreeObjList holds this list of unused objects). > > This is a common 'trick' used by many C programs to optimize memory > allocation. Keeping separate 'free lists' of some commonly used/reused > data objects avoids searching through the heap for just the right sized > piece of memory. Freeing is also quicker, since there is no need to > search the heap for adjacent memory blocks to join the freed object with. > > > > > George > > -- > Robert Heller -- 978-544-6933 > Deepwoods Software -- Download the Model Railroad Systemhttp://www.deepsoft.com/ -- Binaries for Linux and MS-Windows > hel...(a)deepsoft.com --http://www.deepsoft.com/ModelRailroadSystem/ Is there any way to know how much memory is actually used or Just I only can count the object used?
From: Georgios Petasis on 26 Jan 2010 11:55
στις 26/1/2010 18:10, O/H pmarin έγραψε: > On Jan 26, 4:59 pm, Robert Heller<hel...(a)deepsoft.com> wrote: >> At Tue, 26 Jan 2010 17:13:42 +0200 Georgios Petasis<peta...(a)iit.demokritos.gr> wrote: >> >> >> >>> And also keep in mind that Tcl never frees a Tcl object. Instead it is >>> placed in a pool of "unused" objects, to be returned when a new tcl >>> object is required. The details are in generic/tclObj.c (global variable >>> tclFreeObjList holds this list of unused objects). >> >> This is a common 'trick' used by many C programs to optimize memory >> allocation. Keeping separate 'free lists' of some commonly used/reused >> data objects avoids searching through the heap for just the right sized >> piece of memory. Freeing is also quicker, since there is no need to >> search the heap for adjacent memory blocks to join the freed object with. >> >> >> >>> George >> >> -- >> Robert Heller -- 978-544-6933 >> Deepwoods Software -- Download the Model Railroad Systemhttp://www.deepsoft.com/ -- Binaries for Linux and MS-Windows >> hel...(a)deepsoft.com --http://www.deepsoft.com/ModelRailroadSystem/ > > Is there any way to know how much memory is actually used or Just I > only can count the object used? You have to re-compile Tcl with memory debugging enabled. This will add a "memory" command that will provide relevant information. George |