IIS 5 Hang
in [IIS]
|
From: Juan on 25 Aug 2006 06:32 Pat, Thanks once again. I'll keep you posted of my progress. Is there anything that I can do to return the favor? Cheers -- Juan "Pat [MSFT}" wrote: > Having a large template cache can cause performance problems - basically > there is less memory available for the app to run. So it isn't optimal but > depending on how you are running the code it may not be that big a deal. I > would take it under advisement, but not worry too much about it. > > Pat > > > "Juan" <Juan(a)discussions.microsoft.com> wrote in message > news:85DBE238-BA1F-4BC8-BCA8-5981CEEF5EBC(a)microsoft.com... > > Hi Pat, > > > > Thanks a lot for your points of view. That was the type of expert opinnion > > that we were missing. > > I have not enough words to thank you for your help on this! > > > > We had the MDAC upgrade in our list but now we are going to prioritize it. > > > > I wasn't awre at all about the COM+ fix/rollup, we are going to implement > > it. Is it this one? http://support.microsoft.com/?kbid=912816 > > > > We are going to work with the developers tracing down those areas of the > > code involved in the blocked threads. > > > > Regarding your point 1 (no attribution to long ASPs/many includes), the > > only > > reason why we believe that it might be a contributor is because in the > > Crash > > and Memory reports that we sent you and in many others there are a lot of > > ASP > > Cache related functions and modules ranked high in memory leak > > probability, > > examples: > > > > We find this in many dump reports: > > > > ... > > Memory allocations by the asp.dll template cache manager are responsible > > for > > 297.92 MBytes worth of outstanding allocations. > > > > Large amounts of memory allocated for the ASP Template Cache are usually > > caused by the usage of too much VBScript or JScript within a page (or it's > > associated include files), and may result in degraded ASP performance due > > to > > heap fragmentation or memory consumption by the host process. > > > > ASP Template Cache Size: 236.70 MBytes > > > > This was detected in > > inetinfo__PID__2192__Date__08_14_2006__Time_03_32_04PM__15__Second_Chance_Exception_C0000005.dmp > > ... > > ... > > inetinfo.exe__PID__3428__Date__08_11_2006__Time_07_47_45AM > > The heap 0x02bb0000 contains 64 segments, with very little committed > > memory > > in the heap. Any allocation request for this heap greater than 32.00 > > MBytes > > will fail. > > > > Memory statistics for this heap: > > > > Reserved Memory: 434.23 MBytes > > Committed Memory: 10.30 MBytes (2.37% of reserved) > > Heap Name: ALLOC_CACHE_HANDLER heap for ASP:ResponseBuffer > > > > Function asp!CTemplate::LargeReAlloc+1c > > Allocation type Heap allocation(s) > > Heap handle 0x02c50000 > > Allocation Count 248 allocation(s) > > Allocation Size 314.80 MBytes > > Leak Probability 93% > > > > Function vbscript!DllCanUnloadNow+1c537 > > Allocation type C/C++ runtime allocation(s) > > Allocation Count 47 allocation(s) > > Allocation Size 38.37 MBytes > > Leak Probability 95% > > ... > > > > or > > > > ... > > Function asp!CTemplate::LargeReAlloc+1c > > Allocation type Heap allocation(s) > > Heap handle 0x02e40000 > > Allocation Count 48 allocation(s) > > Allocation Size 52.49 MBytes > > Leak Probability 98% > > ... > > > > Additionally, I see the performance counter ASP Template Cache Misses > > growing exponentially minutes or hours before the hangs. While in normal > > scenarios it grows smoothly towards the recycling points. > > > > Please see the following doc for additional evidence: > > http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc > > > > Also I read in the IIS 5.0 Resource documentation that the delta between > > the > > two following performance counters should be small or inexistent, but in > > our > > case is pretty big: > > > > Maximum File Cache Memory Usage and Current File Cache Memory Usage. > > > > Please see the following doc for additional evidence: > > http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc > > > > Thanks a lot again! > > > > -- > > Juan > > > > > > "Pat [MSFT}" wrote: > > > >> 1) There is _no_ indication whatsoever that page size/include count is a > >> factor. > >> 2) You have at least one thread blocked trying to create an object - > >> there > >> could be valid reasons for this; consider it a symptom not a cause. > >> 3) You have a bunch of threads blocked on thread 112 - has most likely > >> thrown an exception & leaked a lock which is leading to a hang. I would > >> say > >> (based on the blocked threads) that the most likely culprit is MDAC. I > >> would update that first and see if it helps. The #2 culprit is COM+ (you > >> would need to contact MS-Support for the latest COM+ rollup). > >> > >> > >> Pat > >> > >> > >> > >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message > >> news:E97D51CF-9FB6-4F16-B7A1-BCCA3A28FA05(a)microsoft.com... > >> > Hi Bernard, > >> > > >> > Thanks for your quick answer! > >> > > >> > Below you'll see the Crash/Hanhg analysis report. I will send the > >> > Memory > >> > analysis in a separated message. > >> > > >> > A little bit more of background: > >> > > >> > Using MS Web Application Stress Tool in a paralel environment we can > >> > stress > >> > the the apps getting hang rule dumps but we cannot really hang the > >> > service > >> > as > >> > in the production incidents. The load generated in our WAS scripts is > >> > not > >> > as > >> > wide and realistic as the prod load. > >> > > >> > Thanks a lot > >> > -- > >> > Juan > >> > > >> > > >> > Analysis Summary > >> > Type Description Recommendation > >> > Warning Detected possible blocking or leaked critical section at > >> > NTDLL!LoaderLock owned by thread 112 in > >> > > >> > inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS > >> > Hang > >> > Dump.dmp > >> > > >> > Impact of this lock > >> > > >> > 71.88% of executing ASP Requests blocked > >> > > >> > 19.71% of threads blocked > >> > > >> > (Threads 30 52 55 56 59 66 68 74 75 76 77 79 87 101 103 104 106 107 115 > >> > 116 > >> > 117 122 124 128 130 132 136) > >> > > >> > The following functions are trying to enter this critical section > >> > NTDLL!LdrpGetProcedureAddress+122 > >> > NTDLL!LdrUnloadDll+5f > >
From: Juan on 25 Aug 2006 06:32 Pat, Thanks once again. I'll keep you posted of my progress. Is there anything that I can do to return the favor? Cheers -- Juan "Pat [MSFT}" wrote: > Having a large template cache can cause performance problems - basically > there is less memory available for the app to run. So it isn't optimal but > depending on how you are running the code it may not be that big a deal. I > would take it under advisement, but not worry too much about it. > > Pat > > > "Juan" <Juan(a)discussions.microsoft.com> wrote in message > news:85DBE238-BA1F-4BC8-BCA8-5981CEEF5EBC(a)microsoft.com... > > Hi Pat, > > > > Thanks a lot for your points of view. That was the type of expert opinnion > > that we were missing. > > I have not enough words to thank you for your help on this! > > > > We had the MDAC upgrade in our list but now we are going to prioritize it. > > > > I wasn't awre at all about the COM+ fix/rollup, we are going to implement > > it. Is it this one? http://support.microsoft.com/?kbid=912816 > > > > We are going to work with the developers tracing down those areas of the > > code involved in the blocked threads. > > > > Regarding your point 1 (no attribution to long ASPs/many includes), the > > only > > reason why we believe that it might be a contributor is because in the > > Crash > > and Memory reports that we sent you and in many others there are a lot of > > ASP > > Cache related functions and modules ranked high in memory leak > > probability, > > examples: > > > > We find this in many dump reports: > > > > ... > > Memory allocations by the asp.dll template cache manager are responsible > > for > > 297.92 MBytes worth of outstanding allocations. > > > > Large amounts of memory allocated for the ASP Template Cache are usually > > caused by the usage of too much VBScript or JScript within a page (or it's > > associated include files), and may result in degraded ASP performance due > > to > > heap fragmentation or memory consumption by the host process. > > > > ASP Template Cache Size: 236.70 MBytes > > > > This was detected in > > inetinfo__PID__2192__Date__08_14_2006__Time_03_32_04PM__15__Second_Chance_Exception_C0000005.dmp > > ... > > ... > > inetinfo.exe__PID__3428__Date__08_11_2006__Time_07_47_45AM > > The heap 0x02bb0000 contains 64 segments, with very little committed > > memory > > in the heap. Any allocation request for this heap greater than 32.00 > > MBytes > > will fail. > > > > Memory statistics for this heap: > > > > Reserved Memory: 434.23 MBytes > > Committed Memory: 10.30 MBytes (2.37% of reserved) > > Heap Name: ALLOC_CACHE_HANDLER heap for ASP:ResponseBuffer > > > > Function asp!CTemplate::LargeReAlloc+1c > > Allocation type Heap allocation(s) > > Heap handle 0x02c50000 > > Allocation Count 248 allocation(s) > > Allocation Size 314.80 MBytes > > Leak Probability 93% > > > > Function vbscript!DllCanUnloadNow+1c537 > > Allocation type C/C++ runtime allocation(s) > > Allocation Count 47 allocation(s) > > Allocation Size 38.37 MBytes > > Leak Probability 95% > > ... > > > > or > > > > ... > > Function asp!CTemplate::LargeReAlloc+1c > > Allocation type Heap allocation(s) > > Heap handle 0x02e40000 > > Allocation Count 48 allocation(s) > > Allocation Size 52.49 MBytes > > Leak Probability 98% > > ... > > > > Additionally, I see the performance counter ASP Template Cache Misses > > growing exponentially minutes or hours before the hangs. While in normal > > scenarios it grows smoothly towards the recycling points. > > > > Please see the following doc for additional evidence: > > http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc > > > > Also I read in the IIS 5.0 Resource documentation that the delta between > > the > > two following performance counters should be small or inexistent, but in > > our > > case is pretty big: > > > > Maximum File Cache Memory Usage and Current File Cache Memory Usage. > > > > Please see the following doc for additional evidence: > > http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc > > > > Thanks a lot again! > > > > -- > > Juan > > > > > > "Pat [MSFT}" wrote: > > > >> 1) There is _no_ indication whatsoever that page size/include count is a > >> factor. > >> 2) You have at least one thread blocked trying to create an object - > >> there > >> could be valid reasons for this; consider it a symptom not a cause. > >> 3) You have a bunch of threads blocked on thread 112 - has most likely > >> thrown an exception & leaked a lock which is leading to a hang. I would > >> say > >> (based on the blocked threads) that the most likely culprit is MDAC. I > >> would update that first and see if it helps. The #2 culprit is COM+ (you > >> would need to contact MS-Support for the latest COM+ rollup). > >> > >> > >> Pat > >> > >> > >> > >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message > >> news:E97D51CF-9FB6-4F16-B7A1-BCCA3A28FA05(a)microsoft.com... > >> > Hi Bernard, > >> > > >> > Thanks for your quick answer! > >> > > >> > Below you'll see the Crash/Hanhg analysis report. I will send the > >> > Memory > >> > analysis in a separated message. > >> > > >> > A little bit more of background: > >> > > >> > Using MS Web Application Stress Tool in a paralel environment we can > >> > stress > >> > the the apps getting hang rule dumps but we cannot really hang the > >> > service > >> > as > >> > in the production incidents. The load generated in our WAS scripts is > >> > not > >> > as > >> > wide and realistic as the prod load. > >> > > >> > Thanks a lot > >> > -- > >> > Juan > >> > > >> > > >> > Analysis Summary > >> > Type Description Recommendation > >> > Warning Detected possible blocking or leaked critical section at > >> > NTDLL!LoaderLock owned by thread 112 in > >> > > >> > inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS > >> > Hang > >> > Dump.dmp > >> > > >> > Impact of this lock > >> > > >> > 71.88% of executing ASP Requests blocked > >> > > >> > 19.71% of threads blocked > >> > > >> > (Threads 30 52 55 56 59 66 68 74 75 76 77 79 87 101 103 104 106 107 115 > >> > 116 > >> > 117 122 124 128 130 132 136) > >> > > >> > The following functions are trying to enter this critical section > >> > NTDLL!LdrpGetProcedureAddress+122 > >> > NTDLL!LdrUnloadDll+5f > >
From: Pat [MSFT} on 25 Aug 2006 16:54 No worries. Just post up when you get it finally fixed - it will help everyone else. Pat "Juan" <Juan(a)discussions.microsoft.com> wrote in message news:007E6022-8862-471D-8790-1BCCBDB50163(a)microsoft.com... > Pat, > Thanks once again. > I'll keep you posted of my progress. > Is there anything that I can do to return the favor? > Cheers > -- > Juan > > > "Pat [MSFT}" wrote: > >> Having a large template cache can cause performance problems - basically >> there is less memory available for the app to run. So it isn't optimal >> but >> depending on how you are running the code it may not be that big a deal. >> I >> would take it under advisement, but not worry too much about it. >> >> Pat >> >> >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message >> news:85DBE238-BA1F-4BC8-BCA8-5981CEEF5EBC(a)microsoft.com... >> > Hi Pat, >> > >> > Thanks a lot for your points of view. That was the type of expert >> > opinnion >> > that we were missing. >> > I have not enough words to thank you for your help on this! >> > >> > We had the MDAC upgrade in our list but now we are going to prioritize >> > it. >> > >> > I wasn't awre at all about the COM+ fix/rollup, we are going to >> > implement >> > it. Is it this one? http://support.microsoft.com/?kbid=912816 >> > >> > We are going to work with the developers tracing down those areas of >> > the >> > code involved in the blocked threads. >> > >> > Regarding your point 1 (no attribution to long ASPs/many includes), the >> > only >> > reason why we believe that it might be a contributor is because in the >> > Crash >> > and Memory reports that we sent you and in many others there are a lot >> > of >> > ASP >> > Cache related functions and modules ranked high in memory leak >> > probability, >> > examples: >> > >> > We find this in many dump reports: >> > >> > ... >> > Memory allocations by the asp.dll template cache manager are >> > responsible >> > for >> > 297.92 MBytes worth of outstanding allocations. >> > >> > Large amounts of memory allocated for the ASP Template Cache are >> > usually >> > caused by the usage of too much VBScript or JScript within a page (or >> > it's >> > associated include files), and may result in degraded ASP performance >> > due >> > to >> > heap fragmentation or memory consumption by the host process. >> > >> > ASP Template Cache Size: 236.70 MBytes >> > >> > This was detected in >> > inetinfo__PID__2192__Date__08_14_2006__Time_03_32_04PM__15__Second_Chance_Exception_C0000005.dmp >> > ... >> > ... >> > inetinfo.exe__PID__3428__Date__08_11_2006__Time_07_47_45AM >> > The heap 0x02bb0000 contains 64 segments, with very little committed >> > memory >> > in the heap. Any allocation request for this heap greater than 32.00 >> > MBytes >> > will fail. >> > >> > Memory statistics for this heap: >> > >> > Reserved Memory: 434.23 MBytes >> > Committed Memory: 10.30 MBytes (2.37% of reserved) >> > Heap Name: ALLOC_CACHE_HANDLER heap for ASP:ResponseBuffer >> > >> > Function asp!CTemplate::LargeReAlloc+1c >> > Allocation type Heap allocation(s) >> > Heap handle 0x02c50000 >> > Allocation Count 248 allocation(s) >> > Allocation Size 314.80 MBytes >> > Leak Probability 93% >> > >> > Function vbscript!DllCanUnloadNow+1c537 >> > Allocation type C/C++ runtime allocation(s) >> > Allocation Count 47 allocation(s) >> > Allocation Size 38.37 MBytes >> > Leak Probability 95% >> > ... >> > >> > or >> > >> > ... >> > Function asp!CTemplate::LargeReAlloc+1c >> > Allocation type Heap allocation(s) >> > Heap handle 0x02e40000 >> > Allocation Count 48 allocation(s) >> > Allocation Size 52.49 MBytes >> > Leak Probability 98% >> > ... >> > >> > Additionally, I see the performance counter ASP Template Cache Misses >> > growing exponentially minutes or hours before the hangs. While in >> > normal >> > scenarios it grows smoothly towards the recycling points. >> > >> > Please see the following doc for additional evidence: >> > http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc >> > >> > Also I read in the IIS 5.0 Resource documentation that the delta >> > between >> > the >> > two following performance counters should be small or inexistent, but >> > in >> > our >> > case is pretty big: >> > >> > Maximum File Cache Memory Usage and Current File Cache Memory Usage. >> > >> > Please see the following doc for additional evidence: >> > http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc >> > >> > Thanks a lot again! >> > >> > -- >> > Juan >> > >> > >> > "Pat [MSFT}" wrote: >> > >> >> 1) There is _no_ indication whatsoever that page size/include count is >> >> a >> >> factor. >> >> 2) You have at least one thread blocked trying to create an object - >> >> there >> >> could be valid reasons for this; consider it a symptom not a cause. >> >> 3) You have a bunch of threads blocked on thread 112 - has most likely >> >> thrown an exception & leaked a lock which is leading to a hang. I >> >> would >> >> say >> >> (based on the blocked threads) that the most likely culprit is MDAC. >> >> I >> >> would update that first and see if it helps. The #2 culprit is COM+ >> >> (you >> >> would need to contact MS-Support for the latest COM+ rollup). >> >> >> >> >> >> Pat >> >> >> >> >> >> >> >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message >> >> news:E97D51CF-9FB6-4F16-B7A1-BCCA3A28FA05(a)microsoft.com... >> >> > Hi Bernard, >> >> > >> >> > Thanks for your quick answer! >> >> > >> >> > Below you'll see the Crash/Hanhg analysis report. I will send the >> >> > Memory >> >> > analysis in a separated message. >> >> > >> >> > A little bit more of background: >> >> > >> >> > Using MS Web Application Stress Tool in a paralel environment we can >> >> > stress >> >> > the the apps getting hang rule dumps but we cannot really hang the >> >> > service >> >> > as >> >> > in the production incidents. The load generated in our WAS scripts >> >> > is >> >> > not >> >> > as >> >> > wide and realistic as the prod load. >> >> > >> >> > Thanks a lot >> >> > -- >> >> > Juan >> >> > >> >> > >> >> > Analysis Summary >> >> > Type Description Recommendation >> >> > Warning Detected possible blocking or leaked critical section at >> >> > NTDLL!LoaderLock owned by thread 112 in >> >> > >> >> > inetinfo.exe__PID
From: Prophet on 25 Aug 2006 16:57
Wow guys, some troubleshooting. I have been on vacation for a while. And I will surely catch up and (very carefully) read this thread. Just wanted to drop a line to say thanks to Pat and Bernard. And a hello to Juan. |