From: Juan on
Hi,

I wonder if you guys can give us a hand with our analysis of this IIS hangs.

We are running 6 in-process ASP websites (only 2 of them have 20+ virtual
dirs / logical apps) the other four are pretty light. The 2 main websites
have several hundreds of ASP pages.

A little bit more than a thousand unique visitors per day.

Connections to SQL 2000, DB2, and MQ.

The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB RAM.

The inetinfo hangs randomly (no-load related) with an average of 2/3 times a
week. Recycling IIS the problem goes away.

We are using IIS 5 Process Recycle on daily basis. No too much help.

We are using IIS Diagnostic Tools to obtain and analyze memory dumps.

Based on your incredible IIS Diagnostic Tools reports, we have the theory
that the root cause of our problems is the large number of big pages and the
extensive number of include files (more than 100 includes in some pages). We
think that they produce heap fragmentation in the ASP template cache that
materializes in memory lacks.

We have more than 2 months analyzing the symptoms, and we obtained some
interesting stuff, but the advise of the experts will help us a lot
confirming our theories or giving us new leads.

Can I paste in a next message either dump files or IIS diagnostics HTML
reports for your opinion?

Your help will be really appreciated.

Thanks a lot,

--
Juan
From: Bernard Cheah [MVP] on
You can start with the html and provide a link to the dump file if needed.

--
Regards,
Bernard Cheah
http://www.iis.net/
http://www.iis-resources.com/
http://msmvps.com/blogs/bernard/


"Juan" <Juan(a)discussions.microsoft.com> wrote in message
news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com...
> Hi,
>
> I wonder if you guys can give us a hand with our analysis of this IIS
> hangs.
>
> We are running 6 in-process ASP websites (only 2 of them have 20+ virtual
> dirs / logical apps) the other four are pretty light. The 2 main websites
> have several hundreds of ASP pages.
>
> A little bit more than a thousand unique visitors per day.
>
> Connections to SQL 2000, DB2, and MQ.
>
> The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB
> RAM.
>
> The inetinfo hangs randomly (no-load related) with an average of 2/3 times
> a
> week. Recycling IIS the problem goes away.
>
> We are using IIS 5 Process Recycle on daily basis. No too much help.
>
> We are using IIS Diagnostic Tools to obtain and analyze memory dumps.
>
> Based on your incredible IIS Diagnostic Tools reports, we have the theory
> that the root cause of our problems is the large number of big pages and
> the
> extensive number of include files (more than 100 includes in some pages).
> We
> think that they produce heap fragmentation in the ASP template cache that
> materializes in memory lacks.
>
> We have more than 2 months analyzing the symptoms, and we obtained some
> interesting stuff, but the advise of the experts will help us a lot
> confirming our theories or giving us new leads.
>
> Can I paste in a next message either dump files or IIS diagnostics HTML
> reports for your opinion?
>
> Your help will be really appreciated.
>
> Thanks a lot,
>
> --
> Juan


From: Juan on
Hi Bernard,

Thanks for your quick answer!

Below you'll see the Crash/Hanhg analysis report. I will send the Memory
analysis in a separated message.

A little bit more of background:

Using MS Web Application Stress Tool in a paralel environment we can stress
the the apps getting hang rule dumps but we cannot really hang the service as
in the production incidents. The load generated in our WAS scripts is not as
wide and realistic as the prod load.

Thanks a lot
--
Juan


Analysis Summary
Type Description Recommendation
Warning Detected possible blocking or leaked critical section at
NTDLL!LoaderLock owned by thread 112 in

inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
Dump.dmp

Impact of this lock

71.88% of executing ASP Requests blocked

19.71% of threads blocked

(Threads 30 52 55 56 59 66 68 74 75 76 77 79 87 101 103 104 106 107 115 116
117 122 124 128 130 132 136)

The following functions are trying to enter this critical section
NTDLL!LdrpGetProcedureAddress+122
NTDLL!LdrUnloadDll+5f
KERNEL32!GetModuleFileNameW+ad
NTDLL!LdrShutdownThread+38

The following module(s) are involved with this critical section
C:\WINNT\system32\NTDLL.DLL from Microsoft Corporation
C:\WINNT\system32\KERNEL32.DLL from Microsoft Corporation
The following vendors were identified for follow up based on root cause
analysis

....

Warning The following threads in
inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang

Dump.dmp are processing a client request and is/are not fully resolved.
Further analysis of these threads is

recommended in order to determine what may be blocking the request(s).

( 0 32 78 81 105 109 113 118 123 127 )

28.13% of executing ASP Requests blocked

6.25% of IIS worker threads blocked

7.30% of threads blocked


....

Warning The COM+ STA ThreadPool may have been depleted prior to the time
of the dump in

inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
Dump.dmp. There is more than one

activity bound to the following COM+ STA ThreadPool threads:

52 68 56 76 32 75 105 81 30 31

See the COM+ STA ThreadPool Report for more detail. 52% of the STA
ThreadPool threads are idle, however some

threads still have more than one activity bound to them. This can happen for
various reasons:
If COM+ objects are being leaked. If there is a memory leak, you should use
DebugDiag to monitor the leak for

root cause. See the "Troubleshooting Memory Leaks with DebugDiag" topic in
the DebugDiag.chm file for more

information.
....



Warning 3 client connection(s) in
inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang

Dump.dmp have been executing a request for more than 90 seconds. Please see
the Client Connections section of

this report for more detailed information about the connection(s).
Information
DebugDiag did not detect any known problems in H:\Logs\IIS Diag
analysis\Prod Memory Dumps

08_16_2006_Outage\aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang Dump.dmp using the

current set of scripts.





Analysis Details
Table Of Contents
dllhost.exe__System
Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
Dump.dmp
Top 5 threads by CPU time
Thread report
Well-Known COM STA Threads Report

inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
Dump.dmp
Top 5 threads by CPU time
Locked critical section report
Thread report
Well-Known COM STA Threads Report
HTTP Report
ASP Report

aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang
Dump.dmp
Top 5 threads by CPU time
Thread report
Well-Known COM STA Threads Report


Report for dllhost.exe__System
Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
Dump.dmp
Type of Analysis Performed Hang Analysis
Machine Name NLGPWEBIIS05
Operating System Windows 2000 Service Pack 4
Number Of Processors 4
Process ID 3020
Process Image C:\WINNT\system32\dllhost.exe
System Up-Time 1 day(s) 05:05:30
Process Up-Time 1 day(s) 05:04:28


Top 5 Threads by CPU time
Note - Times include both user mode and kernel mode for each thread Thread
ID: 5 Total CPU Time: 0 day(s)

00:00:05.562 Entry Point for Thread: msvcrt!_endthreadex+32
Thread ID: 6 Total CPU Time: 0 day(s) 00:00:00.750 Entry Point for
Thread: rpcrt4!ThreadStartRoutine
Thread ID: 4 Total CPU Time: 0 day(s) 00:00:00.31 Entry Point for
Thread: msvcrt!_endthread+3f
Thread ID: 7 Total CPU Time: 0 day(s) 00:00:00.31 Entry Point for
Thread:

comsvcs!CEventServer::DispatchEvents
Thread ID: 3 Total CPU Time: 0 day(s) 00:00:00.15 Entry Point for
Thread: txfaux!WORK_QUEUE::ThreadLoop



Thread report

Thread 0 - System ID 3016
Entry point dllhost!WinMainCRTStartup
Create time 08/15/2006 6:08:50 AM
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.15


This thread is not fully resolved and may or may not be a problem. Further
analysis of these threads may be

required.

Function Source
NTDLL!ZwWaitForSingleObject+b
KERNEL32!WaitForSingleObjectEx+71
KERNEL32!WaitForSingleObject+f



....


....

Thread 7 - System ID 3060
Entry point comsvcs!CEventServer::DispatchEvents
Create time 08/15/2006 6:08:50 AM
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.31


This thread is making a COM call to neutral apartment (NA) within the same
process

Function Source
NTDLL!ZwWaitForMultipleObjects+b
KERNEL32!WaitForMultipleObjectsEx+ea
KERNEL32!WaitForMultipleObjects+17
rpcrt4!Invoke+30
rpcrt4!NdrStubCall2+664
rpcrt4!CStdStubBuffer_Invoke+c8
OLE32!SyncStubInvoke+61
OLE32!StubInvoke+a8
OLE32!CCtxComChnl::ContextInvoke+bb
OLE32!MTAInvoke+18
OLE32!AppInvoke+b5
OLE32!ComInvokeWithLockAndIPID+297
OLE32!ComInvoke+41
OLE32!ThreadDispatch+21
OLE32!DispatchCall+24
OLE32!CRpcChannelBuffer::SwitchAptAndDispatchCall+34
OLE32!CRpcChannelBuffer::SendReceive2+96
OLE32!CRpcChannelBuffer::SendReceive+11
OLE32!CAptRpcChnl::SendReceive+a9
OLE32!CCtxComChnl::SendReceive+124
From: Juan on
Hi Bernard,

Now the main fragments of the memory analysis report. Thanks again.

Type Description Recommendation
Warning rpcrt4.dll is responsible for 6.90 KBytes worth of outstanding
allocations. The following are the top 2 memory consuming functions:



rpcrt4!operator new+12: 6.81 KBytes worth of outstanding allocations.

rpcrt4!NdrpCreateNonDelegatedAsyncStub+71: 60 Bytes worth of outstanding
allocations.



This was detected in dllhost.exe__System
Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
Dump.dmp If this is unexpected, please contact the vendor of this module,
Microsoft Corporation, for further assistance with this issue.
Warning clbcatq.dll is responsible for 3.17 KBytes worth of outstanding
allocations. The following are the top 2 memory consuming functions:



clbcatq!CComCLBCatalog::GetClassInfoW+6b: 1,016 Bytes worth of outstanding
allocations.

clbcatq!CComClass::Init+8b5: 488 Bytes worth of outstanding allocations.



This was detected in dllhost.exe__System
Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
Dump.dmp If this is unexpected, please contact the vendor of this module,
Microsoft Corporation, for further assistance with this issue.
Warning Detected symptoms of high fragmentation in the following heaps in
inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
Dump.dmp





0x00070000 (Default process heap - 75.86% Fragmented)

0x02da0000 (ASP!CTemplate::sm_hLargeHeap - 93.68% Fragmented) Heap
fragmentation is often caused by one of the following two reasons



1. Small heap memory blocks that are leaked (allocated but never freed) over
time

2. Mixing long lived small allocations with short lived long allocations



Both of these reasons can prevent the NT heap manager from using free memory
effeciently since they are spread as small fragments that cannot be used as a
single large allocation
Information DebugDiag did not detect LeakTrack.dll loaded in
inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
Dump.dmp, so no leak analysis was performed on this file. If you are
troubleshooting a memory leak, please ensure LeakTrack.dll is injected into
the target process using the DebugDiag tool before or generating new dumps.



For information regarding installation and usage of the IISDiag tool, please
see the included help file.
Information
DebugDiag did not detect any known native heap(unmanaged) problems in
aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang
Dump.dmp using the current set of scripts.



Information DebugDiag did not detect LeakTrack.dll loaded in
aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang
Dump.dmp, so no leak analysis was performed on this file. If you are
troubleshooting a memory leak, please ensure LeakTrack.dll is injected into
the target process using the DebugDiag tool before or generating new dumps.



For information regarding installation and usage of the IISDiag tool, please
see the included help file.
Information
DebugDiag did not detect any known native heap(unmanaged) problems in
dllhost.exe__System
Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
Dump.dmp using the current set of scripts.








Analysis Details
Table Of Contents
dllhost.exe__System
Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
Dump.dmp

Virtual Memory Analysis Report

Heap Analysis Report

Leak Analysis Report

Outstanding allocation summary

Detailed module report (Memory)

Detailed module report (Handles)



inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
Dump.dmp

Virtual Memory Analysis Report

Heap Analysis Report



aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang
Dump.dmp

Virtual Memory Analysis Report

Heap Analysis Report




Report for dllhost.exe__System
Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
Dump.dmp



Type of Analysis Performed Memory Pressure Analysis
Machine Name NLGPWEBIIS05
Operating System Windows 2000 Service Pack 4
Number Of Processors 4
Process ID 3020
Process Image C:\WINNT\system32\dllhost.exe
System Up-Time 1 day(s) 05:05:30
Process Up-Time 1 day(s) 05:04:28

Virtual Memory Analysis


Virtual Memory Summary
Size of largest free VM block 1.17 GBytes
Free memory fragmentation 40.32%
Free Memory 1.96 GBytes (98.16% of Total Memory)
Reserved Memory 12.79 MBytes (0.62% of Total Memory)
Committed Memory 24.84 MBytes (1.21% of Total Memory)
Total Memory 2.00 GBytes
Largest free block at 0x00000000`1a47d000





Virtual Memory Details
Virtual Allocations
Loaded Modules
Threads
System
Page Heaps
Native Heaps






7.44 MBytes
19.67 MBytes
4.32 MBytes
12.00 KBytes
0 Bytes
6.19 MBytes






Virtual Allocation Summary
Reserved memory 6.00 MBytes
Committed memory 1.44 MBytes
Mapped memory 3.32 MBytes
Reserved block count 6 blocks
Committed block count 30 blocks
Mapped block count 17 blocks

Loaded Module Summary
Number of Modules 64 Modules
Total reserved memory 0 Bytes
Total committed memory 19.67 MBytes


....

Report for dllhost.exe__System
Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
Dump.dmp
Type of Analysis Performed Memory Pressure Analysis
Machine Name NLGPWEBIIS05
Operating System Windows 2000 Service Pack 4
Number Of Processors 4
Process ID 3020
Process Image C:\WINNT\system32\dllhost.exe
System Up-Time 1 day(s) 05:05:30
Process Up-Time 1 day(s) 05:04:28

Virtual Memory Analysis


Virtual Memory Summary
Size of largest free VM block 1.17 GBytes
Free memory fragmentation 40.32%
Free Memory 1.96 GBytes (98.16% of Total Memory)
Reserved Memory 12.79 MBytes (0.62% of Total Memory)
Committed Memory 24.84 MBytes (1.21% of Total Memory)
Total Memory 2.00 GBytes
Largest free block at 0x00000000`1a47d000



....

Virtual Allocation Summary
Reserved memory 6.00 MBytes
Committed memory 1.44 MBytes
Mapped memory
From: Juan on
Hi Bernard,

Here are the links to the full IIS Diagnostics tool reports.

The actual dumps are more than 100 MB large, so I cannot upload them by the
time being, let me see what I can do.

http://www.compilar.com/garage/IIS_Report__Date_08_17_2006__Time_08_18_50AM__117.htm

http://www.compilar.com/garage/Memory_Report__Date_08_17_2006__Time_09_04_50AM__215.htm

Comment: In other outages and high stress dumps, I got much more leak
probability occurencies and higher percentages than in this memory report
sample.

Thanks again.

--
Juan


"Bernard Cheah [MVP]" wrote:

> You can start with the html and provide a link to the dump file if needed.
>
> --
> Regards,
> Bernard Cheah
> http://www.iis.net/
> http://www.iis-resources.com/
> http://msmvps.com/blogs/bernard/
>
>
> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com...
> > Hi,
> >
> > I wonder if you guys can give us a hand with our analysis of this IIS
> > hangs.
> >
> > We are running 6 in-process ASP websites (only 2 of them have 20+ virtual
> > dirs / logical apps) the other four are pretty light. The 2 main websites
> > have several hundreds of ASP pages.
> >
> > A little bit more than a thousand unique visitors per day.
> >
> > Connections to SQL 2000, DB2, and MQ.
> >
> > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB
> > RAM.
> >
> > The inetinfo hangs randomly (no-load related) with an average of 2/3 times
> > a
> > week. Recycling IIS the problem goes away.
> >
> > We are using IIS 5 Process Recycle on daily basis. No too much help.
> >
> > We are using IIS Diagnostic Tools to obtain and analyze memory dumps.
> >
> > Based on your incredible IIS Diagnostic Tools reports, we have the theory
> > that the root cause of our problems is the large number of big pages and
> > the
> > extensive number of include files (more than 100 includes in some pages).
> > We
> > think that they produce heap fragmentation in the ASP template cache that
> > materializes in memory lacks.
> >
> > We have more than 2 months analyzing the symptoms, and we obtained some
> > interesting stuff, but the advise of the experts will help us a lot
> > confirming our theories or giving us new leads.
> >
> > Can I paste in a next message either dump files or IIS diagnostics HTML
> > reports for your opinion?
> >
> > Your help will be really appreciated.
> >
> > Thanks a lot,
> >
> > --
> > Juan
>
>
>