Prev: Solaris 9 tcpAttemptFails
Next: showrev -p broke after Solaris 10 upgrade
From: yuvalyuval on 5 Mar 2007 03:45

Hi,

I run a database application on Solaris 10 and see very high values in
the sys column of vmstat and mpstat.

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys
wt idl
0 0 0 117 529 447 392 23 55 1860 0 286 3 97
0 0
1 0 0 17 95 7 453 33 73 1106 0 317 2 98
0 0
2 0 0 23 97 10 342 36 54 899 0 172 3 97
0 0
3 0 0 289 3207 184 272 30 65 1894 0 106 1 99
0 0
16 0 0 27 108 5 834 52 61 1515 0 750 6 94
0 0
17 0 0 96 328 253 222 24 52 1131 0 131 1 99
0 0
18 0 0 88 339 236 426 44 61 947 0 262 2 98
0 0
19 0 0 15 1889 5 283 30 56 2771 0 138 2 98
0 0

vmstat 3 10
kthr memory page disk faults
cpu
r b w swap free re mf pi po fr de sr m0 m1 m2 m5 in sy cs
us sy id
69 0 0 54830576 13610072 0 29 4 0 0 0 0 0 0 0 0 3976 2001 2253
2 98 0
45 0 0 55141872 13610064 0 0 0 0 0 0 0 0 0 0 0 4308 1852 2327
2 98 0

How can I check what the CPUs are doing that causes this high sys
value?

Thanks,

Yuval

From: Noel R. Nihill on 5 Mar 2007 07:09
On Mar 5, 8:45 am, "yuvalyuval" <yuvalyuv...(a)gmail.com> wrote:
> Hi,
>
> I run a database application on Solaris 10 and see very high values in
> the sys column of vmstat and mpstat.
>
> CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys
> wt idl
> 0 0 0 117 529 447 392 23 55 1860 0 286 3 97
> 0 0
> 1 0 0 17 95 7 453 33 73 1106 0 317 2 98
> 0 0
> 2 0 0 23 97 10 342 36 54 899 0 172 3 97
> 0 0
> 3 0 0 289 3207 184 272 30 65 1894 0 106 1 99
> 0 0
> 16 0 0 27 108 5 834 52 61 1515 0 750 6 94
> 0 0
> 17 0 0 96 328 253 222 24 52 1131 0 131 1 99
> 0 0
> 18 0 0 88 339 236 426 44 61 947 0 262 2 98
> 0 0
> 19 0 0 15 1889 5 283 30 56 2771 0 138 2 98
> 0 0
>
> vmstat 3 10
> kthr memory page disk faults
> cpu
> r b w swap free re mf pi po fr de sr m0 m1 m2 m5 in sy cs
> us sy id
> 69 0 0 54830576 13610072 0 29 4 0 0 0 0 0 0 0 0 3976 2001 2253
> 2 98 0
> 45 0 0 55141872 13610064 0 0 0 0 0 0 0 0 0 0 0 4308 1852 2327
> 2 98 0
>
> How can I check what the CPUs are doing that causes this high sys
> value?

Use "prstat -Lvm" to find which process's LWPs are spending time in
the
kernel, then use truss and pfiles to try to see what they are doing.

> Thanks,
>
> Yuval


From: yuvalyuval on 5 Mar 2007 07:22
On Mar 5, 4:09 am, "Noel R. Nihill" <Noel.Nih...(a)Vallent.com> wrote:
> On Mar 5, 8:45 am, "yuvalyuval" <yuvalyuv...(a)gmail.com> wrote:
>
>
>
>
>
> > Hi,
>
> > I run a database application on Solaris 10 and see very high values in
> > the sys column of vmstat and mpstat.
>
> > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys
> > wt idl
> > 0 0 0 117 529 447 392 23 55 1860 0 286 3 97
> > 0 0
> > 1 0 0 17 95 7 453 33 73 1106 0 317 2 98
> > 0 0
> > 2 0 0 23 97 10 342 36 54 899 0 172 3 97
> > 0 0
> > 3 0 0 289 3207 184 272 30 65 1894 0 106 1 99
> > 0 0
> > 16 0 0 27 108 5 834 52 61 1515 0 750 6 94
> > 0 0
> > 17 0 0 96 328 253 222 24 52 1131 0 131 1 99
> > 0 0
> > 18 0 0 88 339 236 426 44 61 947 0 262 2 98
> > 0 0
> > 19 0 0 15 1889 5 283 30 56 2771 0 138 2 98
> > 0 0
>
> > vmstat 3 10
> > kthr memory page disk faults
> > cpu
> > r b w swap free re mf pi po fr de sr m0 m1 m2 m5 in sy cs
> > us sy id
> > 69 0 0 54830576 13610072 0 29 4 0 0 0 0 0 0 0 0 3976 2001 2253
> > 2 98 0
> > 45 0 0 55141872 13610064 0 0 0 0 0 0 0 0 0 0 0 4308 1852 2327
> > 2 98 0
>
> > How can I check what the CPUs are doing that causes this high sys
> > value?
>
> Use "prstat -Lvm" to find which process's LWPs are spending time in
> the
> kernel, then use truss and pfiles to try to see what they are doing.
>
>
>
> > Thanks,
>
> > Yuval- Hide quoted text -
>
> - Show quoted text -- Hide quoted text -
>
> - Show quoted text -



Hi Noel,

Here is the output I see, anything interesting here (lwp_park)?

prstat -Lvm -
0.1 0.0 0.0 0.0 92 5.9 1.2 107 21 117 0 asiqsrv12/269
24726 sybase 1.1 0.1 0.0 0.0 0.0 93 4.9 1.2 94 22 102 0
asiqsrv12/301
24726 sybase 1.1 0.1 0.0 0.0 0.0 91 6.2 1.2 106 20 114 0
asiqsrv12/279
24726 sybase 1.1 0.1 0.0 0.0 0.0 92 5.3 1.1 98 20 106 0
asiqsrv12/261
24726 sybase 1.1 0.1 0.0 0.0 0.0 92 5.2 1.2 92 21 99 0
asiqsrv12/270
19431 root 0.8 0.4 0.0 0.0 0.0 0.0 99 0.1 215 0 3K 0 seosd/
1
24726 sybase 1.1 0.1 0.0 0.0 0.0 92 5.4 1.2 94 20 102 0
asiqsrv12/241
24726 sybase 1.1 0.1 0.0 0.0 0.0 92 5.5 1.1 96 20 104 0
asiqsrv12/251
24726 sybase 1.1 0.1 0.0 0.0 0.0 93 5.1 1.1 93 20 102 0
asiqsrv12/247
24726 sybase 1.0 0.1 0.0 0.0 0.0 92 5.3 1.1 100 19 110 0
asiqsrv12/291
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 5.2 1.1 96 19 105 0
asiqsrv12/266
27308 dsadm 0.5 0.7 0.0 0.0 0.0 0.0 98 0.4 18 12 1K 0
dsapi_slave/1
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 5.3 1.0 91 17 99 0
asiqsrv12/303
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 5.2 1.1 95 18 103 0
asiqsrv12/277
24726 sybase 1.0 0.1 0.0 0.0 0.0 92 5.7 1.1 99 19 108 0
asiqsrv12/264
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 5.3 1.0 99 17 108 0
asiqsrv12/278
24726 sybase 1.0 0.1 0.0 0.0 0.0 92 5.4 1.1 96 18 105 0
asiqsrv12/300
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 4.7 1.1 86 18 94 0
asiqsrv12/294
24726 sybase 1.0 0.1 0.0 0.0 0.0 92 5.5 1.0 96 18 105 0
asiqsrv12/288
24726 sybase 1.0 0.1 0.0 0.0 0.0 92 5.6 1.1 99 18 107 0
asiqsrv12/290
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 4.6 1.0 83 18 91 0
asiqsrv12/242
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 5.3 1.0 97 17 106 0
asiqsrv12/253
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 5.0 1.1 94 18 103 0
asiqsrv12/244
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 4.6 1.0 86 18 94 0
asiqsrv12/246
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 5.0 1.1 91 18 100 0
asiqsrv12/274
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 5.0 1.0 86 17 93 0
asiqsrv12/273
24726 sybase 1.0 0.1 0.0 0.0 0.0 93 4.9 1.0 86 17 93 0
asiqsrv12/272
24726 sybase 0.9 0.1 0.0 0.0 0.0 93 5.2 1.0 95 17 103 0
asiqsrv12/239
24726 sybase 0.9 0.1 0.0 0.0 0.0 93 5.1 1.1 95 18 103 0
asiqsrv12/283
24726 sybase 0.9 0.1 0.0 0.0 0.0 93 4.8 1.0 92 18 100 0
asiqsrv12/254
24726 sybase 0.9 0.1 0.0 0.0 0.0 93 4.7 1.0 84 17 91 0
asiqsrv12/281
24726 sybase 0.9 0.1 0.0 0.0 0.0 94 4.4 0.9 84 17 92 0
asiqsrv12/298
24726 sybase 0.9 0.1 0.0 0.0 0.0 93 5.3 1.0 90 17 97 0
asiqsrv12/282
24726 sybase 0.9 0.1 0.0 0.0 0.0 93 5.0 0.9 89 16 96 0
asiqsrv12/297
24726 sybase 0.9 0.1 0.0 0.0 0.0 94 4.4 0.9 86 16 95 0
asiqsrv12/295


Here is what I get with truss:

truss -afv all -Dd -c -o truss.out -p 2538

psargs: asiqsrv12 @/PM/db/params.cfg -x tcpip{port=2638} /PM/db/pm.db -
n
signals ------------
SIGPOLL 1249
total: 1249


syscall seconds calls errors
read .370 19933
write .030 944
open .005 112
close .005 140
time .010 696
stat .000 16
lseek .219 20620
getpid .001 102
fstat .001 110
times .017 35
ioctl .002 179
fdsync .002 8
fcntl .000 37
lwp_park 5.271 252130 1374
lwp_unpark 1.453 63005
putmsg .005 102
getcontext .000 12
setcontext .031 1261
sysconfig .000 2
yield .391 15860
lwp_sema_post .025 1285
lwp_sigmask .018 1421
lwp_mutex_wakeup .046 2085
pread 534.026 32134
pwrite 229.456 15597
pollsys .046 2554
door_info .003 102
door_call .002 102
lwp_mutex_timedlock .067 2257
lwp_sema_timedwait .041 1273
accept .003 37
shutdown .002 28
recv .044 2392 1177
recvfrom .000 9
send .166 4341 6
sendto .000 6
getsockname .000 37
setsockopt .000 37
-------- ------ ----
sys totals: 771.775 441001 2557
usr time: 176.409
elapsed: 130.480




From: Noel R. Nihill on 5 Mar 2007 10:00
On Mar 5, 12:22 pm, "yuvalyuval" <yuvalyuv...(a)gmail.com> wrote:
> On Mar 5, 4:09 am, "Noel R. Nihill" <Noel.Nih...(a)Vallent.com> wrote:
>
>
>
>
>
> > On Mar 5, 8:45 am, "yuvalyuval" <yuvalyuv...(a)gmail.com> wrote:
>
> > > Hi,
>
> > > I run a database application on Solaris 10 and see very high values in
> > > the sys column of vmstat and mpstat.
>
> > > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys
> > > wt idl
> > > 0 0 0 117 529 447 392 23 55 1860 0 286 3 97
> > > 0 0
> > > 1 0 0 17 95 7 453 33 73 1106 0 317 2 98
> > > 0 0
> > > 2 0 0 23 97 10 342 36 54 899 0 172 3 97
> > > 0 0
> > > 3 0 0 289 3207 184 272 30 65 1894 0 106 1 99
> > > 0 0
> > > 16 0 0 27 108 5 834 52 61 1515 0 750 6 94
> > > 0 0
> > > 17 0 0 96 328 253 222 24 52 1131 0 131 1 99
> > > 0 0
> > > 18 0 0 88 339 236 426 44 61 947 0 262 2 98
> > > 0 0
> > > 19 0 0 15 1889 5 283 30 56 2771 0 138 2 98
> > > 0 0
>
> > > vmstat 3 10
> > > kthr memory page disk faults
> > > cpu
> > > r b w swap free re mf pi po fr de sr m0 m1 m2 m5 in sy cs
> > > us sy id
> > > 69 0 0 54830576 13610072 0 29 4 0 0 0 0 0 0 0 0 3976 2001 2253
> > > 2 98 0
> > > 45 0 0 55141872 13610064 0 0 0 0 0 0 0 0 0 0 0 4308 1852 2327
> > > 2 98 0
>
> > > How can I check what the CPUs are doing that causes this high sys
> > > value?
>
> > Use "prstat -Lvm" to find which process's LWPs are spending time in
> > the
> > kernel, then use truss and pfiles to try to see what they are doing.
>
> > > Thanks,
>
> > > Yuval- Hide quoted text -
>
> > - Show quoted text -- Hide quoted text -
>
> > - Show quoted text -
>
> Hi Noel,
>
> Here is the output I see, anything interesting here (lwp_park)?

Not really - you have many sybase threads each doing a little bit
of reading. I'm thinking maybe the datafile layout is suboptimal.

What is your disk configuration?

 | 
Pages: 1
Prev: Solaris 9 tcpAttemptFails
Next: showrev -p broke after Solaris 10 upgrade