Prev: Solaris 10 installation and RAID config
Next: Anyone using ZFS on a Dell 1850 w/ LSI Megaraid?
From: neilsolent on 9 Sep 2009 07:55
Not sure about the space available to create a new /var.

> In addition, perhaps you should just reboot.  You'll need to do so anyway
> to get a new /var working (because of /var/adm/{u,w}tmp{x,}).
>
> It would be annoying to go through all that work only to find out that a
> plain reboot would have fixed the problem.

Yeah sure. If I reboot, from experience the system won't come up -
then I'm in a darker place :-)
I know a reboot will be needed, just wondered what I could do to make
the reboot work up front.

> Personally, I strongly suspect that there's still a file there that you
> haven't seen.

OK. I am not sure how else to explain this output (other than a
corrupt /var):

dsys01a # pwd
/var
dsys01a # df -k .
Filesystem kbytes used avail capacity Mounted on
/dev/md/dsk/d6 5161437 5155030 0 100% /var
dsys01a # du -sk *
12622 adm
1 audit
307 core
2 crash
18609 cron
29 dmi
14 dt
1 empty
1 inet
6 ld
1 ldap
42 log
32 lost+found
2 lp
698 mail
1 news
1 nfs
1 nis
2 ntp
138967 opt
283410 preserve
40 run
74698 sadm
121 saf
131 snmp
29 spool
4 statmon
109374 tmp
4 yp

From: hume.spamfilter on 9 Sep 2009 08:51
neilsolent <n(a)solenttechnology.co.uk> wrote:
> OK. I am not sure how else to explain this output (other than a
> corrupt /var):

Observe... and don't get hung up on the fact that I'm using a zvol, it'll
work the same way with plain (or SVM, like you're using) devices:

Den:/mnt# df -h .
Filesystem size used avail capacity Mounted on
/dev/zvol/dsk/rpool/test
1.9G 1.0G 900M 54% /mnt
Den:/mnt# ls -l
total 2098208
-rw------T 1 root root 1073741824 Sep 9 09:42 file
drwx------ 2 root root 8192 Sep 9 09:39 lost+found

Den:/mnt# tail -0f file > /dev/null &
[1] 26266

Den:/mnt# rm file

Den:/mnt# ls -l
total 16
drwx------ 2 root root 8192 Sep 9 09:39 lost+found

Den:/mnt# du -sk .
9 .

Den:/mnt# df -h .
Filesystem size used avail capacity Mounted on
/dev/zvol/dsk/rpool/test
1.9G 1.0G 900M 54% /mnt

^^^^--- See?

Den:/mnt# lsof +D /mnt
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tcsh 26169 root cwd VDIR 256,3 512 2 /mnt
tcsh 26229 root cwd VDIR 256,3 512 2 /mnt
tail 26266 root cwd VDIR 256,3 512 2 /mnt
lsof 26279 root cwd VDIR 256,3 512 2 /mnt
lsof 26280 root cwd VDIR 256,3 512 2 /mnt


Den:/mnt# lsof | grep /mnt/
Den:/mnt#
Den:/mnt# ls -lL /dev/zvol/dsk/rpool/test
brw------- 1 root sys 256, 3 Sep 9 09:38 /dev/zvol/dsk/rpool/test

Den:/mnt# lsof | grep '256,3' | grep -v VDIR
tail 26266 root 0r VREG 256,3 1073741824 4 /mnt (/dev/zvol/dsk/rpool/test)

.... So I see that "tail" is holding open a regular file (VREG) of 1G in size
on /mnt.

Den:/mnt# ps -fp 26266
UID PID PPID C STIME TTY TIME CMD
root 26266 26229 0 09:43:30 pts/4 0:00 tail -0f file
Den:/mnt# kill 26266

Den:/mnt# df -h .
Filesystem size used avail capacity Mounted on
/dev/zvol/dsk/rpool/test
1.9G 2.0M 1.9G 1% /mnt


And I have my space back.

I think the main problem here is that lsof/pfiles aren't showing the
missing file name like I thought they would. That's my mistake.

--
Brandon Hume - hume -> BOFH.Ca, http://WWW.BOFH.Ca/
From: hume.spamfilter on 9 Sep 2009 08:52
neilsolent <n(a)solenttechnology.co.uk> wrote:
> /dev/md/dsk/d6 5161437 5155030 0 100% /var

Can you do an 'ls -lL /dev/md/dsk/d6' for me, please?

--
Brandon Hume - hume -> BOFH.Ca, http://WWW.BOFH.Ca/
From: Casper H.S. Dik on 9 Sep 2009 08:58
neilsolent <n(a)solenttechnology.co.uk> writes:

>> You've got some process holding a deleted file open, most likely. =A0Use
>> pfiles or lsof, and look for a file that is supposedly open but doesn't
>> exist.
>>
>> First thing to try is to HUP syslogd, of course.
>>
>> > Questions are: is it possible/worthwhile to recreate /var afresh, and
>> > how could this be done?
>>
>> It would be an extreme hassle, and you'd need to reboot. =A0Check for ope=
>n,
>> deleted files first.

>I ran the following, and it lists a few files of course, but nothing
>that has been supposedly deleted and nothing large:

Are there any messages in //var/adm/messages or in the kernel; what
does "fsck -n" report on /var. (free blocks, fragments).

Have you tried:

find /proc/ -type f -links 0

better than lsof as it is guaranteed to work.

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
From: neilsolent on 9 Sep 2009 11:18

> Can you do an 'ls -lL /dev/md/dsk/d6'  for me, please?

dsys01a # ls -lL /dev/md/dsk/d6
brw-r----- 1 root sys 85, 6 Jun 5 2004 /dev/md/dsk/d6


First  |  Prev  |  Next  |  Last
Pages: 1 2 3 4
Prev: Solaris 10 installation and RAID config
Next: Anyone using ZFS on a Dell 1850 w/ LSI Megaraid?