Linux 2.6.32-rc1
in [Kernel]
|
From: Linus Torvalds on 28 Sep 2009 11:50 On Mon, 28 Sep 2009, Eric Dumazet wrote: > Linus Torvalds a �crit : > > > > Go wild, test it out, and let us know about any regressions you find, > > Something seems wrong with process time accounting. > > Following program should consume 10*8 seconds of cpu on a 8 cpu machine, but > with 2.6.32-rc1 numbers are crazy. Ok, so the top-level process time looks sane _while_ the thing is running (sum of all threads), but the per-thread times look broken (as if they had randomly had the times of the other threads added into them - looks to me like they on average had half the other threads' time accounted into them). And then at the end, it looks like the time of the threads (which was over-accounted) get re-accounted back into the main process, so the final time is indeed wildly inflated. IOW, looks like we're adding thread times multiple times to other threads (and then finally to the parent). I'm adding the usual suspects to the cc, and leaving your results and test-program quoted here for them.. Thomas, Martin, John - if you're not the people I should blame for this, let me know. Linus --- > $ gcc -O2 -o process process.c -lpthread > $ ./process > PID TTY STAT TIME COMMAND > 5532 pts/1 - 0:08 ./process > - - Sl+ 0:00 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:09 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:01 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 0:16 ./process > - - Sl+ 0:00 - > - - Rl+ 0:14 - > - - Rl+ 0:06 - > - - Rl+ 0:14 - > - - Rl+ 0:10 - > - - Rl+ 0:06 - > - - Rl+ 0:06 - > - - Rl+ 0:10 - > - - Rl+ 0:02 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 0:24 ./process > - - Sl+ 0:00 - > - - Rl+ 0:20 - > - - Rl+ 0:11 - > - - Rl+ 0:24 - > - - Rl+ 0:15 - > - - Rl+ 0:07 - > - - Rl+ 0:07 - > - - Rl+ 0:24 - > - - Rl+ 0:03 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 0:32 ./process > - - Sl+ 0:00 - > - - Rl+ 0:29 - > - - Rl+ 0:12 - > - - Rl+ 0:29 - > - - Rl+ 0:21 - > - - Rl+ 0:08 - > - - Rl+ 0:08 - > - - Rl+ 0:29 - > - - Rl+ 0:04 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 0:40 ./process > - - Sl+ 0:00 - > - - Rl+ 0:35 - > - - Rl+ 0:13 - > - - Rl+ 0:35 - > - - Rl+ 0:26 - > - - Rl+ 0:09 - > - - Rl+ 0:09 - > - - Rl+ 0:35 - > - - Rl+ 0:05 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 0:48 ./process > - - Sl+ 0:00 - > - - Rl+ 0:40 - > - - Rl+ 0:14 - > - - Rl+ 0:40 - > - - Rl+ 0:31 - > - - Rl+ 0:10 - > - - Rl+ 0:10 - > - - Rl+ 0:40 - > - - Rl+ 0:06 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 0:56 ./process > - - Sl+ 0:00 - > - - Rl+ 0:45 - > - - Rl+ 0:15 - > - - Rl+ 0:45 - > - - Rl+ 0:37 - > - - Rl+ 0:11 - > - - Rl+ 0:11 - > - - Rl+ 0:45 - > - - Rl+ 0:07 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 1:05 ./process > - - Sl+ 0:00 - > - - Rl+ 0:51 - > - - Rl+ 0:21 - > - - Rl+ 0:55 - > - - Rl+ 0:46 - > - - Rl+ 0:12 - > - - Rl+ 0:12 - > - - Rl+ 0:59 - > - - Rl+ 0:08 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 1:13 ./process > - - Sl+ 0:00 - > - - Rl+ 0:56 - > - - Rl+ 0:22 - > - - Rl+ 1:00 - > - - Rl+ 0:52 - > - - Rl+ 0:13 - > - - Rl+ 0:13 - > - - Rl+ 1:04 - > - - Rl+ 0:09 - > PID TTY STAT TIME COMMAND > 5532 pts/1 - 5:13 ./process > - - S+ 5:13 - > > > $ cat process.c > #include <pthread.h> > #include <time.h> > #include <stdio.h> > > /* > * Burn cpu cycles for about 10 seconds > */ > static void *do_unit(void *arg) > { > time_t t0,t1; > > t0 = time(NULL); > do { > t1 = time(NULL); > } while (t1 - t0 <= 10); > return NULL; > } > > > > > int main(int argc, char *argv[]) > { > pthread_t tids[8]; > int i; > char cmd[128]; > > sprintf(cmd, "ps m -p %d", getpid()); > for (i = 0; i < 8 ; i++) > pthread_create(&tids[i], NULL, do_unit, NULL); > for (i = 0; i < 9; i++) { > sleep(1); > system(cmd); > } > for (i = 0; i < 8 ; i++) > pthread_join(tids[i], NULL); > system(cmd); > return 0; > } > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Martin Schwidefsky on 28 Sep 2009 13:20 On Mon, 28 Sep 2009 08:39:31 -0700 (PDT) Linus Torvalds <torvalds(a)linux-foundation.org> wrote: > On Mon, 28 Sep 2009, Eric Dumazet wrote: > > Linus Torvalds a écrit : > > > > > > Go wild, test it out, and let us know about any regressions you find, > > > > Something seems wrong with process time accounting. > > > > Following program should consume 10*8 seconds of cpu on a 8 cpu machine, but > > with 2.6.32-rc1 numbers are crazy. > > Ok, so the top-level process time looks sane _while_ the thing is running > (sum of all threads), but the per-thread times look broken (as if they had > randomly had the times of the other threads added into them - looks to me > like they on average had half the other threads' time accounted into > them). > > And then at the end, it looks like the time of the threads (which was > over-accounted) get re-accounted back into the main process, so the final > time is indeed wildly inflated. > > IOW, looks like we're adding thread times multiple times to other threads > (and then finally to the parent). > > I'm adding the usual suspects to the cc, and leaving your results and > test-program quoted here for them.. Thomas, Martin, John - if you're not > the people I should blame for this, let me know. Uh-oh.. usual suspects, eh? For starters I run the test program on an s390 with VIRT_CPU_ACCOUNTING=y: time ./burn-cpu PID TTY STAT TIME COMMAND 2979 pts/0 - 0:08 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:01 - - - Rl+ 0:01 - - - Rl+ 0:01 - - - Rl+ 0:01 - - - Rl+ 0:01 - - - Rl+ 0:01 - - - Rl+ 0:01 - - - Rl+ 0:01 - PID TTY STAT TIME COMMAND 2979 pts/0 - 0:16 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:02 - - - Rl+ 0:02 - - - Rl+ 0:02 - - - Rl+ 0:02 - - - Rl+ 0:02 - - - Rl+ 0:02 - - - Rl+ 0:02 - - - Rl+ 0:02 - PID TTY STAT TIME COMMAND 2979 pts/0 - 0:25 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:03 - - - Rl+ 0:03 - - - Rl+ 0:03 - - - Rl+ 0:03 - - - Rl+ 0:03 - - - Rl+ 0:03 - - - Rl+ 0:03 - - - Rl+ 0:03 - PID TTY STAT TIME COMMAND 2979 pts/0 - 0:33 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:04 - - - Rl+ 0:04 - - - Rl+ 0:04 - - - Rl+ 0:04 - - - Rl+ 0:04 - - - Rl+ 0:04 - - - Rl+ 0:04 - - - Rl+ 0:04 - PID TTY STAT TIME COMMAND 2979 pts/0 - 0:41 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:05 - - - Rl+ 0:05 - - - Rl+ 0:05 - - - Rl+ 0:05 - - - Rl+ 0:05 - - - Rl+ 0:05 - - - Rl+ 0:05 - - - Rl+ 0:05 - PID TTY STAT TIME COMMAND 2979 pts/0 - 0:50 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:06 - - - Rl+ 0:06 - - - Rl+ 0:06 - - - Rl+ 0:06 - - - Rl+ 0:06 - - - Rl+ 0:06 - - - Rl+ 0:06 - - - Rl+ 0:06 - PID TTY STAT TIME COMMAND 2979 pts/0 - 0:58 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:07 - - - Rl+ 0:07 - - - Rl+ 0:07 - - - Rl+ 0:07 - - - Rl+ 0:07 - - - Rl+ 0:07 - - - Rl+ 0:07 - - - Rl+ 0:07 - PID TTY STAT TIME COMMAND 2979 pts/0 - 1:07 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:08 - - - Rl+ 0:08 - - - Rl+ 0:08 - - - Rl+ 0:08 - - - Rl+ 0:08 - - - Rl+ 0:08 - - - Rl+ 0:08 - - - Rl+ 0:08 - PID TTY STAT TIME COMMAND 2979 pts/0 - 1:15 ./burn-cpu - - Sl+ 0:00 - - - Rl+ 0:09 - - - Rl+ 0:09 - - - Rl+ 0:09 - - - Rl+ 0:09 - - - Rl+ 0:09 - - - Rl+ 0:09 - - - Rl+ 0:09 - - - Rl+ 0:09 - PID TTY STAT TIME COMMAND 2979 pts/0 - 1:25 ./burn-cpu - - S+ 1:25 - real 0m10.708s user 1m25.051s sys 0m0.174s looks better, gives an average of 10.63 seconds per thread and the per thread numbers are fine. I'll see what happens with the test case on my pc(a)home. -- blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Eric Dumazet on 28 Sep 2009 14:50 Martin Schwidefsky a écrit : > On Mon, 28 Sep 2009 08:39:31 -0700 (PDT) > Linus Torvalds <torvalds(a)linux-foundation.org> wrote: > >> On Mon, 28 Sep 2009, Eric Dumazet wrote: >>> Linus Torvalds a écrit : >>>> Go wild, test it out, and let us know about any regressions you find, >>> Something seems wrong with process time accounting. >>> >>> Following program should consume 10*8 seconds of cpu on a 8 cpu machine, but >>> with 2.6.32-rc1 numbers are crazy. >> Ok, so the top-level process time looks sane _while_ the thing is running >> (sum of all threads), but the per-thread times look broken (as if they had >> randomly had the times of the other threads added into them - looks to me >> like they on average had half the other threads' time accounted into >> them). >> >> And then at the end, it looks like the time of the threads (which was >> over-accounted) get re-accounted back into the main process, so the final >> time is indeed wildly inflated. >> >> IOW, looks like we're adding thread times multiple times to other threads >> (and then finally to the parent). >> >> I'm adding the usual suspects to the cc, and leaving your results and >> test-program quoted here for them.. Thomas, Martin, John - if you're not >> the people I should blame for this, let me know. > > Uh-oh.. usual suspects, eh? > > For starters I run the test program on an s390 with > VIRT_CPU_ACCOUNTING=y: > > time ./burn-cpu > PID TTY STAT TIME COMMAND > 2979 pts/0 - 0:08 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:01 - > - - Rl+ 0:01 - > - - Rl+ 0:01 - > - - Rl+ 0:01 - > - - Rl+ 0:01 - > - - Rl+ 0:01 - > - - Rl+ 0:01 - > - - Rl+ 0:01 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 0:16 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:02 - > - - Rl+ 0:02 - > - - Rl+ 0:02 - > - - Rl+ 0:02 - > - - Rl+ 0:02 - > - - Rl+ 0:02 - > - - Rl+ 0:02 - > - - Rl+ 0:02 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 0:25 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:03 - > - - Rl+ 0:03 - > - - Rl+ 0:03 - > - - Rl+ 0:03 - > - - Rl+ 0:03 - > - - Rl+ 0:03 - > - - Rl+ 0:03 - > - - Rl+ 0:03 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 0:33 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:04 - > - - Rl+ 0:04 - > - - Rl+ 0:04 - > - - Rl+ 0:04 - > - - Rl+ 0:04 - > - - Rl+ 0:04 - > - - Rl+ 0:04 - > - - Rl+ 0:04 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 0:41 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > - - Rl+ 0:05 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 0:50 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:06 - > - - Rl+ 0:06 - > - - Rl+ 0:06 - > - - Rl+ 0:06 - > - - Rl+ 0:06 - > - - Rl+ 0:06 - > - - Rl+ 0:06 - > - - Rl+ 0:06 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 0:58 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:07 - > - - Rl+ 0:07 - > - - Rl+ 0:07 - > - - Rl+ 0:07 - > - - Rl+ 0:07 - > - - Rl+ 0:07 - > - - Rl+ 0:07 - > - - Rl+ 0:07 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 1:07 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:08 - > - - Rl+ 0:08 - > - - Rl+ 0:08 - > - - Rl+ 0:08 - > - - Rl+ 0:08 - > - - Rl+ 0:08 - > - - Rl+ 0:08 - > - - Rl+ 0:08 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 1:15 ./burn-cpu > - - Sl+ 0:00 - > - - Rl+ 0:09 - > - - Rl+ 0:09 - > - - Rl+ 0:09 - > - - Rl+ 0:09 - > - - Rl+ 0:09 - > - - Rl+ 0:09 - > - - Rl+ 0:09 - > - - Rl+ 0:09 - > PID TTY STAT TIME COMMAND > 2979 pts/0 - 1:25 ./burn-cpu > - - S+ 1:25 - > > real 0m10.708s > user 1m25.051s > sys 0m0.174s > > looks better, gives an average of 10.63 seconds per thread and the per > thread numbers are fine. I'll see what happens with the test case on my > pc(a)home. > I did a bisection and found commit def0a9b2573e00ab0b486cb5382625203ab4c4a6 was the origin of the problem on my x86_32 machine. def0a9b2573e00ab0b486cb5382625203ab4c4a6 is first bad commit commit def0a9b2573e00ab0b486cb5382625203ab4c4a6 Author: Peter Zijlstra <a.p.zijlstra(a)chello.nl> Date: Fri Sep 18 20:14:01 2009 +0200 sched_clock: Make it NMI safe Arjan complained about the suckyness of TSC on modern machines, and asked if we could do something about that for PERF_SAMPLE_TIME. Make cpu_clock() NMI safe by removing the spinlock and using cmpxchg. This also makes it smaller and more robust. Affects architectures that use HAVE_UNSTABLE_SCHED_CLOCK, i.e. IA64 and x86. Signed-off-by: Peter Zijlstra <a.p.zijlstra(a)chello.nl> LKML-Reference: <new-submission> Signed-off-by: Ingo Molnar <mingo(a)elte.hu> git bisect start # bad: [17d857be649a21ca90008c6dc425d849fa83db5c] Linux 2.6.32-rc1 git bisect bad 17d857be649a21ca90008c6dc425d849fa83db5c # good: [74fca6a42863ffacaf7ba6f1936a9f228950f657] Linux 2.6.31 git bisect good 74fca6a42863ffacaf7ba6f1936a9f228950f657 # good: [74fca6a42863ffacaf7ba6f1936a9f228950f657] Linux 2.6.31 git bisect good 74fca6a42863ffacaf7ba6f1936a9f228950f657 # good: [5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3] Staging: vt6656: Integrate vt6656 into build system. git bisect good 5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3 # good: [5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3] Staging: vt6656: Integrate vt6656 into build system. git bisect good 5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3 # bad: [c720f5655df159a630fa0290a0bd67c93e92b0bf] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-2.6 git bisect bad c720f5655df159a630fa0290a0bd67c93e92b0bf # good: [a03fdb7612874834d6847107198712d18b5242c7] Merge branch 'timers-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip git bisect good a03fdb7612874834d6847107198712d18b5242c7 # good: [3530c1886291df061e3972c55590777ef1cb67f8] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4 git bisect good 3530c1886291df061e3972c55590777ef1cb67f8 # good: [84d6ae431f315e8973aac3c3fe1d550fc9240ef3] V4L/DVB (13033): pt1: Don't use a deprecated DMA_BIT_MASK macro git bisect good 84d6ae431f315e8973aac3c3fe1d550fc9240ef3 # bad: [ebc79c4f8da0f92efa968e0328f32334a2ce80cf] Merge git://git.kernel.org/pub/scm/linux/kernel/git/jaswinder/linux-2.6 git bisect bad ebc79c4f8da0f92efa968e0328f32334a2ce80cf # good: [7bd032dc2793afcbaf4a350056768da84cdbd89b] USB serial: update the console driver git bisect good 7bd032dc2793afcbaf4a350056768da84cdbd89b # good: [1281a49b7aa865a1f0d60e2b28410e6234fc686b] perf trace: Sample timestamp and cpu when using record flag git bisect good 1281a49b7aa865a1f0d60e2b28410e6234fc686b # bad: [e11c675ede0d42a405ae595528bf0b29ce1ae56f] Merge git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty-2.6 git bisect bad e11c675ede0d42a405ae595528bf0b29ce1ae56f # bad: [6161352142d5fed4cd753b32e5ccde66e705b14e] tracing, perf: Convert the power tracer into an event tracer git bisect bad 6161352142d5fed4cd753b32e5ccde66e705b14e # bad: [929bf0d0156562ce631728b6fa53d68004d456d2] Merge branch 'linus' into perfcounters/core git bisect bad 929bf0d0156562ce631728b6fa53d68004d456d2 # good: [5622f295b53fb60dbf9bed3e2c89d182490a8b7f] x86, perf_counter, bts: Optimize BTS overflow handling git bisect good 5622f295b53fb60dbf9bed3e2c89d182490a8b7f # bad: [def0a9b2573e00ab0b486cb5382625203ab4c4a6] sched_clock: Make it NMI safe git bisect bad def0a9b2573e00ab0b486cb5382625203ab4c4a6 My .config file is attached to this mail
From: Martin Schwidefsky on 28 Sep 2009 17:00 On Mon, 28 Sep 2009 20:41:41 +0200 Eric Dumazet <eric.dumazet(a)gmail.com> wrote: > I did a bisection and found commit def0a9b2573e00ab0b486cb5382625203ab4c4a6 > was the origin of the problem on my x86_32 machine. > > def0a9b2573e00ab0b486cb5382625203ab4c4a6 is first bad commit > commit def0a9b2573e00ab0b486cb5382625203ab4c4a6 > Author: Peter Zijlstra <a.p.zijlstra(a)chello.nl> > Date: Fri Sep 18 20:14:01 2009 +0200 > > sched_clock: Make it NMI safe > > Arjan complained about the suckyness of TSC on modern machines, and > asked if we could do something about that for PERF_SAMPLE_TIME. > > Make cpu_clock() NMI safe by removing the spinlock and using > cmpxchg. This also makes it smaller and more robust. > > Affects architectures that use HAVE_UNSTABLE_SCHED_CLOCK, i.e. IA64 > and x86. > > Signed-off-by: Peter Zijlstra <a.p.zijlstra(a)chello.nl> > LKML-Reference: <new-submission> > Signed-off-by: Ingo Molnar <mingo(a)elte.hu> Confirmed. The bisect run on my machine gave me the same bad commit. The new logic in sched_clock_remove seems racy: the old code got the locks for the sched_clock_data of the local and the remove cpu before it changed any value. The new code tries to get to the same result with a single cmpxchg. Bad things happen if two cpus try to update the clock values crosswise, no? -- blue skies, Martin. "Reality continues to ruin my life." - Calvin. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: devzero on 28 Sep 2009 18:20
>and as usual that means that the merge window is all over and done with so, all the "heavy" stuff is in now - but no DRBD ? List: linux-kernel Subject: Linux 2.6.32-rc1 From: Linus Torvalds <torvalds () linux-foundation ! org> Date: 2009-09-27 22:34:53 Message-ID: alpine.LFD.2.01.0909271511190.3349 () localhost ! localdomain [Download message RAW] It's been two weeks (and then some - but with last week being LinuxCon and Plumbers conf I extended it by a few days), and as usual that means that the merge window is all over and done with. 2.6.32-rc1 is out, so give it a whirl. What can I say? 67% drivers (the bulk of which is from 'staging', but there's driver changes all over), 10% firmware, 10% arch updates (dominated by arm, but MIPS, POWER, SH and x86 updates are there too, along with the new 'SCore' architecture), 5% Documentation, and a random smattering of other things (ie the normal filesystem, kernel, networking etc updates). For a change, I don't think we have a single new filesystem this time around, but we do have updates to existing ones (ocfs2, btrfs, nfs, nilfs, xfs, gfs2, ext4 - you name it). Some of the more interesting changes (but perhaps that's just me) are some of the VM updates (ZERO_PAGE is back!) and the writeback work by Jens and others to spread out writeback by backing store. Go wild, test it out, and let us know about any regressions you find, Linus ______________________________________________________ GRATIS f�r alle WEB.DE-Nutzer: Die maxdome Movie-FLAT! Jetzt freischalten unter http://movieflat.web.de -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |