[watchdog] combine nmi_watchdog and softlockup
in [Kernel]
|
Prev: sys_unshare: simplify the not-really-implemented CLONE_THREAD/SIGHAND/VM code
Next: FS: libfs, implement simple_write_to_buffer
From: Frederic Weisbecker on 21 Apr 2010 13:30 Some minor things: On Tue, Apr 20, 2010 at 11:23:58AM -0400, Don Zickus wrote: > +#ifdef CONFIG_PERF_EVENTS_NMI > +struct perf_event_attr wd_hw_attr = { > + .type = PERF_TYPE_HARDWARE, > + .config = PERF_COUNT_HW_CPU_CYCLES, > + .size = sizeof(struct perf_event_attr), > + .pinned = 1, > + .disabled = 1, > +}; Shouldn't it be static? > + > +/* Callback function for perf event subsystem */ > +void watchdog_overflow_callback(struct perf_event *event, int nmi, > + struct perf_sample_data *data, > + struct pt_regs *regs) > +{ > + int this_cpu = smp_processor_id(); > + unsigned long touch_ts = per_cpu(watchdog_touch_ts, this_cpu); > + char warn = per_cpu(watchdog_warn, this_cpu); You can use __get_cpu_var() here > + > + if (touch_ts == 0) { > + __touch_watchdog(); > + return; > + } > + > + /* check for a hardlockup > + * This is done by making sure our timer interrupt > + * is incrementing. The timer interrupt should have > + * fired multiple times before we overflow'd. If it hasn't > + * then this is a good indication the cpu is stuck > + */ > + if (is_hardlockup(this_cpu)) { > + /* only print hardlockups once */ > + if (warn & HARDLOCKUP) > + return; > + > + if (hardlockup_panic) > + panic("Watchdog detected hard LOCKUP on cpu %d", this_cpu); > + else > + WARN(1, "Watchdog detected hard LOCKUP on cpu %d", this_cpu); > + > + per_cpu(watchdog_warn, this_cpu) = warn | HARDLOCKUP; and here. > + return; > + } > + > + per_cpu(watchdog_warn, this_cpu) = warn & ~HARDLOCKUP; > + return; > +} > +static void watchdog_interrupt_count(void) > +{ > + __get_cpu_var(hrtimer_interrupts)++; > +} > +#else > +static void watchdog_interrupt_count(void) { return; } Off case should be inline (although gcc will probably inline it by itself) > +#endif /* CONFIG_PERF_EVENTS_NMI */ > + > +/* watchdog kicker functions */ > +static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) > +{ > + int this_cpu = smp_processor_id(); > + unsigned long touch_ts = per_cpu(watchdog_touch_ts, this_cpu); __get_cpu_var -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Don Zickus on 21 Apr 2010 14:00 On Wed, Apr 21, 2010 at 07:27:33PM +0200, Frederic Weisbecker wrote: > Some minor things: > > > On Tue, Apr 20, 2010 at 11:23:58AM -0400, Don Zickus wrote: > > +#ifdef CONFIG_PERF_EVENTS_NMI > > +struct perf_event_attr wd_hw_attr = { > > + .type = PERF_TYPE_HARDWARE, > > + .config = PERF_COUNT_HW_CPU_CYCLES, > > + .size = sizeof(struct perf_event_attr), > > + .pinned = 1, > > + .disabled = 1, > > +}; > > > > Shouldn't it be static? yes. thanks. > > > > + > > +/* Callback function for perf event subsystem */ > > +void watchdog_overflow_callback(struct perf_event *event, int nmi, > > + struct perf_sample_data *data, > > + struct pt_regs *regs) > > +{ > > + int this_cpu = smp_processor_id(); > > + unsigned long touch_ts = per_cpu(watchdog_touch_ts, this_cpu); > > + char warn = per_cpu(watchdog_warn, this_cpu); > > > > You can use __get_cpu_var() here well, I already have this_cpu and need it later, I figured I would just use it with per_cpu and save _get_cpu_var the work of re-running smp_processor_id(). > > > > > + > > + if (touch_ts == 0) { > > + __touch_watchdog(); > > + return; > > + } > > + > > + /* check for a hardlockup > > + * This is done by making sure our timer interrupt > > + * is incrementing. The timer interrupt should have > > + * fired multiple times before we overflow'd. If it hasn't > > + * then this is a good indication the cpu is stuck > > + */ > > + if (is_hardlockup(this_cpu)) { > > + /* only print hardlockups once */ > > + if (warn & HARDLOCKUP) > > + return; > > + > > + if (hardlockup_panic) > > + panic("Watchdog detected hard LOCKUP on cpu %d", this_cpu); > > + else > > + WARN(1, "Watchdog detected hard LOCKUP on cpu %d", this_cpu); > > + > > + per_cpu(watchdog_warn, this_cpu) = warn | HARDLOCKUP; > > > > and here. same arguement as above. > > > > > + return; > > + } > > + > > + per_cpu(watchdog_warn, this_cpu) = warn & ~HARDLOCKUP; > > + return; > > +} > > +static void watchdog_interrupt_count(void) > > +{ > > + __get_cpu_var(hrtimer_interrupts)++; > > +} > > +#else > > +static void watchdog_interrupt_count(void) { return; } > > > > Off case should be inline (although gcc will probably inline it by itself) yup. thanks. > > > > > +#endif /* CONFIG_PERF_EVENTS_NMI */ > > + > > +/* watchdog kicker functions */ > > +static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) > > +{ > > + int this_cpu = smp_processor_id(); > > + unsigned long touch_ts = per_cpu(watchdog_touch_ts, this_cpu); > > > > __get_cpu_var again same as above. Thanks for the review. Cheers, Don -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Frederic Weisbecker on 21 Apr 2010 16:30 On Wed, Apr 21, 2010 at 01:50:21PM -0400, Don Zickus wrote: > On Wed, Apr 21, 2010 at 07:27:33PM +0200, Frederic Weisbecker wrote: > > Some minor things: > > > > > > On Tue, Apr 20, 2010 at 11:23:58AM -0400, Don Zickus wrote: > > > +#ifdef CONFIG_PERF_EVENTS_NMI > > > +struct perf_event_attr wd_hw_attr = { > > > + .type = PERF_TYPE_HARDWARE, > > > + .config = PERF_COUNT_HW_CPU_CYCLES, > > > + .size = sizeof(struct perf_event_attr), > > > + .pinned = 1, > > > + .disabled = 1, > > > +}; > > > > > > > > Shouldn't it be static? > > yes. thanks. > > > > > > > > + > > > +/* Callback function for perf event subsystem */ > > > +void watchdog_overflow_callback(struct perf_event *event, int nmi, > > > + struct perf_sample_data *data, > > > + struct pt_regs *regs) > > > +{ > > > + int this_cpu = smp_processor_id(); > > > + unsigned long touch_ts = per_cpu(watchdog_touch_ts, this_cpu); > > > + char warn = per_cpu(watchdog_warn, this_cpu); > > > > > > > > You can use __get_cpu_var() here > > well, I already have this_cpu and need it later, I figured I would just > use it with per_cpu and save _get_cpu_var the work of re-running > smp_processor_id(). This is more about code clarity in fact. per_cpu() suggests we are fetching something from another cpu. This is a very minor issue though. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Don Zickus on 21 Apr 2010 16:50 On Wed, Apr 21, 2010 at 10:24:21PM +0200, Frederic Weisbecker wrote: > On Wed, Apr 21, 2010 at 01:50:21PM -0400, Don Zickus wrote: > > On Wed, Apr 21, 2010 at 07:27:33PM +0200, Frederic Weisbecker wrote: > > > Some minor things: > > > > > > > > > On Tue, Apr 20, 2010 at 11:23:58AM -0400, Don Zickus wrote: > > > > +#ifdef CONFIG_PERF_EVENTS_NMI > > > > +struct perf_event_attr wd_hw_attr = { > > > > + .type = PERF_TYPE_HARDWARE, > > > > + .config = PERF_COUNT_HW_CPU_CYCLES, > > > > + .size = sizeof(struct perf_event_attr), > > > > + .pinned = 1, > > > > + .disabled = 1, > > > > +}; > > > > > > > > > > > > Shouldn't it be static? > > > > yes. thanks. > > > > > > > > > > > > + > > > > +/* Callback function for perf event subsystem */ > > > > +void watchdog_overflow_callback(struct perf_event *event, int nmi, > > > > + struct perf_sample_data *data, > > > > + struct pt_regs *regs) > > > > +{ > > > > + int this_cpu = smp_processor_id(); > > > > + unsigned long touch_ts = per_cpu(watchdog_touch_ts, this_cpu); > > > > + char warn = per_cpu(watchdog_warn, this_cpu); > > > > > > > > > > > > You can use __get_cpu_var() here > > > > well, I already have this_cpu and need it later, I figured I would just > > use it with per_cpu and save _get_cpu_var the work of re-running > > smp_processor_id(). > > > > This is more about code clarity in fact. per_cpu() suggests we are > fetching something from another cpu. ah gotcha. > > This is a very minor issue though. I have no problem changing it. Cheers, Don -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Frederic Weisbecker on 28 Apr 2010 08:40
On Fri, Apr 23, 2010 at 12:13:29PM -0400, Don Zickus wrote: > +void watchdog_overflow_callback(struct perf_event *event, int nmi, > + struct perf_sample_data *data, > + struct pt_regs *regs) > +{ > + int this_cpu = smp_processor_id(); > + unsigned long touch_ts = per_cpu(watchdog_touch_ts, this_cpu); > + char warn = __get_cpu_var(watchdog_warn); > + > + if (touch_ts == 0) { > + __touch_watchdog(); > + return; > + } > + > + /* check for a hardlockup > + * This is done by making sure our timer interrupt > + * is incrementing. The timer interrupt should have > + * fired multiple times before we overflow'd. If it hasn't > + * then this is a good indication the cpu is stuck > + */ > + if (is_hardlockup(this_cpu)) { > + /* only print hardlockups once */ > + if (warn & HARDLOCKUP) > + return; > + > + if (hardlockup_panic) > + panic("Watchdog detected hard LOCKUP on cpu %d", this_cpu); > + else > + WARN(1, "Watchdog detected hard LOCKUP on cpu %d", this_cpu); > + > + __get_cpu_var(watchdog_warn) = warn | HARDLOCKUP; > + return; > + } > + > + __get_cpu_var(watchdog_warn) = warn & ~HARDLOCKUP; > + return; > +} [...] > +static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) > +{ > + int this_cpu = smp_processor_id(); > + unsigned long touch_ts = __get_cpu_var(watchdog_touch_ts); > + char warn = __get_cpu_var(watchdog_warn); > + struct pt_regs *regs = get_irq_regs(); > + int duration; > + > + /* kick the hardlockup detector */ > + watchdog_interrupt_count(); > + > + /* kick the softlockup detector */ > + wake_up_process(__get_cpu_var(softlockup_watchdog)); > + > + /* .. and repeat */ > + hrtimer_forward_now(hrtimer, ns_to_ktime(get_sample_period())); > + > + if (touch_ts == 0) { > + __touch_watchdog(); > + return HRTIMER_RESTART; > + } > + > + /* check for a softlockup > + * This is done by making sure a high priority task is > + * being scheduled. The task touches the watchdog to > + * indicate it is getting cpu time. If it hasn't then > + * this is a good indication some task is hogging the cpu > + */ > + duration = is_softlockup(touch_ts, this_cpu); > + if (unlikely(duration)) { > + /* only warn once */ > + if (warn & SOFTLOCKUP) > + return HRTIMER_RESTART; > + > + printk(KERN_ERR "BUG: soft lockup - CPU#%d stuck for %us! [%s:%d]\n", > + this_cpu, duration, > + current->comm, task_pid_nr(current)); > + print_modules(); > + print_irqtrace_events(current); > + if (regs) > + show_regs(regs); > + else > + dump_stack(); > + > + if (softlockup_panic) > + panic("softlockup: hung tasks"); > + __get_cpu_var(watchdog_warn) = warn | SOFTLOCKUP; > + } else > + __get_cpu_var(watchdog_warn) = warn & ~SOFTLOCKUP; Note these watchdog_warn modifications are racy against the same that happens with HARDLOCKUP. You might clear what did the nmi. The race is harmless enough that we don't care much I think, but that's why it would have make sense to separate watchdog_warn tracking space between both. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |