Prev: RapidIO Kernel Module?
Next: [PATCH] cgroup: alloc_css_id() increments hierarchy depth
From: Peter Zijlstra on 31 May 2010 10:50
On Mon, 2010-05-31 at 16:36 +0200, Frederic Weisbecker wrote:
> On Mon, May 31, 2010 at 10:54:59AM +0200, Peter Zijlstra wrote:
> > On Mon, 2010-05-31 at 10:12 +0200, Peter Zijlstra wrote:
> > > On Mon, 2010-05-31 at 10:00 +0200, Ingo Molnar wrote:
> > > > >
> > > > > NAK, aside from a few corner cases wakeup and sleep are the important
> > > > > points.
> > > > >
> > > > > The activate and deactivate functions are implementation details.
> > > >
> > > > Frederic, can you show us a concrete example of where we dont know what is
> > > > going on due to inadequate instrumentation? Can we fix that be extending the
> > > > existing tracepoints?
> > >
> > > Right, so a few of those corner cases I mentioned above are things like
> > > re-nice, PI-boosts etc.. Those use deactivate, modify task-state,
> > > activate cycles. so if you want to see those, we can add an explicit
> > > tracepoint for those actions.
> > >
> > > An explicit nice/PI-boost tracepoint is much clearer than trying to
> > > figure out wth the deactivate/activate cycle was for.
> >
> > Another advantage of explicit tracepoints is that you'd see them even
> > for non-running tasks, because we only do the deactivate/activate thingy
> > for runnable tasks.
>
>
> Yeah. So I agree with you that activate/deactivate are too much
> implementation related, they even don't give much sense as we
> don't know the cause of the event, could be a simple renice, or
> could be a sleep.
>
> So agreed, this sucks.
>
> For the corner cases like re-nice and PI-boost or so, we can indeed plug
> some higher level tracepoints there.
>
> But there is one more important problem these tracepoints were solving and
> that still need something:
>
> We don't know when a task goes to sleep. We have two wait tracepoints,
> sched_wait_task() to wait for a task to unschedule, and sched_process_wait()
> that is a hooks for waitid and wait4 syscalls. So we are missing all
> the event waiting from inside the kernel. But even with that, wait and sleep
> doesn't mean the same thing. Sleeping don't always involve using the waiting
> API.
>
> I think we need such tracepoint:
>
> diff --git a/kernel/sched.c b/kernel/sched.c
> index 8c0b90d..5f67c04 100644
> --- a/kernel/sched.c
> +++ b/kernel/sched.c
> @@ -3628,8 +3628,10 @@ need_resched_nonpreemptible:
> if (prev->state && !(preempt_count() & PREEMPT_ACTIVE)) {
> if (unlikely(signal_pending_state(prev->state, prev)))
> prev->state = TASK_RUNNING;
> - else
> + else {
> + trace_sched_task_sleep(prev);
> deactivate_task(rq, prev, DEQUEUE_SLEEP);
> + }
> switch_count = &prev->nvcsw;
> }

> And concerning the task waking up, if it is not migrated, it means it stays
> on its orig cpu. This is something that can be dealt from the post-processing.

Hurm,.. I was thinking trace_sched_switch(.prev_state != TASK_RUNNING)
would be enough, but its not for preemptible kernels.

Should we maybe cure this and rely on sched_switch() to detect sleeps?
It seems natural since only the current task can go to sleep, its just
that the whole preempt state gets a bit iffy.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Frederic Weisbecker on 31 May 2010 11:00
On Mon, May 31, 2010 at 04:43:33PM +0200, Peter Zijlstra wrote:
> On Mon, 2010-05-31 at 16:36 +0200, Frederic Weisbecker wrote:
> > On Mon, May 31, 2010 at 10:54:59AM +0200, Peter Zijlstra wrote:
> > > On Mon, 2010-05-31 at 10:12 +0200, Peter Zijlstra wrote:
> > > > On Mon, 2010-05-31 at 10:00 +0200, Ingo Molnar wrote:
> > > > > >
> > > > > > NAK, aside from a few corner cases wakeup and sleep are the important
> > > > > > points.
> > > > > >
> > > > > > The activate and deactivate functions are implementation details.
> > > > >
> > > > > Frederic, can you show us a concrete example of where we dont know what is
> > > > > going on due to inadequate instrumentation? Can we fix that be extending the
> > > > > existing tracepoints?
> > > >
> > > > Right, so a few of those corner cases I mentioned above are things like
> > > > re-nice, PI-boosts etc.. Those use deactivate, modify task-state,
> > > > activate cycles. so if you want to see those, we can add an explicit
> > > > tracepoint for those actions.
> > > >
> > > > An explicit nice/PI-boost tracepoint is much clearer than trying to
> > > > figure out wth the deactivate/activate cycle was for.
> > >
> > > Another advantage of explicit tracepoints is that you'd see them even
> > > for non-running tasks, because we only do the deactivate/activate thingy
> > > for runnable tasks.
> >
> >
> > Yeah. So I agree with you that activate/deactivate are too much
> > implementation related, they even don't give much sense as we
> > don't know the cause of the event, could be a simple renice, or
> > could be a sleep.
> >
> > So agreed, this sucks.
> >
> > For the corner cases like re-nice and PI-boost or so, we can indeed plug
> > some higher level tracepoints there.
> >
> > But there is one more important problem these tracepoints were solving and
> > that still need something:
> >
> > We don't know when a task goes to sleep. We have two wait tracepoints,
> > sched_wait_task() to wait for a task to unschedule, and sched_process_wait()
> > that is a hooks for waitid and wait4 syscalls. So we are missing all
> > the event waiting from inside the kernel. But even with that, wait and sleep
> > doesn't mean the same thing. Sleeping don't always involve using the waiting
> > API.
> >
> > I think we need such tracepoint:
> >
> > diff --git a/kernel/sched.c b/kernel/sched.c
> > index 8c0b90d..5f67c04 100644
> > --- a/kernel/sched.c
> > +++ b/kernel/sched.c
> > @@ -3628,8 +3628,10 @@ need_resched_nonpreemptible:
> > if (prev->state && !(preempt_count() & PREEMPT_ACTIVE)) {
> > if (unlikely(signal_pending_state(prev->state, prev)))
> > prev->state = TASK_RUNNING;
> > - else
> > + else {
> > + trace_sched_task_sleep(prev);
> > deactivate_task(rq, prev, DEQUEUE_SLEEP);
> > + }
> > switch_count = &prev->nvcsw;
> > }
>
> > And concerning the task waking up, if it is not migrated, it means it stays
> > on its orig cpu. This is something that can be dealt from the post-processing.
>
> Hurm,.. I was thinking trace_sched_switch(.prev_state != TASK_RUNNING)
> would be enough, but its not for preemptible kernels.
>
> Should we maybe cure this and rely on sched_switch() to detect sleeps?
> It seems natural since only the current task can go to sleep, its just
> that the whole preempt state gets a bit iffy.


Sounds good, we have the preempt depth in the common tracepoint headers, I'll
try to rebuild a reliable cpu runqueue from post-processing and see if all that
is enough.

Thanks.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Peter Zijlstra on 31 May 2010 12:20
On Mon, 2010-05-31 at 16:48 +0200, Frederic Weisbecker wrote:
> > Should we maybe cure this and rely on sched_switch() to detect sleeps?
> > It seems natural since only the current task can go to sleep, its just
> > that the whole preempt state gets a bit iffy.

How about something like the below?

Steve, is that proper usage of CREATE_TRACE_POINT?

---
Subject: sched, trace: Fix sched_switch() prev_state argument
From: Peter Zijlstra <a.p.zijlstra(a)chello.nl>
Date: Mon May 31 18:13:25 CEST 2010

For CONFIG_PREEMPT=y kernels the sched_switch(.prev_state) argument
isn't useful because we can get preempted with current->state !=
TASK_RUNNING without actually getting removed from the runqueue.

Cure this by treating all preempted tasks as runnable from the
tracer's point of view.

Signed-off-by: Peter Zijlstra <a.p.zijlstra(a)chello.nl>
---
include/trace/events/sched.h | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)

Index: linux-2.6/include/trace/events/sched.h
===================================================================
--- linux-2.6.orig/include/trace/events/sched.h
+++ linux-2.6/include/trace/events/sched.h
@@ -115,6 +115,23 @@ DEFINE_EVENT(sched_wakeup_template, sche
TP_PROTO(struct task_struct *p, int success),
TP_ARGS(p, success));

+#ifdef CREATE_TRACE_POINTS
+static inline long __trace_sched_switch_state(struct task_struct *p)
+{
+ long state = p->state;
+
+#ifdef CONFIG_PREEMPT
+ /*
+ * For all intents and purposes a preempted task is a running task.
+ */
+ if (task_thread_info(p)->preempt_count & PREEMPT_ACTIVE)
+ state = TASK_RUNNING;
+#endif
+
+ return state;
+}
+#endif
+
/*
* Tracepoint for task switches, performed by the scheduler:
*/
@@ -139,7 +156,7 @@ TRACE_EVENT(sched_switch,
memcpy(__entry->next_comm, next->comm, TASK_COMM_LEN);
__entry->prev_pid = prev->pid;
__entry->prev_prio = prev->prio;
- __entry->prev_state = prev->state;
+ __entry->prev_state = __trace_sched_switch_state(prev);
memcpy(__entry->prev_comm, prev->comm, TASK_COMM_LEN);
__entry->next_pid = next->pid;
__entry->next_prio = next->prio;

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Steven Rostedt on 31 May 2010 12:40
Expect slow responses from me today. It's a US Holiday.


On Mon, 2010-05-31 at 18:18 +0200, Peter Zijlstra wrote:
> On Mon, 2010-05-31 at 16:48 +0200, Frederic Weisbecker wrote:
> > > Should we maybe cure this and rely on sched_switch() to detect sleeps?
> > > It seems natural since only the current task can go to sleep, its just
> > > that the whole preempt state gets a bit iffy.
>
> How about something like the below?
>
> Steve, is that proper usage of CREATE_TRACE_POINT?
>
> ---
> Subject: sched, trace: Fix sched_switch() prev_state argument
> From: Peter Zijlstra <a.p.zijlstra(a)chello.nl>
> Date: Mon May 31 18:13:25 CEST 2010
>
> For CONFIG_PREEMPT=y kernels the sched_switch(.prev_state) argument
> isn't useful because we can get preempted with current->state !=
> TASK_RUNNING without actually getting removed from the runqueue.
>
> Cure this by treating all preempted tasks as runnable from the
> tracer's point of view.
>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra(a)chello.nl>
> ---
> include/trace/events/sched.h | 19 ++++++++++++++++++-
> 1 file changed, 18 insertions(+), 1 deletion(-)
>
> Index: linux-2.6/include/trace/events/sched.h
> ===================================================================
> --- linux-2.6.orig/include/trace/events/sched.h
> +++ linux-2.6/include/trace/events/sched.h
> @@ -115,6 +115,23 @@ DEFINE_EVENT(sched_wakeup_template, sche
> TP_PROTO(struct task_struct *p, int success),
> TP_ARGS(p, success));
>
> +#ifdef CREATE_TRACE_POINTS

I guess this could work. I can't think of anything that would cause this
to fail. But this is not exactly what the CREATE_TRACE_POINTS macro was
for.

Maybe we could make a CREATE_UTIL_FUNCTIONS macro that the
define_trace.h can unset like it does with CREATE_TRACE_POINTS before
recursively including the trace headers.

Maybe I'm a bit paranoid, but I'm a little nervous to extend the
CREATE_TRACE_POINTS macro to be used within the header to create utility
functions, although, currently I don't think there's anything
technically wrong in doing so.

-- Steve

> +static inline long __trace_sched_switch_state(struct task_struct *p)
> +{
> + long state = p->state;
> +
> +#ifdef CONFIG_PREEMPT
> + /*
> + * For all intents and purposes a preempted task is a running task.
> + */
> + if (task_thread_info(p)->preempt_count & PREEMPT_ACTIVE)
> + state = TASK_RUNNING;
> +#endif
> +
> + return state;
> +}
> +#endif
> +
> /*
> * Tracepoint for task switches, performed by the scheduler:
> */
> @@ -139,7 +156,7 @@ TRACE_EVENT(sched_switch,
> memcpy(__entry->next_comm, next->comm, TASK_COMM_LEN);
> __entry->prev_pid = prev->pid;
> __entry->prev_prio = prev->prio;
> - __entry->prev_state = prev->state;
> + __entry->prev_state = __trace_sched_switch_state(prev);
> memcpy(__entry->prev_comm, prev->comm, TASK_COMM_LEN);
> __entry->next_pid = next->pid;
> __entry->next_prio = next->prio;
>


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Frederic Weisbecker on 31 May 2010 13:00
On Mon, May 31, 2010 at 06:18:35PM +0200, Peter Zijlstra wrote:
> On Mon, 2010-05-31 at 16:48 +0200, Frederic Weisbecker wrote:
> > > Should we maybe cure this and rely on sched_switch() to detect sleeps?
> > > It seems natural since only the current task can go to sleep, its just
> > > that the whole preempt state gets a bit iffy.
>
> How about something like the below?
>
> Steve, is that proper usage of CREATE_TRACE_POINT?
>
> ---
> Subject: sched, trace: Fix sched_switch() prev_state argument
> From: Peter Zijlstra <a.p.zijlstra(a)chello.nl>
> Date: Mon May 31 18:13:25 CEST 2010
>
> For CONFIG_PREEMPT=y kernels the sched_switch(.prev_state) argument
> isn't useful because we can get preempted with current->state !=
> TASK_RUNNING without actually getting removed from the runqueue.
>
> Cure this by treating all preempted tasks as runnable from the
> tracer's point of view.
>
> Signed-off-by: Peter Zijlstra <a.p.zijlstra(a)chello.nl>
> ---



Other than Steve's said, the thing looks good.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: RapidIO Kernel Module?
Next: [PATCH] cgroup: alloc_css_id() increments hierarchy depth