Prev: random: simplify fips mode
Next: sched: change nohz idle load balancing logic to push model
From: Frederic Weisbecker on 20 May 2010 06:10
On Thu, May 20, 2010 at 11:31:31AM +0200, Ingo Molnar wrote:
> - [ While it's still a long way off, if this trend continues
> we eventually might even be able to get rid of the
> /debug/tracing/ temporary debug API and get rid of
> the ugly in-kernel pretty-printing bits. This is
> good: it may make Andrew very happy for a change ;-)
>
> The main detail here to be careful of is that lots of
> people are fond of the simplicity of the
> /debug/tracing/ debug UI, so when we replace it we
> want to do it by keeping that simple workflow (or
> best by making it even simpler). I have a few ideas
> how to do this.



How? We can emulate the /debug/tracing result with something
like perf trace, still that won't replace the immediate
availability of the result of any trace, which makes it
valuable for any simplest workflows.




> Regarding performance and complexity, which is our main
> worry atm, fortunately there's work going on in that
> direction - please see PeterZ's recent string of patches
> on lkml:
>
> 4f41c01: perf/ftrace: Optimize perf/tracepoint interaction for single events
> a19d35c: perf: Optimize buffer placement by allocating buffers NUMA aware
> ef60777: perf: Optimize the perf_output() path by removing IRQ-disables
> fa58815: perf: Optimize the hotpath by converting the perf output buffer to local_t



I would like to highlight the following commit too that _totally_
changes the requirements of our next common ring buffer, whatever it is:

c792061: perf: Disallow mmap() on per-task inherited events

Now we only need to care about local contention, we have removed the
support for buffers that contend across cpus in a single process.

Do I understand it right?



> 3) Add the function-tracer and function-graph tracer
> as an event and integrate it into perf.
>
> This will live-test the efficiency of the unification
> and brings over the last big ftrace plugin to perf.


I may start to take care of this soon.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar on 20 May 2010 07:40

* Frederic Weisbecker <fweisbec(a)gmail.com> wrote:

> On Thu, May 20, 2010 at 11:31:31AM +0200, Ingo Molnar wrote:
> > - [ While it's still a long way off, if this trend continues
> > we eventually might even be able to get rid of the
> > /debug/tracing/ temporary debug API and get rid of
> > the ugly in-kernel pretty-printing bits. This is
> > good: it may make Andrew very happy for a change ;-)
> >
> > The main detail here to be careful of is that lots of
> > people are fond of the simplicity of the
> > /debug/tracing/ debug UI, so when we replace it we
> > want to do it by keeping that simple workflow (or
> > best by making it even simpler). I have a few ideas
> > how to do this.
>
> How? We can emulate the /debug/tracing result with
> something like perf trace, still that won't replace the
> immediate availability of the result of any trace, which
> makes it valuable for any simplest workflows.

Firstly, one thing is sure: until there's no full
replacement we obviously dont want to phase out
/sys/kernel/debug/tracing. This was more of a 'our future'
email (as i see it), the process that will lead to solve
some of our more strategic problems in tracing land.

Regarding the issues you raised, there are several
solutions that dont need /sys/kernel/debug/tracing but
still support the very useful and usable 'immediate
tracing' workflow that ftrace prototyped. We can have a
combination of several things:

- Have a simple 'ftrace' command aliased to perf trace.

Means less typing, and it also allows a much more
finegrained tracing workflow: per user and per task/job
workflows, instead of the global/exclusive tracing mode
that /sys/kernel/debug/tracing. There would be ready
equivalents:

ftrace --available-tracers
ftrace --current-tracer
ftrace --start
ftrace --stop

... etc ...

- Immediate availability of a trace: persistent events
combined with roundrobin ('flight-recorder') recording
would solve this.

If events are active then if type 'ftrace' you get the
current trace. Simple to scrip and simple to use - no
need to have access to /sys/kernel/debug/tracing, also
can evidently be turned into a per user facility,
supports multiple plugins active at once, etc.

- For lightweight embedded tracing there are two separate
solutions that would work:

- we already have perf 'minimal builds' (when certain
libraries are not available), we could push that
concept some more to create a 'lightweight' command
that embedded systems can run just fine.

- extend our proxy recording and proxy execution/analysis
concepts. We can already run a perf trace recording
through a pipe and netcat, and we have perf archive
and cross-arch analysis code.

If there's interest, then this could be made more
convenient and the functionality around this could
be collected into a handy proxy tool:

ftrace --proxy smallbox --start
ftrace --proxy smallbox --stop
ftrace --proxy smallbox # prints trace

... etc ...

Thus the recording is done on the small box, while
all the analysis (and even all the commands) is
typed/executed on the bigger box.

So there are lots of possibilities - and there are other
options as well.

Does this address your worries?

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar on 20 May 2010 07:50

* Ingo Molnar <mingo(a)elte.hu> wrote:

> > as long as the tool is built in a way that it does not
> > need updates when we add trace points or other
> > functionality to the kernel.
>
> Yeah, most definitely. The sysfs event_source class will
> ensure that whatever (new) events are available
> propagate through the tool and are available to it.

Note that this approach isnt just for tools/perf/, it will
be useful for other perf events based tools as well:
SysProf and for the libpfm based apps.

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Frederic Weisbecker on 20 May 2010 09:10
On Thu, May 20, 2010 at 01:36:15PM +0200, Ingo Molnar wrote:
>
> * Frederic Weisbecker <fweisbec(a)gmail.com> wrote:
>
> > On Thu, May 20, 2010 at 11:31:31AM +0200, Ingo Molnar wrote:
> > > - [ While it's still a long way off, if this trend continues
> > > we eventually might even be able to get rid of the
> > > /debug/tracing/ temporary debug API and get rid of
> > > the ugly in-kernel pretty-printing bits. This is
> > > good: it may make Andrew very happy for a change ;-)
> > >
> > > The main detail here to be careful of is that lots of
> > > people are fond of the simplicity of the
> > > /debug/tracing/ debug UI, so when we replace it we
> > > want to do it by keeping that simple workflow (or
> > > best by making it even simpler). I have a few ideas
> > > how to do this.
> >
> > How? We can emulate the /debug/tracing result with
> > something like perf trace, still that won't replace the
> > immediate availability of the result of any trace, which
> > makes it valuable for any simplest workflows.
>
> Firstly, one thing is sure: until there's no full
> replacement we obviously dont want to phase out
> /sys/kernel/debug/tracing. This was more of a 'our future'
> email (as i see it), the process that will lead to solve
> some of our more strategic problems in tracing land.



Yeah.




> Regarding the issues you raised, there are several
> solutions that dont need /sys/kernel/debug/tracing but
> still support the very useful and usable 'immediate
> tracing' workflow that ftrace prototyped. We can have a
> combination of several things:
>
> - Have a simple 'ftrace' command aliased to perf trace.
>
> Means less typing, and it also allows a much more
> finegrained tracing workflow: per user and per task/job
> workflows, instead of the global/exclusive tracing mode
> that /sys/kernel/debug/tracing. There would be ready
> equivalents:
>
> ftrace --available-tracers
> ftrace --current-tracer
> ftrace --start
> ftrace --stop
>
> ... etc ...



We should probably more talk about events there, but yeah
I get your point.




>
> - Immediate availability of a trace: persistent events
> combined with roundrobin ('flight-recorder') recording
> would solve this.
>
> If events are active then if type 'ftrace' you get the
> current trace. Simple to scrip and simple to use - no
> need to have access to /sys/kernel/debug/tracing, also
> can evidently be turned into a per user facility,
> supports multiple plugins active at once, etc.



Yep.




>
> - For lightweight embedded tracing there are two separate
> solutions that would work:
>
> - we already have perf 'minimal builds' (when certain
> libraries are not available), we could push that
> concept some more to create a 'lightweight' command
> that embedded systems can run just fine.



Yeah that need to be one, very simple, dedicated to tracing.
With the very minimal requirements about dependencies.




> - extend our proxy recording and proxy execution/analysis
> concepts. We can already run a perf trace recording
> through a pipe and netcat, and we have perf archive
> and cross-arch analysis code.
>
> If there's interest, then this could be made more
> convenient and the functionality around this could
> be collected into a handy proxy tool:
>
> ftrace --proxy smallbox --start
> ftrace --proxy smallbox --stop
> ftrace --proxy smallbox # prints trace
>
> ... etc ...
>
> Thus the recording is done on the small box, while
> all the analysis (and even all the commands) is
> typed/executed on the bigger box.



You mean a client/server control? Controlling the recorder
with remote commands?


> So there are lots of possibilities - and there are other
> options as well.
>
> Does this address your worries?


My worries were more about bringing dependencies to read or launch
traces than the possibility to produce a very easy to use tool, even
easier and more powerful than the debugfs interface.

To sum up, my worries were (and sort of still are a bit): how painful
would that be to set up such a tool on a hard embedded box, compared
to the directly available stream we have.

But Thomas opinion makes me reconsider somehow the situation.

Once we have such tool and it becomes truly easy and intuitive,
people may migrate to it. At this point, it's possible the situation
have evolved enough that it will become obvious to deprecate the
debugfs interface.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Steven Rostedt on 20 May 2010 11:10
On Thu, 2010-05-20 at 10:38 -0400, Steven Rostedt wrote:
> n this machine:
>
> model name : Intel(R) Core(TM)2 Quad CPU Q9450 @ 2.66GHz
>
> It's a 2x2 (4 cores)
>

Oh, and this was done with 2.6.34 freshly checked out.

-- Steve

> With running:
>
> # modprobe ring_buffer_benchmark producer_fifo=10
> # sleep 30
> # cat /debug/tracing/trace
>



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
 |  Next  |  Last
Pages: 1 2
Prev: random: simplify fips mode
Next: sched: change nohz idle load balancing logic to push model