Prev: Simple hack to get $500 to your home
Next: what's the best way to detect cycle in time series?
From: Tim Wescott on 11 Jun 2010 15:31
On 06/11/2010 12:08 PM, Luna Moon wrote:
> On Jun 11, 2:23 pm, Walter Roberson<rober...(a)hushmail.com> wrote:
>> Luna Moon wrote:
>>> I have very spiky signals. Visually it's very easy to detect spikes,
>>
>>> but how do I detect spikes in a computerized appraoch?
>>
>> Visually it isn't easy to detect spikes, because there is no fixed definition
>> for when a "spike" becomes a "peak" or when multiple local maxima close
>> together are multiple spikes, vs regular noise on the system response to a
>> spike, vs regular noise on a spike that is out of phase, vs regular noise on a
>> peak, vs regular noise on a peak that is out of phase, vs regular noise on a
>> peak that is broadened because the sampling rate is not high enough, vs ...
>>
>> If you have an out-of-phase peak and a spike close to it that goes higher, or
>> noise on the peak measurement that takes the reading higher, then many humans
>> would place the peak maximum at the higher point instead of recognizing that
>> the energy absolute magnitude would be higher if you were to do phase
>> correction for the smaller peak.
>>
>> These difficulties for humans (which are inherent in the vague meaning of
>> "spike") make it difficult to write a robust spike detection algorithm. Did
>> the reading reach 255 because the signal reached 255, or did the reading reach
>> 255 because from time to time a spike caused by system response to a floating
>> line forced the data lines low and "low" in the signal encoding happens to
>> correspond to binary 1?
>
> Let's say you are monitoring the measurement online and continuously
> the measurements come in,
>
> how do you say hey here is a spike occurred, we need to raise an
> alert...

Walter just tried to tell you this, I'll try in slightly different language:

That depends on your application. You have to subject _your_
application to analysis and measurement, and you have to use that
information to decide (a) what a spike is in _your_ context, and (b) how
to detect that spike, again in _your_ context.

Often (a) is harder than (b), unless you're doing your systems
engineering backward and you've designed the system so that you can't
tell real events from measurement noise.

At any rate, it ain't easy, it ain't universal, and it ain't already
done for your application.

--
Tim Wescott
Control system and signal processing consulting
www.wescottdesign.com
From: Walter Roberson on 11 Jun 2010 15:39
Luna Moon wrote:

> Let's say you are monitoring the measurement online and continuously
> the measurements come in,


> how do you say hey here is a spike occurred, we need to raise an
> alert...

> What's the best way to do this?

Again: what is a "spike" for your purposes, and how do you distinguish it from
noise, and how do you distinguish those from peaks?

The answer for distinguishing "peaks" is likely going to depend upon your
model of what a peak should look like, which is going to depend upon the
process involved in producing the peak.

The answer for distinguishing "spikes" is likely going to depend upon what
point the spikes occur at: for example if the spikes are occurring before the
sensor, then as the sensor has a discrete sampling rate then the spike might
be smeared over several samples, but an accurate reading of the spike would
ensue. We would also have to know how long the system needs to recover from a
spike -- e.g., if the spike was such that it saturated the sensor, then it
might require time for the sensor to drain off charge before it was accurate
again. Is the spike fundamentally analog or digital -- is it going to affect
(say) a voltage that will be sampled, or is it going to affect each
measurement bit individually, or is a spike such that it could corrupt all
remaining bits in a reading (e.g., if the voltage measurement is a
cascade-style divider to get the individual bits, then a spike that made it
through after some divisions had already been done would affect the rest of
the divisions.)

If the spike is occurring after the measurement has been made, then it is
going to affect the reading of a measurement, in which case we need to know
things like the data transmission protocol, bit encoding for the transmission,
likely sources of spikes and their duration, whether the spikes will tend to
drive signals high or low or invert them, are individual bauds or groups of
bauds going to be affected, what error detection is in place, what error
correction is in place...


You cannot just ask how to detect spikes: you need a good model of what a
spike _is_ in your particular environment and how it will change your data.
From: ImageAnalyst on 11 Jun 2010 16:54
For something quick and dirty, just try comparing your signal to the
median filtered version of your signal, or you might take a look at
Brett Shoelson's "deleteoutliers"
http://www.mathworks.com/matlabcentral/fileexchange/3961

If either one suits your needs, great. Otherwise keep looking. There
are lots of spike detecting algorithms and noise reduction algorithms,
for example see:

"Comparison of public peak detection algorithms for MALDI mass
spectrometry data analysis"
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631518/

Why don't you post your signal or image so we can see it? Just saying
you want to detect spikes or remove noise doesn't mean as much as if
you show us the signal and explain the context, or what you really
need to measure. In other words, so many people say "Tell me how to
measure X" but don't say "because I really need to measure Y." So
they never know that method Z was really the best approach to
measuring Y because they never gave anyone any larger context.
From: Clay on 11 Jun 2010 17:16
On Jun 11, 2:12 pm, Luna Moon <lunamoonm...(a)gmail.com> wrote:
> Hi all,
>
> I have very spiky signals. Visually it's very easy to detect spikes,
>
> but how do I detect spikes in a computerized appraoch?
>
> thanks a lot!

Given the sketchy details. I'd hazard a guess at something like run
your data through a median filter and subtract that result from a
equivalent delayed version of the original signal to emphasize the
spikes. Then maybe you can threshold detect the spikes. How well this
works really depends on how much different your spikes are from real
desired features of your signal. Like a lot of parametric signal
processing, you have to play around with it to see what really works.

Clay
From: Luna Moon on 11 Jun 2010 22:47
On Jun 11, 5:16 pm, Clay <c...(a)claysturner.com> wrote:
> On Jun 11, 2:12 pm, Luna Moon <lunamoonm...(a)gmail.com> wrote:
>
> > Hi all,
>
> > I have very spiky signals. Visually it's very easy to detect spikes,
>
> > but how do I detect spikes in a computerized appraoch?
>
> > thanks a lot!
>
> Given the sketchy details. I'd hazard a guess at something like run
> your data through a median filter and subtract that result from a
> equivalent delayed version of the original signal to emphasize the
> spikes. Then maybe you can threshold detect the spikes. How well this
> works really depends on how much different your spikes are from real
> desired features of your signal. Like a lot of parametric signal
> processing, you have to play around with it to see what really works.
>
> Clay

Here is my signal:

http://img810.imageshack.us/img810/5043/spikes.jpg

any thoughts?
First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: Simple hack to get $500 to your home
Next: what's the best way to detect cycle in time series?