How to find similar events of a signal in python - python-3.x

Question

I have one signal and now I want to find the similar events happened multiple times in that signal. For example the below picture we can see there are three events.
I have created this using below values:
import matplotlib.pyplot as plt
y = [
0, 1, 2,
.3, .4, .4, .6, .5, .4, .3, .4, .5, .6, .7, .6, .5, .4, .3, .5, .7, .8, .6, .3, .4, .5, .6, .5, .4, .3, #1
.3, .3, .3, .3, .3, 5, 4,
0, 1, 3, 4, 8, 9, 13, 10, 7, 4, 6, 3, 4, 3, #2
.3, 4, 4.4, 4.3, 3, 3.4, 3.2, 4, 3.8, 4, 6, 6, 5, 4, 1,
.3, .4, .5, .6, .5, .4, .3, .4, .5, .6, .7, .6, .3, .4, .3, .5, .7, .8, .6, .3, .4, .6, .6, .5, .4, .3, # 1
0, 1, 3, 4, 6, 9, 13.5, 9.5, 7, 4, 6, 3, 4, 3, #2
.3, .4, .5, .4, .5, .4, .3, .4, .5, .6, .7, .6, .5, .4, .3, .5, .7, .8, .6, .3, .4, .5, .6, .5, .4, .3, # 1
0, 1, 3, 4, 8, 9, 14, 10, 7, 4, 6, 3, 4, 3, #2
.3, 2, 1, 1,
2, 3, 4, 4.5, 4, 3, 2, 3, 4, 4, 4, 3, 2, #3
1,2,2,1,1,
2, 3, 4, 4, 4, 3, 2, 3, 4, 5, 4, 3, 2, #3
1,2,3, .2, .1, 0
]
plt.plot(y)
plt.show()
First event:
Second event:
Third event:
First event happend 3 times, second event 3 times and third event 2 times. They are almost similar. In real situation the similarity will be little bit less than shown in the above picture.
Now i want to find:
1. similar type events
2. how similar they are
3. where it happend.
For example first event occured again in:
timestamp | similarity
04 - 22 | 100%
60 - 90 | 95%
110 - 130 | 96%
I want to do it in python. How can i do this? Is there any signal processing libraries for this kind of task?

Answer 1

Using numpy you could differentiate using numpy.diff which could give you a clue based on a certain threshold where your events happen. After you get those events, you can use numpy.correlate between your events and the input signal to find the most likely places where the events occur.
correlation
differentiation
Follow-up
If you want to completely automate it for any arbitrary signal, there's gonna have to be some machine learning in there somewhere. I won't give you a machine learning algo outright. I will, however, give you pointers as to what parameters could be considered as factors in your algo.
Take a look at the following
import matplotlib.pyplot as plt
import numpy as np
y = [0, 1, 2,
.3, .4, .4, .6, .5, .4, .3, .4, .5, .6, .7, .6, .5, .4, .3, .5, .7, .8, .6, .3, .4, .5, .6, .5, .4, .3,
.3, .3, .3, .3, .3, 5, 4,
0, 1, 3, 4, 8, 9, 13, 10, 7, 4, 6, 3, 4, 3,
.3, 4, 4.4, 4.3, 3, 3.4, 3.2, 4, 3.8, 4, 6, 6, 5, 4, 1,
.3, .4, .5, .6, .5, .4, .3, .4, .5, .6, .7, .6, .3, .4, .3, .5, .7, .8, .6, .3, .4, .6, .6, .5, .4, .3,
0, 1, 3, 4, 6, 9, 13.5, 9.5, 7, 4, 6, 3, 4, 3,
.3, .4, .5, .4, .5, .4, .3, .4, .5, .6, .7, .6, .5, .4, .3, .5, .7, .8, .6, .3, .4, .5, .6, .5, .4, .3,
0, 1, 3, 4, 8, 9, 14, 10, 7, 4, 6, 3, 4, 3,
.3, 2, 1, 1,
2, 3, 4, 4.5, 4, 3, 2, 3, 4, 4, 4, 3, 2,
1, 2, 2, 1, 1,
2, 3, 4, 4, 4, 3, 2, 3, 4, 5, 4, 3, 2,
1, 2, 3, .2, .1, 0
]
x = 5 #Mean Convolution Window Width
v = np.full(x,1/x) #Convolution vector
y=np.convolve(y,v) #Apply X-Width Convolution on signal to get average between X samples
diff_y = np.abs(np.diff(y))
plt.figure()
plt.plot(diff_y)
plt.show()
diff_y_filtered=diff_y
diff_y_filtered[diff_y < 0.20*np.nanmax(diff_y)] = 0 #Differientiate the signal to look for large variations
plt.figure()
plt.plot(y)
plt.plot(diff_y_filtered)
plt.show()
diff_y_peaks = np.diff(diff_y_filtered) #Second derivative
diff_y_peaks = np.convolve(diff_y_peaks,[-1/8,-1/4,-1/2,1.75,-1/2,-1/4,-1/8]) #You may wanna tweak the filter
plt.figure()
plt.plot(diff_y_peaks)
plt.show()
diff_y_peaks[diff_y_peaks < 0.5] = 0; #This is a parameter that would be tweaked
diff_y_peaks[diff_y_peaks > 0]=np.nanmax(y) #Make things pretty
plt.figure()
plt.stem(diff_y_peaks)
plt.plot(y)
plt.show()
First and second derivatives come to mind because you'd be looking for large variations, I presume, in the signal. Even if events contain large variations themselves, this shouldn't be a problem because of a factor I will point out later on.
I first smoothed the signal to avoid having large variations within events. Then I applied the first derivative to find large variations. I then cut small peaks with a threshold. That threshold's value could be a parameter in your model.
I then filter a second time and convolve again. You can tweak that filter to give you better results. Some filters find more possible cutoff points, some less. It's not a big deal if you get many cutoff points next to each other, because you can train your model to find event lengths that are too small, or you can directly specify that, say, 2 cutoff points 5 samples apart don't constitute an event.
Personally, I believe you might want to look at getting more events that are contained within a bigger event, that will probably get you the best results.