I tried searching but came up short on my particular problem. I should mention that I am fairly new to MATLAB, so this may be something obvious but has slipped over my head.
I have an Excel file with accelerometer recordings of 5 events with some space inbetween. These events take place at times (ie rows) I have to specify, such as 120:250, 280:390, 430:943, and so on and so forth.
What I would like to do is to be able to loop through and extract the required data, and have it stored in variables such that each event will have it's own 'section' if you will, and each 'section' would contain it's own set of 'sub-sections' with the X, Y, Z accelerometer data.
My current set up is a manual one, and it looks like this:
X1 = xlsread('location.xlsx','sheet1','d110:d367');
X2 = xlsread('location.xlsx','sheet1','d367:d631');
X3 = xlsread('location.xlsx','sheet1','d631:d891');
X4 = xlsread('location.xlsx','sheet1','d891:d1134');
X5 = xlsread('location.xlsx','sheet1','d1134:d1361');
Y1 = xlsread('location.xlsx','sheet1','e110:e367');
Y2 = xlsread('location.xlsx','sheet1','e367:e631');
Y3 = xlsread('location.xlsx','sheet1','e631:e891');
Y4 = xlsread('location.xlsx','sheet1','E891:e1134');
Y5 = xlsread('location.xlsx','sheet1','e1134:e1361');
Z1 = xlsread('location.xlsx','sheet1','f110:f367');
Z2 = xlsread('location.xlsx','sheet1','f367:f631');
Z3 = xlsread('location.xlsx','sheet1','f631:f891');
Z4 = xlsread('location.xlsx','sheet1','f891:f1134');
Z5 = xlsread('location.xlsx','sheet1','f1134:f1361');
So you can see how it is not favorable. The other thing I'd like to do is to eventually use loops for cross correlation against other data sets, but again I'm not sure of the nature of the loops when dealing with 'dynamic' variables or what have you.
Right now the way I am thinking of doing it is that I specify the blocks of rows in a vector or something like that, and loop through for each activity, and then each axis.
Running XLSREAD for every variable won;t be optiomal for performance. This function uses COM interface (at least under Windows) and slow. If the data is not very big and can fit into the memory, it's better to read the whole sheet at once into a temporary variable and then sort the values into variables.
Another advice is not to use X1, X2 etc. You will have problems if you want to use those variable in a loop. If they have different length create a cell array, so they will become X{1}, X{2}, etc.
So, first read the whole file:
data = xlsread('location.xlsx','sheet1','D:F');
If you data all numeric you will get them in data matrix.
The index you can enter manually or get it from the data.
index = {120:250, 280:390, 430:943};
for ii = 1:numel(index)
X{ii} = data(index{ii},1);
Y{ii} = data(index{ii},2);
Z{ii} = data(index{ii},3);
end
Related
I have some code in Julia I've just wrote:
using FFTW
using Plots
using WAV, PlotlyJS
snd, sampFreq = wavread("input.wav")
N, _ = size(snd)
t = 0:1/(N-1):1;
s = snd[:,1]
y = fft(s)
y1 = copy(y)
for i = 1:N
if abs(y1[i]) > 800
y1[i] = 0
end
end
s_new = real(ifft(y1))
wavwrite(s_new, "output1.wav", Fs = sampFreq)
y2 = copy(y)
for i = 1:N
if abs(y2[i]) < 800
y2[i] = 0
end
end
s_new = real(ifft(y2))
wavwrite(s_new, "output2.wav", Fs = sampFreq)
sticks((abs.(y1)))
sticks!((abs.(y2)))
s1,k1 = wavread("output1.wav")
s2,k2 = wavread("output2.wav")
for i = 1:N
s1[i] += s2[i]
end
wavwrite(s1, "output3.wav", Fs = sampFreq)
it's the code that reads file input.wav, next do fft on the sound, dividing it into two files output1 with only frequencies > 800 and output2 with frequencies < 800.
In next part I merge the two files into output3. I expected something similar to input, but what I get sounds terrible (I mean it sounds like input, but is quieter and with hum bigger than expected).
My question is on which part of a code I loose the most information about input and is it a way to improve it, to get as output3 something almost like input?
You appear to not understand what the fft (fast fourier transform) returns. It returns a vector of amplitudes, not frequencies. The vector's components correspond to a the amplitude of a sine wave at a frequency that you can find using the fftfreq() function, but be sure to provide the fftfreq() function with its second argument, your sampFreq variable.
To decompose the sound, then, you need to zero the vector components you do not want, based on what fftfreq() tells you the frequencies corresponding to the bins (vector postions in the vector returned by fft().
You will still see a big drop in sound quality with reversing the process with ifft, because the fft will basically average parts of the signal by splitting it into the frequency dimension's bins.
I suggest a tutorial on fft() before you fix your code further -- you can google several of these.
Here is background information to the problem I am encountering:
1) output is a cell array, each cell contains a matrix of size = 1024 x 1024, type = double
2) labelbout is a cell array which is the identical to output, except that each matrix has been binarized.
3) I am using the function regionprops to extract the mean intensity and centroid values for ROIs (there are multiple ROIs in each image) for each cell of output
4) props is a 5 x 1 struct with 2 fields (centroid and mean intensity)
The problem: I would like to take the mean intensity values for each ROI in every matrix and export to excel. Here is what I have so far:
for i = 1:size(output,2)
props = regionprops(labelboutput{1,i},output{1,i},'MeanIntensity','Centroid');
end
for i = 1:size(output,2)
meanValues = getfield(props(1:length(props),'MeanIntensity'));
end
writetable(struct2table(props), 'advanced_test.xlsx');
There seem to be a few issues:
1) my getfield command is not working and gets the error: "Index exceeds matrix dimensions"
2) when the information is being stored into props, it overwrites the values for each matrix. How do I make props a 5 x n (where n = number of cells in output)?
Please help!!
1) my getfield command is not working and gets the error: "Index exceeds matrix dimensions"
An easier way to get numeric values out of the same field in an array of structs, as an array is: [structArray.fieldName]. In your case this will be:
meanValues = [props.MeanIntensity];
2) when the information is being stored into props, it overwrites the values for each matrix. How do I make props a 5 x n (where n = number of cells in output)?
One option would be to preallocate an empty cell of the necessary dimensions and then fill it in with your regionprops output. Like this:
props = cell(size(output,1),1);
for k = 1:size(output,2)
props{k} = regionprops(labelboutput{1,k},output{1,k},'MeanIntensity','Centroid');
end
for k = 1:size(output,2)
meanValues = [props{k}.MeanIntensity];
end
...
Another option would be to combine your loops so that you can use your matrix data before it is overwritten. Like this:
for i = 1:size(output,2)
props = regionprops(labelboutput{1,i},output{1,i},'MeanIntensity','Centroid');
meanValues = [props.MeanIntensity];
% update this call to place props in non-overlapping parts of your file (e.g. append)
% writetable(struct2table(props), 'advanced_test.xlsx');
end
The bad thing about this second one is it has a file I/O step right inside your loop which can really slow things down; not to mention you will need to curtail your writetable call so it places the resulting table in non-overlapping regions of 'advanced_test.xlsx'.
I would like to convert my data in frequency domain into time domain. In this attached excel sheet (book1.xlxs) Column A is Frequency. Column B and C is real and imaginary data (B+jC). Also attached you can see my code. But its not working. I would like to have the my result something shown in figure in time domain (green curve part-1).
[num, data, raw] = xlsread('Book1.xlsx');
ln=length(raw)-1; %find the length of the sequence
xk=zeros(1,ln); %initilise an array of same size as that of input sequence
ixk=zeros(1,ln); %initilise an array of same size as that of input sequence
rx = zeros(1,ln); %real value of fft
ix = zeros(1,ln); %imaginary value of fft
for i= 2:length(raw)
rx(i-1) = cell2mat(raw(i,2));
ix(i-1) = cell2mat(raw(i,3));
xk(i-1) = sqrt(rx(i-1)^2 + ix(i-1)^2);
end
for n=0:ln-1
for k=0:ln-1
ixk(n+1)=ixk(n+1)+(xk(k+1)*exp(i*2*pi*k*n/ln));
end
end
ixk=10*log(ixk./ln);
t=0:ln-1
plot(t, ixk)
In this image this code should give me the result similar to the green curve-part1
Instead of doing the FFT yourself, you could use the built-in Matlab functions to do it - much easier.
A good example from Mathworks is given here. The following is some code I have based myself on. The passed-in parameter f is your time domain trace, and fsampling is your sampling rate. The passed-out parameters freq and finv are your frequency vector and fourier transform, respectively.
function [freq, finv] = FourierTransform(f,fsampling)
% Fast Fourier Transform
fsampling = round(fsampling);
finv = fft(f,fsampling);
finv = finv(1:length(finv)/2+1); % Truncate out only the second half, due to symmetry
finv(2:end - 1) = 2*finv(2:end - 1); % Adjust amplitude to account for truncation
finv = finv./length(f);
freq = 0:fsampling/2;
end
Please help my poor knowledge of signal processing.
I want to smoothen some data. Here is my code:
import numpy as np
from scipy.signal import butter, filtfilt
def testButterworth(nyf, x, y):
b, a = butter(4, 1.5/nyf)
fl = filtfilt(b, a, y)
return fl
if __name__ == '__main__':
positions_recorded = np.loadtxt('original_positions.txt', delimiter='\n')
number_of_points = len(positions_recorded)
end = 10
dt = end/float(number_of_points)
nyf = 0.5/dt
x = np.linspace(0, end, number_of_points)
y = positions_recorded
fl = testButterworth(nyf, x, y)
I am pretty satisfied with results except one point:
it is absolutely crucial to me that the start and end point in returned values equal to the start and end point of input. How can I introduce this restriction?
UPD 15-Dec-14 12:04:
my original data looks like this
Applying the filter and zooming into last part of the graph gives following result:
So, at the moment I just care about the last point that must be equal to original point. I try to append copy of data to the end of original list this way:
the result is as expected even worse.
Then I try to append data this way:
And the slice where one period ends and next one begins, looks like that:
To do this, you're always going to cheat somehow, since the true filter applied to the true data doesn't behave the way you require.
One of the best ways to cheat with your data is to assume it's periodic. This has the advantages that: 1) it's consistent with the data you actually have and all your changing is to append data to the region you don't know about (so assuming it's periodic as as reasonable as anything else -- although may violate some unstated or implicit assumptions); 2) the result will be consistent with your filter.
You can usually get by with this by appending copies of your data to the beginning and end of your real data, or just small pieces, depending on your filter.
Since the FFT assumes that the data is periodic anyway, that's often a quick and easy approach, and is fully accurate (whereas concatenating the data is an estimation of an infinitely periodic waveform). Here's an example of the FFT approach for a step filter.
import numpy as np
import matplotlib.pyplot as plt
x = np.arange(0, 128)
y = (np.sin(.22*(x+10))>0).astype(np.float)
# filter
y2 = np.fft.fft(y)
f0 = np.fft.fftfreq(len(x))
y2[(f0<-.25) | (f0>.25)] = 0
y3 = abs(np.fft.ifft(y2))
plt.plot(x, y)
plt.plot(x, y3)
plt.xlim(-10, 140)
plt.ylim(-.1, 1.1)
plt.show()
Note how the end points bend towards each other at either end, even though this is not consistent with the periodicity of the waveform (since the segments at either end are very truncated). This can also be seen by adjusting waveform so that the ends are the same (here I used x+30 instead of x+10, and here the ends don't need to bend to match-up so they stay at level with the end of the data.
Note, also, to have the endpoints actually be exactly equal you would have to extend this plot by one point (at either end), since it periodic with exactly the wavelength of the original waveform. Doing this is not ad hoc though, and the result will be entirely consistent with your analysis, but just representing one extra point of what was assumed to be infinite repeats all along.
Finally, this FFT trick works best with waveforms of length 2n. Other lengths may be zero padded in the FFT. In this case, just doing concatenations to either end as I mentioned at first might be the best way to go.
The question is how to filter data and require that the left endpoint of the filtered result matches the left endpoint of the data, and same for the right endpoint. (That is, in general, the filtered result should be close to most of the data points, but not necessarily exactly match any of them, but what if you need a match at both endpoints?)
To make the filtered result exactly match the endpoints of a curve, one could add a padding of points at either end of the curve and adjust the y-position of this padding so that the endpoints of the valid part of the filter exactly matched the end points of the original data (without the padding).
In general, this can be done by either iterating towards a solution, adjusting the padding y-position until the ends line up, or by calculating a few values and then interpolating to determine the y-positions that would be required for the matched endpoints. I'll do the second approach.
Here's the code I used, where I simulated the data as a sine wave with two flat pieces on either side (note, that these flat pieces are not the padding, but I'm just trying to make data that looks a bit like the OPs).
import numpy as np
from scipy.signal import butter, filtfilt
import matplotlib.pyplot as plt
#### op's code
def testButterworth(nyf, x, y):
#b, a = butter(4, 1.5/nyf)
b, a = butter(4, 1.5/nyf)
fl = filtfilt(b, a, y)
return fl
def do_fit(data):
positions_recorded = data
#positions_recorded = np.loadtxt('original_positions.txt', delimiter='\n')
number_of_points = len(positions_recorded)
end = 10
dt = end/float(number_of_points)
nyf = 0.5/dt
x = np.linspace(0, end, number_of_points)
y = positions_recorded
fx = testButterworth(nyf, x, y)
return fx
### simulate some data (op should have done this too!)
def sim_data():
t = np.linspace(.1*np.pi, (2.-.1)*np.pi, 100)
y = np.sin(t)
c = np.ones(10, dtype=np.float)
z = np.concatenate((c*y[0], y, c*y[-1]))
return z
### code to find the required offset padding
def fit_with_pads(v, data, n=1):
c = np.ones(n, dtype=np.float)
z = np.concatenate((c*v[0], data, c*v[1]))
fx = do_fit(z)
return fx
def get_errors(data, fx):
n = (len(fx)-len(data))//2
return np.array((fx[n]-data[0], fx[-n]-data[-1]))
def vary_padding(data, span=.005, n=100):
errors = np.zeros((4, n)) # Lpad, Rpad, Lerror, Rerror
offsets = np.linspace(-span, span, n)
for i in range(n):
vL, vR = data[0]+offsets[i], data[-1]+offsets[i]
fx = fit_with_pads((vL, vR), data, n=1)
errs = get_errors(data, fx)
errors[:,i] = np.array((vL, vR, errs[0], errs[1]))
return errors
if __name__ == '__main__':
data = sim_data()
fx = do_fit(data)
errors = vary_padding(data)
plt.plot(errors[0], errors[2], 'x-')
plt.plot(errors[1], errors[3], 'o-')
oR = -0.30958
oL = 0.30887
fp = fit_with_pads((oL, oR), data, n=1)[1:-1]
plt.figure()
plt.plot(data, 'b')
plt.plot(fx, 'g')
plt.plot(fp, 'r')
plt.show()
Here, for the padding I only used a single point on either side (n=1). Then I calculate the error for a range of values shifting the padding up and down from the first and last data points.
For the plots:
First I plot the offset vs error (between the fit and the desired data value). To find the offset to use, I just zoomed in on the two lines to find the x-value of the y zero crossing, but to do this more accurately, one could calculate the zero crossing from this data:
Here's the plot of the original "data", the fit (green) and the adjusted fit (red):
and zoomed in the RHS:
The important point here is that the red (adjusted fit) and blue (original data) endpoints match, even though the pure fit doesn't.
Is this a valid approach? Of the various options, this seems the most reasonable since one isn't usually making any claims about the data that isn't being shown, and also for show region has an accurately applied filter. For example, FFTs usually assume the data is zero or periodic beyond the boundaries. Certainly, though, to be precise one should explain what was done.
I am trying to calculate a few hundred rows of data, solving a system of linear equations using matrices. I am building my matrices based on the row data. I take the inverse of the resultant 3x3 and then multiply twice, once for x's and once for y's. I get 6 variables from this: a,b,c,d,e and f. How can I copy down the example so that is solves for all rows? I am providing the data and then the formulas I am using now. Right now, if I copy down it skips 3 rows or if I copy down with 3 examples it skips 9 rows. I mean I guess I could go an try to insert extra rows into all 300 so I end up with 900 rows, but there has to be an easier way!
I can't figure out how to get my data to populate in here correctly so here is a link: http://codepad.org/qZwua3h9
Note: I split up the matrix rows so you could see them easier, they are not split up on my sheet.
Edit: If anyone can figure out how to paste the example data here I would welcome it so that this post may be of use to someone in the future. I am not sure how long codepad keeps their pastes.
I gave up and came to the conclusion that there is no reasonable amount of effort that will yield the desired results. Not only was the example case only ONE transformation, but the intended case was for 3 transformations - so three times the work. I came up with a Matlab solution in about 15 minutes. I understand that not everyone has access to Matlab though. So, if someone comes up with any reasonable working excel solution, I would welcome the knowledge and mark that answer as the accepted one. Regardless, here is the Matlab script:
M = csvread('pointData.csv');
T1result = zeros(215,6);
T2result = zeros(215,6);
T3result = zeros(215,6);
for i=1:215,
m = [M(i,1) M(i,2) 1; M(i,3) M(i,4) 1; M(i,5) M(i,6) 1];
x = [M(i,7);M(i,9);M(i,11)];
y = [M(i,8);M(i,10);M(i,12)];
xresult = m\x;
yresult = m\y;
T1result(i,:) = [transpose(xresult),transpose(yresult)];
m = [M(i,7) M(i,8) 1; M(i,9) M(i,10) 1; M(i,11) M(i,12) 1];
x = [M(i,13);M(i,15);M(i,17)];
y = [M(i,14);M(i,16);M(i,18)];
xresult = m\x;
yresult = m\y;
T2result(i,:) = [transpose(xresult),transpose(yresult)];
m = [M(i,13) M(i,14) 1; M(i,15) M(i,16) 1; M(i,17) M(i,18) 1];
x = [M(i,19);M(i,21);M(i,23)];
y = [M(i,20);M(i,22);M(i,24)];
xresult = m\x;
yresult = m\y;
T3result(i,:) = [transpose(xresult),transpose(yresult)];
end
LeafId = csvread('extraColumnsForID.csv');
Tresult = [LeafId, T1result, T2result, T3result];
csvwrite('transforms.csv',Tresult);