Optimize run time of octave/matlab function - multithreading

I am using the Google Earth Toolbox, for Octave, which is extremely similar to the same library made for MATLAB. See this link, for the MATLAB doc for the tool.
Unfortunately I am having a bit of trouble with a certain function: ge_quiver. This function creates a quiver plot and translates the result into KML code. My problem with it
is that it is a bit too slow for my liking.
For a data variable of 30x30 size, the function takes around 10 seconds to complete. This is fine, except for the fact that I am trying to make an animation quiver plot, with 23 other same sized data variables (24 total). Since Octave (and MATLAB for that matter) normally only runs on one thread, the whole thing is run serially, and takes 10*24 = 240 s.
I have tried parfor loops, but they have only removed 10 seconds from the total run time. I have a computer with 16 cores. Considering, this is embarrassingly parallel (no dependence between variables), this should ideally take:
10 seconds for the first 16 variables
+ 10 seconds for the 8 remaining variables
------------------------------------------------------
= 20 seconds total.
I haven't even tested arrayfun, because I don't know how to adapt my function (p_ge_quiver). Nonetheless, many have said that arrayfun will not make it faster.
The following code is approximately what I am trying to do. Remember that the variable
data_u is of size 30x30x24. Same thing for data_v, lonand lat.
...
...
[YYYY,MM,DD,HH,mm,ss] = ncdate(NCFILE);
date_s.year = YYYY;
date_s.month = MM;
date_s.day = DD;
date_s.hour = HH;
date_s.minute = mm;
date_s.second = ss;
TIMESTEP = 60;
parfor i = 1:size(data_u,3)
dt = TIMESTEP*i;
kml(i) = p_ge_quiver(lon,lat,data_u,data_v,dt,TIMESTEP,data_s);
endparfor
...
...
function kml = p_ge_quiver(lon,lat,u,v,data_time,step,date_s)
% V_GE_QUIVER - Writes the quiver plot into KML.
% Get date variables.
YYYY = date_s.year;
MM = date_s.month;
DD = date_s.day;
HH = date_s.hour;
mm = date_s.minute;
ss = date_s.second;
% Date format. Use Google's.
F = 'yyyy-mm-ddTHH:MM:SSZ';
% Start and end dates of data.
tStart = datestr(datenum(YYYY,MM,DD,HH,mm+data_time,ss),F);
tEnd = datestr(datenum(YYYY,MM,DD,HH,mm+data_time+step,ss),F);
% Quiver plot.
kml = ge_quiver(lon,lat,u,v,'timeSpanStart', tStart, ...
'timeSpanStop' , tEnd , ...
'visibility' , 0 , ...
'lineColor' , 'AAFFFFFF');
end

Seems to me that the problem is on what ge_quiver does internally which you don't show. Aside that, you should note that this language is designed to write vectorized code (and not only because of the difference in speed).
Looking at your code, you can simplify it lot (and this usually does have an effect in speed as well) by making use of a vectorized syntax:
step = 60;
dt = step:step:(step * size (data_u, 3));
## Get date variables.
YYYY = date_s.year;
MM = date_s.month;
DD = date_s.day;
HH = date_s.hour;
mm = date_s.minute;
ss = date_s.second;
## Date format. Use Google's.
F = "yyyy-mm-ddTHH:MM:SSZ";
tStart = datestr (datenum (YYYY, MM, DD, HH, mm + dt , ss), F);
tEnd = datestr (datenum (YYYY, MM, DD, HH, mm + dt + step, ss), F);
At this point, tStart and tEnd have all the dates that you need, one time per row. The only required loop is the following:
for i=1:rows(tStart)
kml = ge_quiver (lon, lat, data_u, data_v, "timeSpanStart", tStart(i,:), ...
"timeSpanStop" , tEnd(i,:) , ...
"visibility" , 0 , ...
"lineColor" , "AAFFFFFF");
endfor
At this point is all dependent on what ge_quiver does. The mathworks site you linked appears as under maintenance. I looked into the Google Earth toolbox website and couldn't find that function (they have a ge_quiver3 function though).
I would argue that a function for this type of languages should allow for vector inputs, so go and confirm it. Maybe you don't need a loop at all.

Related

Use of fft on sound using Julia

I have some code in Julia I've just wrote:
using FFTW
using Plots
using WAV, PlotlyJS
snd, sampFreq = wavread("input.wav")
N, _ = size(snd)
t = 0:1/(N-1):1;
s = snd[:,1]
y = fft(s)
y1 = copy(y)
for i = 1:N
if abs(y1[i]) > 800
y1[i] = 0
end
end
s_new = real(ifft(y1))
wavwrite(s_new, "output1.wav", Fs = sampFreq)
y2 = copy(y)
for i = 1:N
if abs(y2[i]) < 800
y2[i] = 0
end
end
s_new = real(ifft(y2))
wavwrite(s_new, "output2.wav", Fs = sampFreq)
sticks((abs.(y1)))
sticks!((abs.(y2)))
s1,k1 = wavread("output1.wav")
s2,k2 = wavread("output2.wav")
for i = 1:N
s1[i] += s2[i]
end
wavwrite(s1, "output3.wav", Fs = sampFreq)
it's the code that reads file input.wav, next do fft on the sound, dividing it into two files output1 with only frequencies > 800 and output2 with frequencies < 800.
In next part I merge the two files into output3. I expected something similar to input, but what I get sounds terrible (I mean it sounds like input, but is quieter and with hum bigger than expected).
My question is on which part of a code I loose the most information about input and is it a way to improve it, to get as output3 something almost like input?
You appear to not understand what the fft (fast fourier transform) returns. It returns a vector of amplitudes, not frequencies. The vector's components correspond to a the amplitude of a sine wave at a frequency that you can find using the fftfreq() function, but be sure to provide the fftfreq() function with its second argument, your sampFreq variable.
To decompose the sound, then, you need to zero the vector components you do not want, based on what fftfreq() tells you the frequencies corresponding to the bins (vector postions in the vector returned by fft().
You will still see a big drop in sound quality with reversing the process with ifft, because the fft will basically average parts of the signal by splitting it into the frequency dimension's bins.
I suggest a tutorial on fft() before you fix your code further -- you can google several of these.

frequency to time conversion using MATLAB

I would like to convert my data in frequency domain into time domain. In this attached excel sheet (book1.xlxs) Column A is Frequency. Column B and C is real and imaginary data (B+jC). Also attached you can see my code. But its not working. I would like to have the my result something shown in figure in time domain (green curve part-1).
[num, data, raw] = xlsread('Book1.xlsx');
ln=length(raw)-1; %find the length of the sequence
xk=zeros(1,ln); %initilise an array of same size as that of input sequence
ixk=zeros(1,ln); %initilise an array of same size as that of input sequence
rx = zeros(1,ln); %real value of fft
ix = zeros(1,ln); %imaginary value of fft
for i= 2:length(raw)
rx(i-1) = cell2mat(raw(i,2));
ix(i-1) = cell2mat(raw(i,3));
xk(i-1) = sqrt(rx(i-1)^2 + ix(i-1)^2);
end
for n=0:ln-1
for k=0:ln-1
ixk(n+1)=ixk(n+1)+(xk(k+1)*exp(i*2*pi*k*n/ln));
end
end
ixk=10*log(ixk./ln);
t=0:ln-1
plot(t, ixk)
In this image this code should give me the result similar to the green curve-part1
Instead of doing the FFT yourself, you could use the built-in Matlab functions to do it - much easier.
A good example from Mathworks is given here. The following is some code I have based myself on. The passed-in parameter f is your time domain trace, and fsampling is your sampling rate. The passed-out parameters freq and finv are your frequency vector and fourier transform, respectively.
function [freq, finv] = FourierTransform(f,fsampling)
% Fast Fourier Transform
fsampling = round(fsampling);
finv = fft(f,fsampling);
finv = finv(1:length(finv)/2+1); % Truncate out only the second half, due to symmetry
finv(2:end - 1) = 2*finv(2:end - 1); % Adjust amplitude to account for truncation
finv = finv./length(f);
freq = 0:fsampling/2;
end

sliding window median of correlations of a matrix

Question parts:
Is there a "julia way" to implement a sliding window?
What is needed in julia to ignore NaNs?
There is a matrix with 264 recording points (rows) and 200 time points (columns). I want to get the median correlation of each recording point with every other point over a 10 sample window.
I've tried this the matlab-way (tm) by creating a 3d 264x264x10 matrix where the third dim is the correlation for that window. In matlab, I would do median(cors,3) much like julia can do mean(cors,3). But median does not have support for this. It looks like mapslices(median,cors,3) might be what I want, but some recording points have NaNs. In R, I might look to na.omit() or function options like na.ignore=T But I don't see that for julia.
#oned=readdlm("10152_20111123_preproc_torque.1D")
oned=rand(200,264); oned[:,3]=NaN; oned[:,200]=NaN
windows=10
samplesPerWindow=size(oned,1)/windows
cors=zeros(size(oned,2),size(oned,2),windows)
for i=1:windows
startat=(i-1)*windows+1
endat=i*windows
corofsamples=cor(oned[startat:i*windows,:])
cors[:,:,i]= corofsamples
end
med = mapslices(median,cors,3) # fail b/c NaN
Here's one approach, which uses functions to encapsulate parts of the task. By creating a specialized version of the median function that ignores NaN, it's easier to use mapslices:
function findcors(oned, windows)
samplesPerWindow = size(oned, 1) / windows
cors = zeros(size(oned, 2), size(oned, 2), windows)
for i = 1:windows
startat = (i - 1) * samplesPerWindow + 1
endat = i * samplesPerWindow
corofsamples = cor(oned[startat:endat, :])
cors[:, :, i] = corofsamples
end
return cors
end
function nanmedian(A)
cleanA = A[isfinite(A)]
if isempty(cleanA)
NaN
else
return median(cleanA)
end
end
oned = rand(200, 264)
oned[:, 3] = NaN
oned[:, 200] = NaN
cors = findcors(oned, 10)
med = mapslices(nanmedian, cors, 3)
I believe your original code was using the wrong window length inside the main loop. Hopefully I've fixed that.
The DataFrames package provides an NA value and tools to ignore NA, but still needs to clean up its median function to exploit those tools.

Creating a continuous tone in MATLAB whose frequency varies in real-time depending on user input

I am currently working on a graphing program in MATLAB that takes input and maps a point to x-y space using this input. However, the program should also output a continuous tone whose frequency varies depending on the location of the point.
I was able to get the tone generation done, however could not get the tone to work continuously due to the nature of the program. (Code in between tone generations) I thought I could solve this using a parfor loop with the code that alters the frequency in one iteration of the loop, and the code that generates the tone in another but cannot seem to get it due to the following error:
Warning: The temporary variable frequency will be cleared at the
beginning of each iteration of the parfor loop. Any value assigned to
it before the loop will be lost. If frequency is used before it is
assigned in the parfor loop, a runtime error will occur. See Parallel
for Loops in MATLAB, "Temporary Variables".
In multiThreadingtest at 5 Error using multiThreadingtest (line 5) Reference to a cleared variable frequency.
Caused by:
Reference to a cleared variable
frequency.
And my code:
global frequency
frequency = 100;
parfor ii=1:2
if ii==1
Fs = 1000;
nSeconds = 5;
y = 100*sin(linspace(0, nSeconds*frequency*2*pi, round(nSeconds*Fs)));
sound(y, Fs);
elseif ii==2
frequency = 100
pause(2);
frequency = 200
pause(2);
frequency = 300
pause(2);
end
end
The solution may not come from multithreading, but from the use of another function to output a tone(audioplayer, play, stop). 'audioplayer/play' has the ability to output sounds that overlap in time. So basically, a pseudo code would be:
get the value of the input
generate/play a corresponding 5 second tone
detect if any change in the input
if no change & elapsed time close to 5 seconds
generate/play an identical 5 second tone
if change
generate a new 5 second tone
%no overlapping
stop old
play new
%overlapping (few milliseconds)
play new
stop old
The matlab code showing the 'sound'/'play' differences.
Fs = 1000;
nSeconds = 5;
frequency = 100;
y1 = 100*sin(linspace(0, nSeconds*frequency*2*pi, round(nSeconds*Fs)));
aud1 = audioplayer(y1, Fs);
frequency = 200;
y2 = 100*sin(linspace(0, nSeconds*frequency*2*pi, round(nSeconds*Fs)));
aud2 = audioplayer(y2, Fs);
% overlapping sound impossible
sound(y1, Fs);
pause(1)
sound(y2, Fs);
% overlapping sound possible
play(aud1);
pause(1);
disp('can compute here');
play(aud2);
pause(1);
stop(aud1);
pause(1);
stop(aud2);

How to copy down matrix that references a row in Excel?

I am trying to calculate a few hundred rows of data, solving a system of linear equations using matrices. I am building my matrices based on the row data. I take the inverse of the resultant 3x3 and then multiply twice, once for x's and once for y's. I get 6 variables from this: a,b,c,d,e and f. How can I copy down the example so that is solves for all rows? I am providing the data and then the formulas I am using now. Right now, if I copy down it skips 3 rows or if I copy down with 3 examples it skips 9 rows. I mean I guess I could go an try to insert extra rows into all 300 so I end up with 900 rows, but there has to be an easier way!
I can't figure out how to get my data to populate in here correctly so here is a link: http://codepad.org/qZwua3h9
Note: I split up the matrix rows so you could see them easier, they are not split up on my sheet.
Edit: If anyone can figure out how to paste the example data here I would welcome it so that this post may be of use to someone in the future. I am not sure how long codepad keeps their pastes.
I gave up and came to the conclusion that there is no reasonable amount of effort that will yield the desired results. Not only was the example case only ONE transformation, but the intended case was for 3 transformations - so three times the work. I came up with a Matlab solution in about 15 minutes. I understand that not everyone has access to Matlab though. So, if someone comes up with any reasonable working excel solution, I would welcome the knowledge and mark that answer as the accepted one. Regardless, here is the Matlab script:
M = csvread('pointData.csv');
T1result = zeros(215,6);
T2result = zeros(215,6);
T3result = zeros(215,6);
for i=1:215,
m = [M(i,1) M(i,2) 1; M(i,3) M(i,4) 1; M(i,5) M(i,6) 1];
x = [M(i,7);M(i,9);M(i,11)];
y = [M(i,8);M(i,10);M(i,12)];
xresult = m\x;
yresult = m\y;
T1result(i,:) = [transpose(xresult),transpose(yresult)];
m = [M(i,7) M(i,8) 1; M(i,9) M(i,10) 1; M(i,11) M(i,12) 1];
x = [M(i,13);M(i,15);M(i,17)];
y = [M(i,14);M(i,16);M(i,18)];
xresult = m\x;
yresult = m\y;
T2result(i,:) = [transpose(xresult),transpose(yresult)];
m = [M(i,13) M(i,14) 1; M(i,15) M(i,16) 1; M(i,17) M(i,18) 1];
x = [M(i,19);M(i,21);M(i,23)];
y = [M(i,20);M(i,22);M(i,24)];
xresult = m\x;
yresult = m\y;
T3result(i,:) = [transpose(xresult),transpose(yresult)];
end
LeafId = csvread('extraColumnsForID.csv');
Tresult = [LeafId, T1result, T2result, T3result];
csvwrite('transforms.csv',Tresult);

Resources