I am trying to write a program to analyze data from a simulation. Since the simulation software I am using is what is running the Lua program, I am not sure if this is the right place to ask this question, but I am probably making a programming error.
I am struggling with the difference between using the simple and complete I/O models. I have a block of code, which works, and looks like this:
io.output([[filename_and_location]])
function segment.other_actions
if ion_splat ~= 0 then io.write(ion_px_mm, "\n") end
io.close()
end
Note: ion_splat and ion_px_mm are pre-determined variables that take on number values. This code is run over and over again throughout the simulation.
Then I decided to try achieving the same thing using the complete I/O model like this:
f = io.open([[file_name_and_location]],"w")
function segment.other_actions ()
if ion_splat ~= 0 then f:write(ion_py_mm, "\n") end
f:close()
end
end
This runs, but takes a lot longer than the other way. Why is that?
Example 1:
for i = 1, 1000 do
io.output("test.txt")
io.write("some data to be written\n")
io.close()
end
Example 2:
for i = 1, 1000 do
local f = io.open("test.txt", "w")
f:write("some data to be written\n")
f:close()
end
There is no measurable difference in the execution time.
The latter approach is usually preferable because the used file is identified explicitly.
Related
I have an ODE that uses many functions. I wish to export these "helper" functions so that I may graph them vs the independent variable of the ODE.
function dFfuncvecdW = ODE(W,Ffuncvec);
X = Ffuncvec(1);
y = Ffuncvec(2);
#lots of code
R = ... #R is a function of X,W and y.
#and a few other functions that are a function of X,W and y.
dXdW = ... #some formula
dydW = ... #some formula
dFfuncvecdW = [dXdW; dydW];
end
I call this function with:
Wspan = [0 8000.]
X0 = [0; 1.]
[W,X] = ode45(#ODE, Wspan, X0);
I can easily output X or W to an excel file:
xlswrite(filename,X,'Conversion','A1');
But I what I need is to save "R" and many other functions' values to an Excel file.
How do I do that?
I am still extremely new to Matlab. I usually use Polymath, but for this system of ODE's, Polymath cannot compute the answer within a reasonable amount of time.
EDIT1: The code I use was generated by Polymath. I used a basic version of my problem so that Polymath may excecute the program as it only gives the Matlab code once the Polymath code has succefully run. After the export, the complete set of equations were entered.
The easiest, and possibly fastest, way to handle this is to re-evaluate your functions after ode45 returns W and X. If the functions are vectorized it will be easy. Otherwise, just use a simple for loop that iterates from 1 to length(W).
Alternatively, you can use an output function to save your values on each iteration to a file, or a global, or, most efficiently, a sub-function (a.k.a. nested function) variable that shares scope with an outer function (see here, for example). See this answer of mine for an example of how to use an output function.
I found a rather quick and painless solution to my answer.
I merely appended a text file with code inside the ode function.
EDIT: I am unable to comment because I do have enough rep on this branch of SE.
My solution was add the following code:
fid = fopen('abc1.txt', 'at');
fprintf(fid, '%f\n', T);
fclose(fid);
right above
dYfuncvecdW = [dFAdW; dFBdW; dFCdW; dFDdW; dydW];
at the end of the ode function. This proved to be a temporary solution. I have opened another question about the output I recieved.
Instead of parfor loop I want to create 4 threads and write out the code for each individually. What's the syntax for that?
You have two options here. The first is to use parfeval where you can request several independent function evaluations, like so:
% The following line executes
% out = a(a1, a2, a3) on a worker. (The number 1 is the
% the number of outputs requested from the function evaluation)
% The results can be obtained using
% out = fetchOutputs(f_a);
f_a = parfeval(#a, 1, a1, a2, a3);
% and so on...
f_b = parfeval(#b, 1, b1, b2);
f_c = parfeval(#c, 1, c1);
f_d = parfeval(#d, 1, d1, d2);
You can retrieve the results using fetchOutputs(f_a) etc.
Another option is to use spmd like so:
spmd
switch labindex
case 1
a();
case 2
b();
...
end
end
Generally, for independent tasks, I would suggest parfeval since this approach is not dependent on the number of workers in your parallel pool, whereas the spmd approach is.
I recommend to use Edric's answer or this solution
(I leave the answer here for the comments.)
Forget about cores, you want to distribute your tasks among your worker processes.
Simple "hack" solution:
n=4
result=cell(4,1)
parfor idx=1:n
switch idx
case 1
r=f()
case 2
r=g(1)
case 3
r=g(2)
case 4
r=h()
end
result{idx}=r
end
For a more advanced solution, I recommend to create individual jobs and submit them. This is explained in detail here. In your case you create a job with four tasks, then you submit it. The biggest advantage of this solution is, that you avoid unnecessary broadcasting of variables.
In both solutions you don't control which worker processes which task, but you typically don't want to do this.
I have code that is structurally similar to the following in Matlab:
bestConfiguration = 0;
bestConfAwesomeness = 0;
for i=1:X
% note that providing bestConfAwesomeness to the function helps it stop if it sees the current configuration is getting hopeless anyway
[configuration, awesomeness] = expensive_function(i, bestConfAwesomeness);
if awesomeness > bestConfAwesomeness
bestConfAwesomeness = awesomeness;
bestConfiguration = configuration;
end
end
There is a bit more to it but the basic structure is the above. X can get very large. I am trying to make this code run in parallel, since expensive_function() takes a long time to run.
The problem is that Matlab won't let me just change for to parfor because it doesn't like that I'm updating the best configuration in the loop.
So far what I've done is:
[allConfigurations, allAwesomeness] = deal(cell(1, X));
parfor i=1:X
% note that this is not ideal because I am forced to use 0 as the best awesomeness in all cases
[allConfigurations{i}, allAwesomeness{i}] = expensive_function(i, 0);
end
for i=1:X
configuration = allConfigurations{i};
awesomeness = allAwesomeness{i};
if awesomeness > bestConfAwesomeness
bestConfAwesomeness = awesomeness;
bestConfiguration = configuration;
end
endfor
This is better in terms of time it takes to run; however, for large inputs it takes huge amounts of memory because all the configurations are always saved. Another problem is that using parfor forces me to always provide 0 as the best configuration even though better ones might be known.
Does Matlab provide a better way of doing this?
Basically, if I didn't have to use Matlab and could manage the threads myself, I'd have one central thread which gives jobs to workers (i.e. make them run expensive_function(i)) and once a worker returns, look at the data it produced and compare it to the best found so far and update it accordingly. There would be no need to save all the configurations which seems to be the only way to make parfor work.
Is there a way to do the above in Matlab?
Using the bestConfAwesomeness each time round the loop means that the iterations of your loop are not order-independent, hence why PARFOR is unhappy. One approach you could take is to use SPMD and have each worker perform expensiveFunction in parallel, and then communicate to update bestConfAwesomeness. Something like this:
bestConfiguration = 0;
bestConfAwesomeness = 0;
spmd
for idx = 1:ceil(X/numlabs)
myIdx = labindex + ((idx-1) * numlabs);
% should really guard against myIdx > X here.
[thisConf, thisAwesome] = expensiveFunction(myIdx, bestConfAwesomeness);
% Now, we must communicate to see if who is best
[bestConfiguration, bestAwesomeness] = reduceAwesomeness(...
bestConfiguration, bestConfAwesomeness, thisConf, thisAwesome);
end
end
function [bestConf, bestConfAwesome] = reduceAwesomeness(...
bestConf, bestConfAwesome, thisConf, thisAwesome)
% slightly lazy way of doing this, could be optimized
% but probably not worth it if conf & awesome both scalars.
allConfs = gcat(bestConf);
allAwesome = gcat(thisAwesome);
[maxThisTime, maxLoc] = max(allAwesome);
if maxThisTime > bestConfAwesome
bestConfAwesome = maxThisTime;
bestConf = allConfs(maxLoc);
end
end
I'm not sure that the kind of control over your threads is possible with Matlab. However, since X is very large, it may be worth doing the following, which costs you one more iteration of expensiveFunction:
%# calculate awesomeness
parfor i=1:X
[~,awesomeness(i)] = expensiveFunction(i);
end
%# find the most awesome i
[mostAwesome,mostAwesomeIdx] = min(awesomeness);
%# get the corresponding configuration
bestConfiguration = expensiveFunction(mostAwesomeIdx);
Here's my program.
local t = {}
local match = string.gmatch
local insert = table.insert
val = io.read("*a")
for num in match(val, "%d+") do
insert(t, num)
end
I'm wondering if there is a faster way to load a large (16MB+) array of integers than this. Considering the data is composed of line after line of a single number can this be made faster? Should I be looking at io.read("*n") instead?
Given that your file size is 16MB, your loading routine's performance will be dominated by file IO. How long it takes you to process the loaded data will generally be irrelevant next to that.
Just try it; profile how long it takes to just load the file (stopping the script after io.read), then profile how long the whole script takes. The latter will be longer, but it's only going to be by some relatively small percentage, not vast amounts.
Loading the whole file at once the way you're doing will almost certainly be faster than doing it piecemeal. Filesystems like reading entire blocks of data all at once, rather than bits at a time. Beyond that, how to process the text is relatively irrelevant.
I'm not sure if its faster, but read("*n") is much simpler...
local t = { }
while true do
local n = io.stdin:read("*n")
if n == nil then break end
table.insert ( t , n )
end
Probably, this would be faster:
local t = {}
local match = string.match
for line in io.lines() do
t[#t+1] = match(line, '%d+')
end
Don't forget to convert strings to numbers.
I have a cell array in Matlab:
strings = {'one', 'two', 'three'};
How can I efficiently calculate the length of all three strings? Right now I use a for loop:
lengths = zeros(3,1);
for i = 1:3
lengths(i) = length(strings{i});
end
This is however unusable slow when you have a large amount of strings (I've got 480,863 of them). Any suggestions?
You can also use:
cellfun(#length, strings)
It will not be faster, but makes the code clearer.
Regarding the slowness, you should first run the profiler to check where the bottleneck is. Only then should you optimize.
Edit: I just recalled that 'length' used to be a built-in function in cellfun in older Matlab versions. So it might actually be faster! Try
cellfun('length',strings)
Edit(2) : I have to admit that my first answer was a wild guess. Following #Rodin s comment, I decided to check out the speedup.
Here is the code of the benchmark:
First, the code that generates a lot of strings and saves to disk:
function GenerateCellStrings()
strs = cell(1,10000);
for i=1:10000
strs{i} = GenerateRandomString();
end
save strs;
end
function st = GenerateRandomString()
MAX_STR_LENGTH = 1000;
n = randi(MAX_STR_LENGTH);
st = char(randi([97 122], 1,n ));
end
Then, the benchmark itself:
function CheckRunTime()
load strs;
tic;
disp('Loop:');
for i=1:numel(strs)
n = length(strs{i});
end
toc;
disp('cellfun (String):');
tic;
cellfun('length',strs);
toc;
disp('cellfun (function handle):');
tic;
cellfun(#length,strs);
toc;
end
And the results are:
Loop:
Elapsed time is 0.010663 seconds.
cellfun (String):
Elapsed time is 0.000313 seconds.
cellfun (function handle):
Elapsed time is 0.006280 seconds.
Wow!! The 'length' syntax is about 30 times faster than a loop! I can only guess why it becomes so fast. Maybe the fact that it recognizes length specifically. Might be JIT optimization.
Edit(3) - I found out the reason for the speedup. It is indeed recognition of length specifically. Thanks to #reve_etrange for the info.
Keep an array of the lengths of said strings, and update that array when you update the strings. This will allow you O(1) time access to string lengths. Since you are updating it at the same time you generate or load strings, it shouldn't slow things down much, since integer array operations are (generally) faster than string operations.