I'm in need of optimizing import of .xls files to matlab due to xlsread being very time consuming with large amount of files. Current xlsread script as follows:
scriptName = mfilename('fullpath');
[currentpath, filename, fileextension]= fileparts(scriptName);
xlsnames = dir(fullfile(currentpath,'*.xls'));
xlscount = length(xlsnames);
xlsimportdata = zeros(7,6,xlscount);
for k = 1:xlscount
xlsimport = xlsread(xlsnames(k).name,'D31:I37');
xlsimportdata(:,1:size(xlsimport,2),k) = xlsimport;
end
I have close to 10k files per week that needs processing and with approx. 2sec per file processed on my current workstation, it comes in at about 5½ hours.
I have read that ActiveX can be used for this purpose however that is far beyond my current programming skills and have not been able to find a solution elsewhere. Any help on how to make this would be appreciated.
If it is simple to perform with ActiveX (or other proposed method), I would also be interested in data on cells D5 and G3, which I am currently grabbing from 'xlsnames(k,1).name' and 'xlsnames(k,1).date'
EDIT: updated to reflect the solution
% Get path to .m script
scriptName = mfilename('fullpath');
[currentpath, filename, fileextension]= fileparts(scriptName);
% Generate list of .xls file data
xlsnames = dir(fullfile(currentpath,'*.xls'));
xlscount = length(xlsnames);
SampleInfo = cell(xlscount,2);
xlsimportdata = cell(7,6,xlscount);
% Define xls data ranges to import
SampleID = 'G3';
SampleRuntime = 'D5';
data_range = 'D31:I37';
% Initiate progression bar
h = waitbar(0,'Initiating import...');
% Start actxserver
exl = actxserver('excel.application');
exlWkbk = exl.Workbooks;
for k = 1:xlscount
% Restart actxserver every 100 loops due limited system memory
if mod (k,100) == 0
exl.Quit
exl = actxserver('excel.application');
exlWkbk = exl.Workbooks;
end
exlFile = exlWkbk.Open([dname filesep xlsnames(k).name]);
exlSheet1 = exlFile.Sheets.Item('Page 0');
rngObj1 = exlSheet1.Range(SampleID);
xlsimport_ID = rngObj1.Value;
rngObj2 = exlSheet1.Range(SampleRuntime);
xlsimport_Runtime = rngObj2.Value;
rngObj3 = exlSheet1.Range(data_range);
xlsimport_data = rngObj3.Value;
SampleInfo(k,1) = {xlsimport_ID};
SampleInfo(k,2) = {xlsimport_Runtime};
xlsimportdata(:,:,k) = xlsimport_data;
% Progression bar updater
progress = round((k / xlscount) * 100);
importtext = sprintf('Importing %d of %d', k, xlscount);
waitbar(progress/100,h,sprintf(importtext));
disp(['Import progress: ' num2str(k) '/' num2str(xlscount)]);
end
%close actxserver
exl.Quit
% Close progression bar
close(h)
Give this a try. I am not an ActiveX Excel guru by any means. However, this works for me for my small amount of test XLS files (3). I never close the exlWkbk so I don't know if memory usage is building or if it automatically cleaned up when descoped after the next is opened in its place ... so use at your own risk. I am seeing an almost 2.5x speed increase which seems promising.
>> timeit(#getSomeXLS)
ans =
1.8641
>> timeit(#getSomeXLS_old)
ans =
4.6192
Please leave some feedback if this work on large number of Excel sheets because I am curious how it goes.
function xlsimportdata = getSomeXLS()
scriptName = mfilename('fullpath');
[currentpath, filename, fileextension]= fileparts(scriptName);
xlsnames = dir(fullfile(currentpath,'*.xls'));
xlscount = length(xlsnames);
xlsimportdata = zeros(7,6,xlscount);
exl = actxserver('excel.application');
exlWkbk = exl.Workbooks;
dat_range = 'D31:I37';
for k = 1:xlscount
exlFile = exlWkbk.Open([currentpath filesep xlsnames(k).name]);
exlSheet1 = exlFile.Sheets.Item('Sheet1'); %Whatever your sheet is called.
rngObj = exlSheet1.Range(dat_range);
xlsimport = cell2mat(rngObj.Value);
xlsimportdata(:,:,k) = xlsimport;
end
exl.Quit
Related
I am currently writing a lager script to ease my life.
Right now I am reading raw values from cells from an excel.
So far so good.
These numbers need to be interpreted as seconds and then converted into minutes.
I tried my best with datetime but no luck.
Any suggestions?
elif auswahl == '2':
print("Some user friendly-text:")
excel_download = openpyxl.load_workbook(r'/Path/to/excel.xlsx')
sheet = excel_download.active
Grund_1 = sheet['B2'].value
Grund_2 = sheet['B3'].value
Grund_3 = sheet['B4'].value
Grund_4 = sheet['B5'].value
Grund_5 = sheet['B6'].value
Grund_6 = sheet['B7'].value
Zeit_in_Sekunden_1 = sheet['C2'].value
Zeit_in_Sekunden_2 = sheet['C3'].value
Zeit_in_Sekunden_3 = sheet['C4'].value
Zeit_in_Sekunden_4 = sheet['C5'].value
Zeit_in_Sekunden_5 = sheet['C6'].value
Zeit_in_Sekunden_6 = sheet['C7'].value
print("Du warst heute für", Zeit_in_Sekunden_1, Grund_1, "!")
break
My idea:
raw_seconds_from_C2 = sheet['C2'].value
Then somehow convert to minutes from raw_seconds_from_C2
I am really out of ideas as I then need to put the converted minutes into a print().
Divide the value by 60 to obtain minutes from seconds:
c2_minutes = sheet['C2'].value / 60
Thanks to #Alonso's comment on the question.
I want to save pitch,yaw and roll data in excel file for all frames. Eg: if i have 200 frames then i want to save 200 frames information in excel file. I have tried but my code only stores one frame data.exceldata
fitting_model='models/Chehra_f1.0.mat';
load(fitting_model);
mov=VideoReader('7_a.avi'); %Read video file and create an object
c=mov.NumberOfFrames;
for k=1:c
a = read(mov, k);
img=im2double(a);
disp(['Detecting Face in ']);
faceDetector = vision.CascadeObjectDetector(); % detect face in an image
bbox = step(faceDetector, img); %create boundary box around face
test_init_shape = InitShape(bbox,refShape); %initialize facial points in variable
test_init_shape = reshape(test_init_shape,49,2);
if size(img,3) == 3
test_input_image = im2double(rgb2gray(img));
else
test_input_image = im2double((img));
end
disp(['Fitting']);
MaxIter=6;
test_points = Fitting(test_input_image,test_init_shape,RegMat,MaxIter);
load('3D_Shape_Model.mat');
n=49;
test_image=img;
imshow(test_image);hold on;
% % Compute 3D Head Pose
if(n==49)
test_shape=test_points;
[pitch,yaw,roll] = ComputePose(PDM_49,test_shape(:));
filename='framesdata.xlsx';
header = {'Pitch', 'yaw ','roll'};
new_data = num2cell([pitch(:), yaw(:), roll(:)]);
output = [header; new_data];
xlswrite(filename,output);
end
plot(test_shape(:,1),test_shape(:,2),'b*');
title([num2str(i),' : Pitch = ',num2str(pitch),' ; Yaw = ',num2str(yaw),' ; Roll = ',num2str(roll)]);
set(gcf,'units','normalized','outerposition',[0 0 1 1]);
pause(0.5);
close all;
end
As #excaza stated, you will need to move the xlswrite command out of your loop or specify the cells you are writing. Please see the xlswrite Doc for more information. The correct syntax would be :
xlswrite(filename,A,xlRange)
The following is the example they provide:
filename = 'testdata.xlsx';
A = {'Time','Temperature'; 12,98; 13,99; 14,97};
sheet = 2;
xlRange = 'E1';
xlswrite(filename,A,sheet,xlRange)
You will just need to provide xlswrite the address to start writing data.
I have a code in matlab. After I have run my program, a file 'example2.xlsx' was created.
Now I have the code below and I want matlab to replace the current 'example2.xlsx' by the new 'example2.xlsx' (saving automatically without asking me if I want to replace it):
e = actxserver ('Excel.Application'); % # open Activex server
filename = fullfile(pwd,'example2.xlsx'); % # full path required
ewb = e.Workbooks.Open(filename); % # open the file
esh = ewb.ActiveSheet;
str = num2str(num_rows+1);
esh.Range(strcat('J',str)).Interior.Color = clr;
sheet1 = e.Worksheets.get('Item', 'Sheet1');
range1 = get(sheet1,'Range', strcat('A',str),strcat('I',str));
range1.Value = values{num_rows+1};
[num, txt, raw] = xlsread('example2.xlsx');
num_rows = length(num(:,1));
xlWorkbookDefault = 51; % # it's the Excel constant, not sure how to pass it other way
ewb.SaveAs(fullfile(pwd,'example2'), xlWorkbookDefault)
ewb.Close(false)
e.Quit
e.delete
You can set the DisplayAlerts property of the Excel application object to false to stop these dialogs from appearing.
The following is a simplified version of your code:
e = actxserver ('Excel.Application'); % # open Activex server
filename = fullfile(pwd,'example2.xlsx'); % # full path required
ewb = e.Workbooks.Open(filename); % # open the file
esh = ewb.ActiveSheet;
sheet1 = e.Worksheets.get('Item', 'Sheet1');
range1 = get(sheet1,'Range', 'A1');
range1.Value = 3;
set(e, 'DisplayAlerts', 0); % # Stop dialog!
xlWorkbookDefault = 51; % # it's the Excel constant, not sure how to pass it other way
ewb.SaveAs(fullfile(pwd,'example2'), xlWorkbookDefault)
ewb.Close(false)
e.Quit
e.delete
this code has to color the J1 cell by a given rgb.
row_number_excel = 1;
representative_red = 205;
representative_green = 211;
representative_blue = 201;
headers = {'J'};
rgb = [representative_red representative_green representative_blue]; %# if you have 0 to 1 values multiply by 255 and round
clr = rgb * [1 256 256^2]'; %# convert to long number Excel understands
pwd = 'D:\grapes\main';
e = actxserver ('Excel.Application'); % open Activex server
filename = fullfile(pwd,'example.xls'); %# full path required
if exist(filename,'file')
ewb = e.Workbooks.Open(filename); %# open the file
else
error('File does not exist.') %# or create a new file
end
esh = ewb.ActiveSheet;
for c = 1:row_number_excel
str = num2str(row_number_excel);
esh.Range(strcat(headers{1},str)).Interior.Color = clr;
end
ewb.Save
ewb.Close(false)
e.Quit
I tried to run this code, but the cell was colored by a different color than the given rgb. When I ran the code in the second time, a file was created in my directory, named: "2E60F720". its type is 'file'. Then the program ran and ran and wasn't stopped till I stopped 'EXCEL.EXE' by the task manager. After that the matlab wrote me this:
"??? Error: The remote procedure call failed.
Error in ==> test1 at 212 ewb.
Close(false);"
can someone help me please?
I captured the screen. in the left side, the program is still running, in the right side, this is my directory. I marked the file that was created.
thanks!
the solution that solved my question is:
row_number_excel = 1;
representative_red = 205;
representative_green = 211;
representative_blue = 201;
headers = {'J'};
rgb = [representative_red representative_green representative_blue]; %# if you have 0 to 1 values multiply by 255 and round
clr = rgb * [1 256 256^2]'; %# convert to long number Excel understands
pwd = 'D:\grapes\main';
e = actxserver ('Excel.Application'); % open Activex server
filename = fullfile(pwd,'example.xls'); %# full path required
if exist(filename,'file')
ewb = e.Workbooks.Open(filename); %# open the file
else
error('File does not exist.') %# or create a new file
end
esh = ewb.ActiveSheet;
for c = 1:row_number_excel
str = num2str(row_number_excel);
esh.Range(strcat(headers{1},str)).Interior.Color = clr;
end
xlWorkbookDefault = 51; %# it's the Excel constant, not sure how to pass it other way
ewb.SaveAs(fullfile(pwd,'example2'), xlWorkbookDefault)
ewb.Close(false)
e.Quit
thanks to #yuk!
The trouble line is saving the file with ewb.Save.
Your file is actually in the old format (excel 2003), not sure if it's support true RGB colors, it might be the reason.
I'd recommend to save the file with a new name and the latest format. Put this as the last lines:
xlWorkbookDefault = 51; %# it's the Excel constant, not sure how to pass it other way
ewb.SaveAs(fullfile(pwd,'example2'), xlWorkbookDefault)
ewb.Close(false)
e.Quit
I am using MatlabR2011a on my Windows 7 machine.
I have a folder of 635 text files. Each file contains multiple rows and 2 columns. I am trying to loop through the folder and read each file and write these values to excel. So far this is what I have:
close all; clc;
dirname = uigetdir;
Files = dir(fullfile(dirname,'*.txt'))
for k = 1:635
j =1
filename = fullfile(dirname,Files(k).name);
fid = fopen(filename);
x = fgetl(fid)
while ischar(x)
x = fgetl(fid)
a2 = strtrim(x);
a3 = textscan(a2,'%s');
a4 = a3{1}{1};
a5= a3{1}{2};
pos3 = strcat('A',num2str(j));
xlswrite('sample_output',{a4,a5},'Sheet1',pos3)
j = j+1;
end
fclose(fid);
end
It keeps giving me the following error after finishing reading the first file.
Error using ==> strtrim Input should be a string or a cell array of strings.
A sample input file looks like this:
15076 4636259
15707 4636299
15714 1781552
15721 4204950
15730 2174919
16209 4636510
16413 4758572
16470 4445808
17519 311397
17667 2116489
17739 1729694
18024 3210756
18627 3714194
18695 4192858
19141 632766
19318 1923574
19438 1255216
19493 4635020
19771 4770250
How can I fix this? Would appreciate help with this!
In the while loop, cut the line
x=fgetl(fid)
...and paste it before the end statement right after j=j+1.
The error occurs because you get a line before the while statement, then you use the while statement to test validity of the string (all fine so far), but then immediately get the subsequent line before you perform your string operation. The call to fgetl at the start of the while block could return an EOF, causing subsequent string manipulation functions to fail.
Also... The code would be more robust if you set your for loop like so
for k=1:numel(Files)
Since all your data are numeric this should work. Give it a try.
dirname = uigetdir;
Files = dir(fullfile(dirname,'*.txt'))
j =0;
for k = 1:numel(Files)
filename = fullfile(dirname,Files(k).name);
x = dlmread(filename,'\t'); %# I assume tab-delimiter
j = j + size(x, 1);
xlswrite( 'sample_output', x, 'Sheet1',sprintf('A%d',j) )
end