We have a lot of reports that are generated via VBA & Excel. Only a small percentage of the reports are actual calculations - the majority of the work is sql calls and formatting/writing of cells. The longest of which takes several hours, the majority takes around 20-30 mins each.
The VBA/Excel code plugs into a dll that the VB6 desktop apps use - it's here that all the sql calls are made. While I am sure that there is room for improvement here, it's not this that concerns me - the desktop apps are fairly snappy.
Two VBA functions are used in abundance: These are called GetRange and SetupCell and they nearly always appear together. The GetRange function is a wrapper for the Excel.Range object. It takes a sheet, and 4 values for the extents of the range. Its main use is to pick the cell for editing. There doesn't appear to be much chance of optmising it, but is it the best way?
Its partner is SetupCell. This takes a Excel.Range object, text and a dozen parameters about the cell (font, borders, etc). Most of these parameters are optional booleans but again, it seems very wasteful. Some of these can be set posthumously but some are dependant on the values contained in the cell.
There's quite a lot of code contained in these functions, mainly if statements and work won't appreciate me posting it.
I guess I've got two questions: Is there a better way and what is it and is there are free profiler that I can use to see if the bulk of the time is here or in the dll?
several hours is ridiculous for a report.
If the problem is VBA buy "Professional Excel Development" (stephen Bullen, Rob Bovey et al): this has a free VBA profiler called PerfMon.
If the problem is Excel Calculation see http://msdn.microsoft.com/en-us/library/aa730921.aspx?ppud=4
But I would guess that the problem is the high overhead associated with referencing things cell-by-cell: you should always work in large blocks of cells at a time.
Have you thought about using an actual reporting solution? What's your backend db? If you are using MSSQL 2000 or higher there is a fairly decent reporting solution you can use free of charge. SQL Server Reporting Services.
It sounds as if the reports are spending most of their time formatting cells. This could be why the reports seem so slow and the desktop app doesn't.
Alternatively, if you know the formatting before hand and it is fairly static, you could pre-format the sheets to cut down on some of the work.
I will throw this in there as well. Most reporting solutions will allow for conditional formatting and such, but since they are designed to work as such performance will be much better than having Excel do it.
This isn't a profiler recommendation, but it is a suggestion for speeding up Excel macros that are spending their time updating the screen. I've had excellent results by turning off screen updating while the macro is running: set Application.ScreenUpdating= False, and also using a number of other similar settings. Just be sure to turn them back on again when the macro finishes :P
It's not free but you can profile with this. I suspect the demo will be adequate to your needs: http://www.aivosto.com/vbwatch.html
It sounds like the VBA code (or the VB code that's writing to the sheets) is doing so line by line, this can take ages, and is poor design. Write to Excel as a variant in one go. Format the sheet after the data is all imported.
Thanks
Ross
Related
So I am terrible with excel and my current job requires me to comb through a ton of data. Here's what I am trying to do... I have about 40 pages of data in an excel sheet - it contains about 200 different programs and what each is capable of doing. Then, I have a separate list of people's problems that our programs may be able to help solve. Both lists are written in paragraph format in a word doc, but I have moved it to excel.
I am trying to figure out a way to match just the name of the program to just the title of the problem, by searching/comparing the description of the problem with the description of our program. In the past, doing this manually has taken nearly 80 manhours and it just seems like a waste.
Is there even a way to do this?
How difficult would it be for an excel novice to do, seeing as it's on a standalone system - where I can't copy/paste here?
Thanks in advance for any help/advice. I tried to include examples.
Customer problem
Potential solution
I have a file with the usual sumifs and vlookups. Nothing complicated computationally!
It has a lot of conditional formatting
As I remove the formulas the wait cursor remains there for shorter and shorter.
When I remove all the formulas and the formatting the file is fine as in I do not get the wait cursor. What is left is a file with no calculations and flat data.
Even with calculations that are just summing I am getting the wait cursor
I have tried excel open and repair. This does not work.
My sumifs have been changed to refer to a finite array as opposed to whole columns (which are also finite for the pedantic! but say these are infinite)
I think what I want to know, is what is known to cause a wait cursor to appear in excel? What can I do to minimize the time that the wait cursor appears?
I have looked on google and am not finding anything useful or relevant to my problem
I am starting to realise that wait cursor seems to always appear for a split second when I insert a column! however with my few sumifs formulas the wait cursor is getting longer and longer. Is it possible for sumifs formulas to calculate quickly but add to the wait cursor time the more sumifs you add?
Old question :
I have an excel file that keeps freezing
It calculates and saves very quickly. it is only 3090kb
Every time I insert a column or add a sheet for example I get the egg timer and if I click it starts to say 'not responding'
It only happens with this file so I don't believe it is my computer or excel version
the file was working fine before. I extended the formulas in about ten tabs (although not that many calcs added) and then it started to freeze
even if I delete every tab so there is just a blank sheet in the file it still freezes
Please help. Thanks
Please do not answer yet, I am still investigating and will get back to this. thank you
EDIT: The WAIT cursor not egg timer
Check if there are any hidden sheets with the data that is slowing it down.
Right click on sheet tab and unhide.
Make this practice to use this Macro or set Row Height to 15 so they don't Wrap. Wraptext in large excel files causes it to slow down to the point of freezing:
activesheet.cells.wraptext = false
#Jasmine - I wasn't suggesting that you are making anything up, nor was I my comment intended to be facetious. My point was that your question sounds self-contradictory to me.
When learning a new coding language, learning which questions to ask and how to phrase and share your problem is a skill in itself.
The fact that your file functions (a.k.a. calculates) 'properly'
until more identical formulas are added, sounds to me like it's no longer calculating properly, so I was unsure what you meant.
As for the `size doesn't matter' comment, it's another valid point about how some people ask questions when mistakenly thinking that file size has any significance. I've encountered and solved thousands of [seemingly unique] issues over the years but the last time I looked at the size of my file would have been back when it would've had to fit on a diskette.
Similarly, I often see questions here where the "asker" includes irrelevant statistics like the number of records in their database, as if you need to code differently for 10 records versus 10,000,000 (which isn't generally true, btw). I can't help but chuckle at some questions where the O.P. spends more time describing the formatting of a cell than the formula it contains (that's causing the issue). But from their point of view, it's significant.
Think of the people reading your question as sitting in your office blindfolded while you explain your problem. If you're asking for (free) help from others who cannot see what you can, with zero previous knowledge, does it not make sense to make a question absolutely as clear as possible -- even if that involves Googling a word to update your terminology that you know is incorrect? (or is the onus on the reader to figure out what you mean?
**Did you click the links I posted for you (above) to learn how to better communicate your problem and edit your question to add more specific examples and more information as requested and described in those links? (or was that a "waste of time" as well?) There were specific questions for you as well.
This site was frustrating for me when I first joined too but the rules and posting etiquette were not made immediately known to me by a stranger with a few seconds to spare. I had a major learning curve and some pretty nasty comments from people since I was breaking rules I wasn't even aware of. I am actually breaking a major rule right now, can you identify it perhaps by familiarizing yourself with the site in the Help Center?
Hint: I'm probably going to get "scolded" for the method I am using to give you this advice as opposed to the short comments and links I would normally post on a question that needs more information so I or others can give an informed answer. Some people don't like posting a question and not getting a custom written set of code promptly written just for them. Some people are grateful for every bit of advice they can get from a stranger, others never come back and find a site more suited to their needs.
Remember, there is a ton of existing information on this site and others because it's very unlikely that whatever problem you're having has been experienced - and solved - by many, many people before you. By researching as much as possible, you learn both about your problem and about which questions get "better" answers than others.
Anyhow I hope you read fast, this answer might not be here long (I wanted to be clear but without a whole whack of comments back and forth...)
Jasmine, if you're still angry with me, you can "get me back" if you can find the correct Flag button. The answer to my question for you in is there too... (It's not option #2, as lest not intentionally!)
Good luck & happy holidays to all!
So as many others have asked in the past is there a way to beat the 32k limit per cell in Excel?
I have found ways to do it by splitting the work load into two different .txt files and then merging the two .txt files, however it is a giant PITA and more often then not I end up only using excel to its limits as I do not have time to validate the data after .txt file merges anymore this is a long process and tedious IMO.
However I think that if the limitation is there it is there because it was coded when Microsoft developed Excel, and since they have yet to raise it (2013 version the limit is still the same limit so it would do no good to upgrade)
I also know that many will say if you have a need for information in a single cell in that length then you should use ACCESS well I have no idea how to use ACCESS or how to import a tab delimited file into ACCESS like you would into EXCEL, and then even if I could figure that out I still now have to figure out how to learn all the new commands and he EXCEL equivalents if there is even such a thing.
So I was browsing some blog posts the other day on how to beat limitations by software and I read something about reverse engineering.
Would it be possible to load excel into a hex editor, go in and change every instance of 32767 to something greater?
While 32767 may seem like an arbitrary number, it's actually the upper limit of a 16-bit signed integer (called a short in C). The range of a short goes from -32768 to 32767.
A 16-bit integer can also be unsigned, in which case its range is 0 to 65535.
Since it's impossible for a cell to have a negative number of characters, it seems odd that Microsoft would limit a cell's length based on a signed rather than unsigned 16-bit integer. When they wrote the original program, they probably couldn't imagine anyone storing so much information in a single cell. Using shorts may have simplified the code. (My first computer had only 4K of memory, so it's still amazing to me that Excel can store 8 times that much information in a single cell.)
Microsoft may have kept the 32767 limit to maintain backward compatibility with previous versions of Excel. However, that doesn't really make sense, because the row and column counts greatly increased in recent versions of Excel, making large spreadsheets incompatible with previous versions.
Now to your question of reverse-engineering Excel. It would be a gargantuan task, but not impossible. In the early '90s, I reverse-engineered and wrote vaccines for a few small computer viruses (several hundred bytes). In the '80s, I reverse-engineered an 8KB computer chess program.
When reverse-engineering an executable, you'll need a good disassembler or decompiler. Depending on what you use, you may get assembly-language or C code as the output. But note that this will not be commented code, and you will not see meaningful variable or function names. You'll have to read every line of code to determine what it does. And you'll quickly discover that the executable is the least of your worries. Excel's executable links in a number of DLL files, which would also need reverse-engineering.
To be successful, you will need an extensive knowledge of Windows programming in addition to C or Intel assembly code – not to mention a large amount of patience. Learning Access would be a much simpler task.
I'd be interested in why 32767 is insufficient for your needs. A database may make more sense, and it wouldn't necessarily need to duplicate the functionality of Excel. I store information in a database for output to Web pages, in which case I use HTML+JavaScript for anything that needs to be interactive.
In case anyone is still having this issue:
I had the same problem with generating a pipe-separated file of longitudinal research data. The header row exceeded the 32767 limit. Not an issue unless the end-user opens the file in excel. Work around is to have end-user open file in google sheets, perform the text-to-columns transformation, then download and open file in excel.
https://support.clarivate.com/ScientificandAcademicResearch/s/article/Web-of-Science-Length-limit-of-cell-contents-in-Excel-when-opening-exported-bibliographic-data?language=en_US
Jack Straw from Wichita (https://stackoverflow.com/users/10327211/jack-straw-from-wichita) surely you can do an import of a pipe separated file directly into Excel, using Data>Get Data? For me it finds the pipe and treats the piped file in the same way as a CSV. Even if for you it did not, you have an option on the import to specify the separator that you are using in your text file.
Kind regards
Sefton Hall
My company is using this program called Visma Contracting. This program is used to type inn a product number (example: 10 240 75) and how many of this item was used (example: 4). So these numbers we (me and my co-workers) put into an excel sheet and deliver to the guy in charge of the systematizing these (A column being product number, an B column being the used amount). From there it is retyped from excel to Visma. This is madness! There must be a way for the two programs to talk to each other? I have talked to the Visma support and they are giving me nothing else then a no. I wish i could give more info about this Visma stuff, but i fear that it is a locked program. I have also been searching around for a 3rd party software that can eliminate this massive annoying problem, with no luck. Does anyone have anything that might ease my itch?
Thanks in advance!
Sounds impossible but...
If you want to try blind keystrokes check out Mouse and Keystroke Recorder I have some experience automating stuff with this program. It works sometimes with varying degrees of reliability.
Be forewarned that nobody recommends this as it could cause problems. It simply plays back keystrokes without being aware of what it is doing. When used with care it can work but it could be dangerous.
Or use SendKeys from Excel VBA; that might work better as the data is already in Excel. But the same warnings apply. Use at your own risk.
We are having problems with RExcel and I am trying to find alternatives. Our users like to use Excel and our developers like to do stats in R. So, we use Excel as a front end for data entry, analyze in R, and present results in Excel.
Because of the problems we are having with RExcel, however, I would really like to have a different solution. Assuming that we cannot remove Excel or R from the equation, are there other ways of connecting them besides RExcel? My searching on the internet has not been helpful.
[edit]
I've been asked to phrase this as a problem that SO can solve. I need to take data input in Excel, pass it to R, and get data back. RExcel is not allowed and the transfer needs to be transparent to the user. How do I do this?
Perhaps such an alternative is writing functions for Excel in C# with ExcelDNA and R.Net libraries.
Some info:
https://github.com/Excel-DNA/Samples/tree/master/UsingRDotNet
How to get ExcelDNA work with R.Net