Looped .Refresh causing slow Excel - excel

I am trying to use VBA in Excel for Mac to query a database. A simple query works fine and I've been using it for years. Now I'm getting into more complex queries, where the results of one query (about 1,000 records) are used sequentially to query the database again (so about 1,000 consecutive queries). The results (about 5,000 records) return in about 2.5 minutes, but clicking in the Excel sheet is not responsive for about a minute afterward. This behavior continues for another 2 minutes or so before clicking becomes pretty much instantaneous. Running the macro again gives similar results but slower. The 3rd time is even slower. I suspect a memory leak. Restarting Excel makes the problem reset. Here is the code for the actual query:
With ActiveSheet.QueryTables.Add(Connection:=strURL, Destination:=Range(strStartCell))
.PostText = "user=" & strUserName & ";password=" & strUserPassword
.RefreshStyle = xlOverwriteCells
.SaveData = True
.BackgroundQuery = False
.Refresh
End With
I've tried to actually send just the first query (so it gets data from which the other queries can be built). The rest of them, I build the query, but don't send it. So, in this experiment, the above code got used only once. Running it this way, it comes back in about 8 seconds. So the other 2 minutes and 20+ seconds are back and forth between my computer and the database. More importantly, after running this way, there is no lag after it is done running. So it seems like if it is a memory leak, the leak is in the query process, or maybe the actual writing of the data.
I have programmatically turned off all of the screen updating, page break showing, and calculating at the beginning and returned them to the original settings at the end.
My computer is a Mac mini (Late 2014) 3GHz Intel Core i7 with Office 2011, but I've tried running it on a newer M1 with the newest version of Excel also. It was much faster, but the lag after the results were returned, though shorter, was still a problem. My computer is representative of where the spreadsheet will be run for the near future.
The lag afterwards is really going to kill this part of the project. Has anybody seen this problem before? Is there something I can do to trace what is causing the problem and if there is a way around it?

ChatGPT solved the crux of my issue. It pointed out that I'm creating a separate Query Table for each of my 1000+ queries and those are eating up my memory, causing my application to become slow. It said that if I add a .Delete line after the With... End With block it would wipe out the previous Query Table and not use up all that memory. The answer was slightly inaccurate. The .Delete needed to go inside the block (at the end). Anyhow, I tried that and it made my code somewhat slower, but when it was done, the application was not slow. That is a win in my book.
It further suggested that ADO might do a better job at this task. So I guess that is something else to learn. Thought this might be useful for someone else experiencing a similar issue.

Related

Halting macro code execution until SQL query is completed

I have written a fairly complex application that uses Excel (2016) as front-end application for data stored in a MS Access database, using the ADO 6.1 library. I have noticed during macro code execution that SQL transactions triggered by my Excel application can take quite long to complete, and often the execution of the next line of code in my Excel macro depends on this SQL transaction first being completed. Unfortunately, macro code execution and SQL transactions are asynchronous operations, which means that the macro jumps to the next line of code even though the SQL transaction hasn't been completed.
My current work-around is to use a Sleep() function using the Windows API to insert a fixed delay, but this is a really ugly solution as it does reduce the performance of my application and depends very much on the CPU load, so it may sometimes work, sometimes not.
I haven't been able to find so far a solution to this problem, I can't find any hints on the Internet either.
Using Application.CalculateUntilAsyncQueriesDone doesn't help here either.
Does anyone have an idea or a hint how to halt macro code execution in Excel until an SQL transaction has been completed? Is there a method in ADO to check the completion of an SQL transaction?
Is your query within the Data/connections section?
I had this problem to, I turned off "Enable Background refresh" and added "DoEvents" to the VBA, this forces the data connection to refresh before it allows the code to continue. Downside to this is it makes excel feel like its locked up, But this resolved my issue.
Sub Button1_Click()
ActiveWorkbook.Connections("ScrapData").Refresh
DoEvents
....Other code....
End With

Getting reasonable performance from excel while receiving RTD feeds and avoid blocking

I am handling RTD feeds again and remembering the difficulties, but now we have multi-core machines and multi-threading. maybe anyone can advise.
As I understand/rememeber: pushing data into Excel is not one(obvious reasons) so it sends a friendly nod to say your parcel is ready come and get it. Then when Excel has done its nails and feels in the mood, it might get the data.
So this kind of architecture is right back on the dance-floor and hopefully I can make it work.
Problem.
I need to run a bot that examines the data and responds differently;
1. looping or running a winapi timer even is still enough to keep excel busy and no data, or so it seems.
Definitely, executing bot logic , however small, will bring on an Excel fainting fit.
Tried responding via calculation event. Very hit-and-miss and definitely not up to the job. There is no logic obvious as to when and why it fires or does not other than a " bad hair day"
Tried winapi timer looking at the newly acquired data every second comparing to old in a separate data structure, running some EMAs and making a decision. No dice.
The timer is enough to put a delay up to 10 or even 20 seconds between occasional delivery of data.
Options I am thinking about:
1. The timer running outside of the excel environment and lookin in a the data. e.g an AQddon via pia etc. What I don't know is whether this Addon, perhaps inc C# or vb.net, could utilze multithreading, via tasks I think and do its bit without "scarin the panties of her ladyship"?
2. I remember hearing that XLL UDFs could be asynchronous, does anyone know if this is a potential option?
Any ideas?

Excel automatic calculation crashes, but manual calculation works

I have a relatively large (7MB) but more importantly formula and macro heavy excel file that I have been developing recently.
For the end user it must be able to be used in Automatic Calculation mode.
While calculating the whole sheet in Manual Calculation mode using F9 (Calculate Now) will work and calculate in ~10 seconds, but when I turn it onto Automatic mode, it will calculate repeatedly (i.e reaches 100% and then immediately starts again from 0%) and so freezes indefinitely.
So far I have tried:
Putting break points in all the VBA macros to see if it is hanging inside a macro
Removing all of the macros from the file (I was worried one of them was being triggered to run repeatedly in automatic mode)
but neither has worked leading me to wonder if the issue is not in fact VBA related.
Does anyone have any ideas about:
What might be causing this?
How I might diagnose the cause?
Happy to give more context if helpful. I am a relatively experienced excel user, and while generally don't write VBA macros from scratch am pretty confident at re-purposing code I inherit / find online. I am using a relatively powerful 6 core machine, but have tested it on others to the same results.
The issue turned out to be a volatile action that a macro was triggering. I had a macro that looped through hiding empty rows, but hiding a row is volatile, so then triggers a new calculation before the next run through the loop. The loop itself was 500 iterations, so that meant 500 sets of 3 second calculations!
See this link for more on volatile actions:
https://msdn.microsoft.com/en-us/library/bb687891(v=office.15).aspx

Slow performance in Mongoose after upgrading from 3.5.7 to 3.8.4

I changed the version number of mongoose from 3.5.7 to 3.8.4 and performance took a huge hit in an import process. This process reads lines from a file and populates an empty database (no indexes) with about 2.5 million rows.
This is the only change I've made; just upgrading the version. I can switch back and forth and see the difference in performance.
The performance hits are: 1) The process pegs at 100% CPU, where before it ran upwards of maybe 25% or so. 2) Entry into the database into the database is slow. 3.5.7 inserted about 10K records every 20 seconds while 3.8.4 seems to be inserting at more around 1 per second. 3) nodejs seems to "disappear" into something CPU intensive and all other I/O functions get blocked (http requests, etc.); previously the system remained very responsive.
It's hard to isolate the code, but roughly here's what's happening:
Read a line from a file
Parse it
Run a query to see if the record already exists
Insert/update a record with the values from the line read
Write the existing record to an already-open file stream
Continue with the next line in the file
At a guess, I would say it's related to a change in how requests are throttled either in the underlying driver that mongoose depends on or mongoose itself. My first thought was to try and handle the case where requests are getting queued up and put a pause on the file read. This works really well when writing the results of a query (pausing the querystream when the file starts caching writes, then resuming on drain). But I haven't been able to find where mongoose might be emitting information about its back-pressure.
The reason I upgraded in the first place is because of a memory leak error I was getting when setting an event handled in mongoose that I had read was fixed (sorry I lost reference to that).
I'd like to stay upgraded and figure out the issue. Any thoughts on what it might be? Is there an event given somewhere that notifies me of back-pressure so I can pause/resume the input stream?
I solved this by simply reverting back to 3.5.7 and solving the original memory leak warnings another way.
I attempted to isolate my code, but the worst issue I was able to raise was high memory consumption (which I resolved by nulling objects, which apparently helps the GC figure out what it can collect). I started adding in unrelated code, but at some point it became clear that the issue wasn't with mongoose or the mongodb driver itself.
My only guess on what really causes the performance issue when I upgrade mongoose is that some library depended on by mongoose introduced a change that my non-mongoose-related code isn't playing well with.
If I ever get to the bottom of it, I'll come back and post a clearer answer.

Too much data (at a time) for Core Data?

My iPhone app uses core data and things are fine for most part. But here is a problem:
after a certain amount of data, it stalls at first time execution (where core data entities must be loaded).
Some experimenting showed that things are OK up to a certain amount of data loaded in Core Data at start.
If I go over a critical amount the installation starts failing. The bigger the amount of data for start, the higher the probability that it fails.
By making separate tests I made sure the data themselves are not faulty.
I also can say this problem does not appear in the simulator.
It also does not happen when I connect the debugger to the device.
It looks like too much data loaded in core data in a short amount of time creates some kind of overload.
Is that true? Any idea on a possible solution?
At this point I made up a partial solution using a UIActionSheet object to kill some time (asking the user to push a button). But this is not very satisfactory, though for the time being it works.
Any comment or advice for a better way would be appreciated.
It is not quite clear what do you mean by "it fails".
However if you are using SQLite, by loading into CoreData, if you mean "create and save" entities at start up to populate CoreData, then remember to not call [managedObjectContext save...] only at the end especially with large amount of data, but create and save a reasonable set of NSManagedObject.
Otherwise, if you mean you have large amount of data that are retrieved as NSManagedObject, probably loaded into a UITableView consider using some kind of NSOperation for asynchronous loading.
If those two cases doesn't apply to you just tell us the error you are getting, or what you mean by "fails" os "stalls".

Resources