My Excel applications exports data (6 columns by 300+ rows) to Access DB once a minute.
After exporting, I try to import two sets of older values from Access to Excel.
The primary query is:
SELECT qryStDevPct.StDevPct, atblExcelIntraday.ValueClose
FROM atblExcelIntraday
LEFT JOIN qryStDevPct
ON atblExcelIntraday.SecurityID = qryStDevPct.SecurityID
WHERE atblExcelIntraday.RecordDate Between #2022-05-04 13:55:59# And #2022-05-04 13:54:59#
ORDER BY atblExcelIntraday.Ticker;
The internal Access qryStDevPct is:
SELECT atblExcelIntraday.SecurityID,
Avg(atblExcelIntraday.ValueClose) AS AvgP,
StDev(atblExcelIntraday.ValueClose) AS STD,
[STD]/[AvgP] AS StDevPct
FROM atblExcelIntraday
INNER JOIN atblSecuritiesOpenPriceDaily
ON atblExcelIntraday.SecurityID = atblSecuritiesOpenPriceDaily.SecurityID
GROUP BY atblExcelIntraday.SecurityID;
The Price data from table always imports correctly, but StDevPct is either 0 or 1 30% of the time.
I can easily refresh it by doing manual "Data Refresh", but neither .Calculate, nor .QueryTables("qryPriceDB_1").Refresh work.
I could split it it two separate queries, but aside from programming it will take more time on the Excel end, and I'm trying to keep my main procedure to under 1,500 msec.
Please advise.
Related
I am supposed to optimize the performance of an old Access DB in my company. It contains several tables with about 20 columns and 50000 rows. The speed is very slow, because the people work with the whole table and set the filters afterwards.
Now I want to compose a query to reduce the amount of data in Excel before transfering the complete rows, but the speed is still very slow.
First I tried the new power query editor from Excel. I first reduced the rows by selecting only the last few ones (by date). Then I made an inner join with the 2nd table.
Finally I got less than 20 rows returned, and I thought I was fine.
But when I started Excel to perform the query, it took 10 - 20 seconds to read the data. I could see, Excel loads the complete tables, before setting the filters.
My next try was to create the same query direcly inside the Access DB, same setting. Then I opened this query in Excel, and the time to load the rows is nearly zero. You select "refresh", and the result is shown instantly.
My question is: Is there any way to perform a query in Excel only (without touching the Access file), that is nearly as fast as a query in Access itself?
Best regards,
Stefan
Of course.
Just run an SQL query from MS Query in Excel. You can create the query in Access, and copy-paste the SQL in MS Query. They're executed by the same database engine, and should run at exactly the same speed.
See this support page on how to run queries using MS Query in Excel.
More complex solutions using VBA are available, but shouldn't be needed.
Suppose I have two tables, TableA(embedded data), TableB(external data).
Scenario 1:
TableB is set On-Demand based on the markings from TableA. When you mark something from TableA, it take some "n" seconds to populate the data in TableB. On-Demand setting on external table is like screenshot named LOD.png
Scenario 2:
On-Demand settings have not been induced on TableB(please note TableB still is External). There has been a relationship created between TableA and TableB. TableB is now limited based on marking from TableA by the option"Limit data using Markings".screenshot named ss2
Questions:
1. Which scenario fetches data quicker.
2. From the debug log, the query passed in both the scenario is the same.Does that mean both scenarios are same or are they different?
Scenario 1 is good if Table B is really large, or records take a long time to fetch from the database. In this case, Table B is only returning rows that are based on what you marked in Table A. This means that the number of rows could be significantly less, but this also means that every time the marking changes, those rows have to be fetched at that time. If the database takes a long time, this can become frustrating. On the flip side, if the database is really fast and you are limiting rows down enough, this can be almost seamless. At the end of the day, you are pulling this data into memory after the query runs, so all Spotfire functionality is available.
Scenario 2 is good if calculations are highly complex and need to take advantage of the power of the external DB to perform. This means that any change to the report, a change of visualization etc., will require a new query to be sent to the external data source resulting in a new table of aggregated data. This means that no changes to a visualization using an in-db data table can be made when you are not connected to the external data source. Please note, there is functionality in Spotfire that is available to in memory data like data on demand that is not available to external data.
I personally like to keep data close to Spotfire to take advantage of all Spotfire functionality, but I cannot tell you exactly which is the correct method in your case. Perhaps these TIBCO links on the difference between in memory data and external data can help:
https://docs.tibco.com/pub/spotfire/6.5.1/doc/html/data/data_overview.htm
https://docs.tibco.com/pub/spotfire/6.5.1/doc/html/data/data_working_with_in-database_data.htm
I am very new to microsoft access. I work in an Immunization program where I routinely collect data about vaccinated children. I used to have excel spreadsheets (different spreadsheet for every campaign) but as the number of spreadsheets grew, comparison between data of different campaigns became difficult.
Now I am trying to get all the data into a database program in which I can bring data from multiple campaigns into a single report easily.
After jumping into access, first I need to get the basic things done that excel could do very easily. e.g.
This is sample data from day 1 and day 2 of the campaign. In access I can create a query which shows data from day 1 only and a totals row at the end. But how do I create a single query/report which shows separate totals row for each day. In other words, how do I reproduce the data in pictures above in a single access report.
Edit:
I am planning a single table that contains data from all the campaigns in various columns.
The table in microsoft access looks like this:
Link to the access database file:
link to access database file
June7 has provided the correct advice - though it is cryptic.
Your table structure appears correct - you want a single table with the identity of the campaign and date (or whatever parameters you seek to differentiate them).
Your query is to collect those records into a record set. (Although there is an aggregate query type - it is not meant for what you seek to do.) You don't want to report on the entire database each time probably so one uses a query rather than the table itself to be the record set of the report. How that query is delimited is up to you.
The report object is where one then groups - and a group can have a total. You will want to look into online/text instructions on this for actual implementation.
Business Case:
I have a list of key IDs in an excel spreadsheet. I want to use Power Query to join these IDs with a details table in a SQL Server database.
Problem
Currently using Power Query I only know how to import the entire table, which is greater than 1 million records, then do a left join on it against an existing query that targets a local table of IDs.
What I want to do is send that set of IDs in the original query so I'm not pulling back the entire table and then filtering it.
Question
Is there an example of placing an IN clause targeting a local table similar to what is shown below?
= Sql.Database("SQLServer001", "SQLDatabase001",
[Query="SELECT * FROM DTree WHERE ParentID
IN(Excel.CurrentWorkbook(){[Name="tbl_IDs"]}[Content])"])
I would first build a "Connection only" Query on the excel spreadsheet key IDs.
Then I would start a new Query by connecting to the SQL table. In that query I would add a Merge step to apply the key IDs query as an Inner Join (filter).
This will download the 1m rows to apply the filter, but it is surprisingly quick as this is mostly done in memory. It will only write the filtered result to an Excel table.
To improve performance, filter the rows and columns as much as you can before the Merge step.
I'm using an up-to-date version of Excel 2016 (via O365 E3 license) and using Power Query / Get & Transform Data. I can successfully create queries and load them to the page. I have also successfully created Power Pivot reports.
I would like to query single data points from the data loaded via Power Query. For instance, imagine a dataset called DivisionalRevenue with:
Date Division Revenue
2016-01-01 Alpha 1000
2016-01-02 Alpha 1500
2016-01-01 Beta 2000
2016-01-02 Beta 400
I could easily load that to an Excel workbook or include it in the data model and create a power pivot. However, Power Pivot doesn't always meet my requirements, particularly around how the data is displayed on the page. In order to achieve my goal I may want to be able to query individual data points.
I would like to have a cell on the page with a formula in it that I can use to query individual data points. If it was in a pivot table I could use something like:
=GETPIVOTDATA("Revenue",$A$3,"Date",DATE(2016,1,1),"Division","Alpha")
The lookup values (date and division) could be retrieved from a cell on the page or hard-coded into the formula. This is a requirement for several reports I'm working on.
Or, I could add a combined lookup column with Date and Division concatenated and use a vlookup to pull the values like:
=VLOOKUP("42371Alpha",I9:L13,4,FALSE)
Finally, I could use a combination of INDEX and MATCH to identify the correct row number and then pull the data.
All of these solutions require the data to be loaded onto a sheet. One requires a pivot table that has to be refreshed to work properly. The other two require creating arbitrary lookup columns so that you can match a row based on more than one field (date and division in this example), and you have to ensure that that lookup field's formula is properly extended down the length of the data table. In both cases I would have concerns when sharing this workbook with my colleagues in case someone affects the rather fragile setup of the pivot table or the lookup.
So, what I truly want to find is something equivalent to pivot table querying against a dataset.
** This doesn't exist, but I would like to know if something like it does **
=GETQUERYDATA("Revenue","DivisionalRevenue","Date",DATE(2016,1,1),"Division","Alpha")
Does such a thing exist? Can such a thing be done? Can I retrieve arbitrary data points from the dataset created through Power Query / Get & Transform Data?
I think that what you want are cubefunctions:
Some Background
How to easy create cubefunctions from a pivot table
There is a feature in Excel that allows you to query off of a PowerPivot model, but it's not highly advertised for some reason.
Once you have the data in your PowerPivot model, go to your Excel -> Data tab -> Existing Connections -> Tables tab
From there, choose the table that you want to start with. Once that table's data is on your excel sheet, you can actually right click that table -> go to "Table" -> "Edit DAX"
From there you can enter the following DAX function, as an example
EVALUATE
FILTER(SampleData,[Date]=DATE(2016,1,1) && SampleData[Division]="Alpha")
Make sure to choose Command Type=DAX in the drop-down. Here's how it looks on my screen:
To further improve your querying power, you can install the optional "DAX Studio" plugin for Excel, which allows you to write custom DAX queries and then export the results directly back to an Excel sheet.