Is there a way to set a max run time or a timeout factor in Power Query so that after a specified period the query would terminate itself regardless of whether it was successfully executed or not?
This is important for me because I have built various queries at my workplace that usually run fine (as in I have been running it daily without issues for months and years) but occasionally, they would hang (likely because of clashing with another process on the server at that moment, when the server was under heavy load) and keep making read requests on the server indefinitely - One time IT have told me they logged more than 7 million reads from my machine into the database within a few hours. In cases, these have caused the servers to crash which leads to extensive downtime.
So I would like to know if there is any setting, or any thing that I can build into the query itself to ensure it terminates after a certain period of time.
I'm proficient with the M Power Query langue.
Thanks.
[Update 1]
Thanks Alexis for the suggestion below regarding setting a CommandTimeout value when coding the connection. It's a great pointer.
I'll try it out, do some tests and report back if it conclusively fixed the issue or not.
But in the meantime, I've done a bit of digging into Microsoft's Power Query documentation and found that the CommandTimeout argument itself already has a default value of 10 minutes built into it, so theoretically even if we didn't specify that argument, the query should have terminated itself. But that wasn't the case. It ran for hours.
I wonder if it is a bug with Excel version of Power Query? Because I do use Power Query within Power BI as well, and over there I haven't seen it crashing and hanging yet (admittedly I've been using the Excel version more frequently than I did Power BI).
However, if anyone has any other suggestions on potential fixes for this problem, that would be much appreciated. Thanks.
I found a pretty decent answer to this here. Here are the steps posted on that forum:
Have the Power BI Desktop file open and in Report View
Click on the arrow for "Edit Queries" (in the External Data section)
A dropdown will appear - then click "Data Source Settings"
Data source settings pop-up window will appear
Click on "Change Source..."
Another pop-up will appear
Click on Advanced Options (a drop-down of sorts will appear within the pop-up)
"Command timeout in minutes (optional)" will be the first option
Enter a value - I chose to enter 60 minutes but feel free to enter any value
Then apply the query changes and wait till the refresh is complete
This was written for Power BI, but it works in the Excel query editor as well. In summary,
Open Query Editor
Choose File > Options and settings > Data source settings
Select your source and click on Change Source...
Expand Advanced options and enter a Command timeout value
When I tried this with a connection to a SQL server it added a CommandTimeout argument to my Source step. You can just use this code instead and skip all the clicking:
= Sql.Database("server", "DB", [CommandTimeout=#duration(0, 0, 15, 0)])
Doing it via the Data source settings may be preferable if you aren't connecting to a SQL server as the parameters might be different. E.g.
= Web.Page(Web.Contents("URL", [Timeout=#duration(0,0,15,0)]))
or
= OData.Feed("http://some.url/service.svc/", null, [Timeout=#duration(0, 0, 15, 0)])
Related
I am trying to use PXDatabase.Delete to remove specific row(s) in a Custom table or even just to get it to fire off. Here is the code that I am trying to get running:
PXDatabase.Delete<SSINHoldRule>(
new PXDataFieldRestrict<SSINHoldRule.inventoryID>(18));
When I watch SQL profiler for this code to get executed, nothing shows up in the SQL profiler which means that this code is not hitting the database. I have different variations of the above code but to avail. Any help would be appreciated. I am running this code via the TestSDK.
Generally, this would be considered direct DB manipulation, which is frowned upon by Acumatica (Documented within Acumatica certification guidelines) and should be used as a last resort. Is there a reason this is not working with general DAC record manipulation? We want to assess using that before considering a more brute force methods such as PXDatabase.Delete.
I would like to create a spreadsheet that I can refresh and pull in each weeks English premier League fixtures, each week I would like to refresh this and see the weeks future fixtures. I have tried to use the import function from Data/From Web and selected the box with the table of fixtures however no data gets pulled into the spreadsheet.
The website I am using is - "http://data.7m.com.cn/matches_data/92/en/index.shtml"
I am open to understand a better way of doing this import and also if there is a better website to use I am also happy to change. I have chosen this one as it seems to have the most simplified listing of the fixtures.
I have also tried this website - https://www.premierleague.com/fixtures
When the import completes it actually skips all the fixtures and returns all the other information.
Should i be looking to some of the HTML elements within the script of the web page to extract the data?
For example on the following site - https://www.premierleague.com/fixtures I am looking for a file received by the website that updates the fixtures each week (after some direction from Google) I hit the F12 command and look within the "Network" tab however I cant understand how the website, this or the others quoted create the weekly fixtures.
Any suggestions on how to pull this into Excel or another tool would be fantastic.
Welcome to [so]! it sounds like you haven't done as much research as you could have. Your first link, in the top corner has links to "Free Feed" which take you to customizable widgets and from there is a link to a customizable live template.The first page also has a link to "Data" , I'm not sure what that consists of or whether it will help (since I'm not much of a sports fan on my continent, and even less on yours!
As for importing into Excel, I didn't have an issue with the table I could see, but once again I'm not clear on what data you're trying to get and what you want to do with it.
On the ribbon's Data tab click From Web.
Enter the first URL from your question and hit Enter
When the Navigator window loads, click "Table 1" and then click Load.
Below is what Excel then automatically loaded as a table:
If instead of clicking Load, you were to click Edit then you are brought into the Power Query Editor, where you can customizable tons of stuff. The one I was interested in was Use First Row on Headers. After choosing that, and clicking Close & Load, and 30 seconds of formatting later I had:
With Power Query you can choose, remove, split, or combine columns from this or other tables. It's fairly advanced but you should be able to find a good Power Query tutorial online, to see examples of what you can do, to learn about other ways you can customize the import and/or analysis of the data.
Edit:
More Information:
Here are the instructions for all versions:
Office Support : Connect to a web page (Power Query)
Hi i wonder if anyone can help me, I am absolutely stuck on this one. I will start from the beginning so you understand what I am trying to accomplish.
I have a web page which displays total figures in a grid, below the grid I have a Report Viewer which displays these figures as a line graph via an SSRS report on the server. Recent requirements would like the grid to be filterable, so if they want to see totals for a specific customer or product. While I have this working I would also like the SSRS report within the report viewer to also so the same information.
Now I have this semi working, I have made the necessary changes to the report on the server and this is working correctly, next was to hook that all up with the Report Viewer. I also have this working to a degree. Basically what is happening is if the Async is flase then the report does not refresh even ater telling it to. If I turn Async on it works as expected on my computer with IIS7 however when I upload this to our server with IIS6 after the initial load, when a postback happens or when I try to filter the grid I just get a blank screen. Its like the report is not being displayed.
Any help would be appreciated as I have previously used SSRS with report viewer to get a different report based on option buttons which works, but this does not seem to on the server. Locally it is fine but not where it needs to be.
Give it a try . Set enable 32 bit to true
Below is how you do it from IIS7 manager:
1 Right click on the Application Pool and select “Advanced Settings…” or select the same from the Actions pane after selecting the Application pool.
2 Change the “Enable 32-bit Applications” to True (if you want the application pool to spawn in a 32-bit mode)
3 Click OK.
Ok so I figured it out and while it has taken me a while to get the time to post it on here, I thought I should, just in-case someone else comes across this issue.
So Async ="true" the report is displayed in the report viewer for the first time after any postback occurs I am left with a blank screen. If I turn Async = "false" then I get the report the first time but the report never refreshes. After fiddling for a while it got me thinking if it works the first time then that is all I need and just pass different parameters to the report. So I set Async = "false" and had a look as to how to completely reset the report viewer control. Turns out all I needed was ReportViwerControl.Reset(). This reset the control to its default like almost like we are using it for the first time. Then I apply the report details and parameters and hey presto the report works as expected and the report changes every time a filter is set.
I uploaded this to the server and it is working as expected. Quite why it wouldn't before I am unsure however, i do have to stress that the server is old and has old technology, the reason for IIS6. While our test server and my local copy have the latest technology.
Thank you all very much for your help.
hi I'm having troubles with the SharePoint list, I've got the list connected to a Visio, and when I'm on-premise, it update fast, I mean I change the list, click refresh in the Visio app and it's updated, but on the Visio web access from SharePoint, It takes too long, I can be pressing refresh, but its like it has a timer, every 2 or 3 minutes it updates, every change I do so far. The problem is that I need to be instant, sometimes its is instant because I luckily change the list when its about to refresh.
Is there any configuration in the server to me updates that fast?
English is not my native language, so I'm sorry
If you want your changes to be shown instantly, you can set Minimum Cache Age to zero in SharePoint:
http://technet.microsoft.com/en-us/library/ee524061.aspx
Note that default value (a few minutes) provides a better rendering performance in multiuser scenario.
Also note that this seetting does not seem to be available in SharePoint Online / Office 365 (please correct me if I am wrong).
We are having issues with the MOSS 2007 Web Service. Specifically, the Query/QueryEx method of the API.
We are using the SQL Syntax to query sharepoint, and for most cases, our search execution time is very fast. However, when we add filters and sorts, things start slowing down. In some cases we even get an error.
For example, this MSSQLFT query will give a System.ServiceProcess.TimeoutException exception about 50% of the time. The timeout will occur around the 10 seconds mark.
SELECT WorkId, Rank, ... , PerforceFolder, PerforceDateSubmitted FROM SCOPE()
WHERE "scope"='Tech_Depot' AND (FileExtension = 'cpp' )
ORDER BY PerforceDateSubmitted, Rank DESC
Is there any settings / query parameter / etc... we can use to allow a bigger SEARCH execution time on the sharepoint back-end?
This could have many causes:
The property used in the search is not indexed, therefore, the search takes longer
The result set is too big, try specifying a row limit
These tools can help you with development:
- SharePoint Search Service Tool`
- SharePoint Search Bench
Hope that helps.
http://msdn.microsoft.com/en-us/library/microsoft.office.server.search.query.query.timeout.aspx
believe it or not this 10 second limit is hard coded. The only way round it is to write your own version of the web service. I've done this and the results are here:
http://eliasbland.wordpress.com/2009/12/10/search-webservice-for-sharepoint-search-server-with-configurable-timeout/
Hope this helps,
Rupert