I am trying to retrieve data with all columns from a large SharePoint list (57000+ items and increasing everyday) using webservice.
The problem that I face is that; in the below code, I have heard that the ndRowLimit's recommended value is 5000. When I am setting 57000, I am getting operation timed out.
XmlNode ndListItems = listService.GetListItems(ListName, null, ndQuery, ndViewFields, ndRowLimit, ndQueryOptions, null);
Does anyone know any method using which I can get all the list items and all the columns?
In SharePoint, you can increase the connection time-out setting, follow these steps.
Note By default, the IIS connection time-out setting is 120 seconds.
You can update it via IIS Manager.
Click the Web Site and then the Connection Timeout is under Advanced Settings.
You may run into a further timeout issue which is controlled in the web.config - (always backing that up). But the property that has a default timeout of 110 seconds is under <system.web>
This sets the timeout to 10 minutes.
<httpRuntime executionTimeout="6000" maxRequestLength="51200" />
Finally, after you beat the timeout issue, you'll probably need to (temporarily) adjust the throttling of results at that list or through Central Administration. Here's a great article on the in's and out's of that: http://blogs.technet.com/b/speschka/archive/2009/10/27/working-with-large-lists-in-sharepoint-2010-list-throttling.aspx
Related
I need to run an aggregate query to calculate the count of records e.g. SELECT r.product_id, r.rating, COUNT(1) FROM product_ratings r GROUP BY r.product_id, r.rating. The query works perfectly fine on the Azure Data Explorer, albeit a little slow. An optimised version of the query takes about 30 seconds when executed on the Data Explorer. However, when I run the same query in my Java app, it appears to be timing out in 5 seconds with the following exception:
com.azure.cosmos.implementation.GoneException: {"innerErrorMessage":"The requested resource is no longer available at the server."}
I believe this is due to a default request timeout of 5 seconds defined in ConnectionPolicy (both Direct and Gateway modes). I can't find a way to override this default. Is there a way to increase the request timeout? Is there another possible reason for this error?
Tried this both on the Java SDK v4 and Spring Data Connector v3 with the same end result i.e. GoneException.
You could consider the following recommendations,
The following should help to address the issue:
Try to increase http connection pool size (default is 1000, you can increase to 2000)
If you are using GateWay mode, try the DirectMode, more traffic will go over tcp and less traffic over http
You can refer the Github code on setting the timeout.
I have been given a task of combatting an occasional slowness starting up an Azure web app. The web app calls makes seven separate Azure API controller calls that each connect to an Azure SQL Server in order to run stored procedures. The application insights show that these calls take less than 250ms 90 percent of the time but will take 7-15 seconds at other times. Adding logging shows the OpenConnection statement accounts for all of the delay.
date1 = DateTime.Now;
_dbContext.Database.OpenConnection();
date2 = DateTime.Now;
if (_logger != null)
{
interval = date2 - date1;
_logger.LogDebug(string.Format("GetUserDetails Opened Connection {0:N0}", interval.TotalMilliseconds));
}
I added a min pool size and max pool size of 200 to the connection string. It did not help and the application insights does not show more than 100 connections at a time. Profiling the Azure SQL Server shows the Audit logout with a similar delay when connecting.
Where else should I look to find a cause for this occasional delay in creating connections?
Thanks in advance,
Hank
With performance, it could be a bunch of things. I can provide a couple of things that you could check.
Is it always the Xth iteration/invocation etc. that is causing the delay ? In that case, see if you have any locks etc. due to the previous queries, etc.
If you are connecting to Managed instance, check your connection policy. Setting it to proxy can also lead to throttling when there is high network load. (or if you are connecting and querying crazy fast let's say in a loop, etc.)
This is something simple, but make sure you are using a connecting string rather than specifying the connection details in code every time. Connection pooling features only kick in when using a connection string. (Could also explain why you have 100 connections)
This maybe related to platforms other than ColdFusion.
IIS 6 Log reports "time-taken" much longer (30 minutes) than 120 seconds set in Connection Timeout for several requests to ColdFusion page.
I assume that ColdFusion was unresponsive at the moment. I would like IIS to stop the request rather than wait this long.
Is there an IIS setting that would force this?
Not really because iis is no longer handling the request once it has been passed to cf. You could try playing with application pool timeout and see if you can get that to throw an error.
This scenario can also be considered as the slow HTTP DoS attack when caused by the client. IIS doesn't provide much protection against it (at least for slow POST body) because Microsoft considers it a protocol bug, not an IIS weakness. Although I think in this case it is your server doing it to itself.
Things to check:
You didn't mention whether it is the request that is slow or the
server's response. You could try tweaking your
MinFileBytesPerSec parameter if it's the response that is slow. By
default it will drop the connection if the client is downloading at
less than 240 bytes per second.
Remember, that 120 second IIS timeout is an idle timeout. As long as the client sends or receives a few bytes inside 120 seconds, that timer will keep getting reset.
You didn't mention if this long wait is happening on all pages or always in a few specific ones. It is possible that your CF script is making another external
connection, e.g. CFQUERY, which is not subject to CF timeouts, but to the timeouts
of the server it is connecting to. Using the timeout attribute inside CFQUERY may prevent this.
You also didn't mention what your Coldfusion settings are. Maybe the IIS timeout setting is being ignored by the Coldfusion
JRUN Connector ISAPI filter, so you should check the settings in
Coldfusion Administrator. Especially check if Timeout Requests
after has been changed. If its still at the default of 60
seconds, check your code to see if it has been overridden there, e.g.
<cfsetting requestTimeOut = "3600">
Finally, there is the matter of the peculiar behavior of CF's requestTimeout that you might have to workaround by replacing some cfscript tags with CFML.
We have an Azure worker role that exposes a RESTful WCF service (using System.ServiceModel.Web) through a ServiceHost. The performance is irreproachable on massive traffic, but it seems like the response time is significantly higher (more than five seconds) on the first request when the role has been idle for some time. Does anyone know what might cause this?
The default AppPool timeout is 20 minutes. Might you be running into this? If so, you can add something like this to a startup script to change the timeout:
%windir%\system32\inetsrv\appcmd set config -section:applicationPools -applicationPoolDefaults.processModel.idleTimeout:00:00:00
Here's another answer I posted, to a different question, discussing this further.
I have a intranet SiteCore website set up on IIS 7 which randomly throws the following error message
HTTP Error 503.2 - Service Unavailable
The serverRuntime#appConcurrentRequestLimit setting is being exceeded.
To fix this issue, I have made following changes
Increased the Queue Length of application pool myrjetAppPool from 1000 to 65535.
Modified Machine.Config to increase requestQueueLimit property of ProcessModel element to 100000
Increased appConcurrentRequestLimit to 10000 by running
C:\Windows\System32\inetsrv\appcmd.exe set config /section:serverRuntime /appConcurrentRequestLimit:100000
But I'm still getting the same error. ANy help is greatly appreaciated.
You might check to see where all your threads are going. We had occurrences where threads for Media Library assets were hanging and blocking up the queue.
In IIS Manager, select the server node from the tree, then the "Worker Processes" feature icon, then right-click the application pool of interest and select "View current requests". You might find something is getting stuck. I sometimes hit F5 on this screen a few dozen times in very quick succession to see the rate the requests are going through (of course Performance Monitor is better for viewing metrics but it won't tell you what URLs are being processed).
Investigate references in the linked url to 'MaxConcurrentReqeustsPerCPU' which you may need to set by creating a new registry key, depending on your OS and framework.
https://learn.microsoft.com/en-us/archive/blogs/tmarq/asp-net-thread-usage-on-iis-7-5-iis-7-0-and-iis-6-0
As already commented - check the actual concurrent request count using performance counters to determine which limit you're hitting i.e. it could be a limit of 5000 or maybe 12 (per cpu).
Edit: I realise this may look like I'm talking about a different setting entirely, but I believe there is overlap here.
We got this problem after an installation of an IIS plugin. After long investigating we saw that the config-file C:\Windows\System32\inetsrv\config\applicationHost.config had an extra location tag for the site with the problem. After removing the extra entry and an iisreset, the site/server worked normally againg. So something must went wrong during the installation....