I need to get all the items of a SharePoint list and send as JSON
But the default is 100
If I set a larger number like 10000 it gives this error on the flow:
The attempted operation is prohibited because it exceeds the list view threshold.
So I am trying to get it in batches of 350 items.
But how can I use the Filter Query to start at a position x and return y items?
I tried
$skip eq 350
But this error occurs:
Column 'skip' does not exist
I think your question has 2 layers to it.
To increase the number of items returned by the Get Items Actions from Sharepoint you need click on the three dots on the left side of the action, and then go to Settings and you will see the image below, then set the size to 5000, that means your Get Items action will now return a maximum of 5000 items.
$top and $skip - I am not too confident what you are trying to do with this but perhaps share a bit more context, it sounds like you want to build a dynamic pagination ? Please explain this part unless the first point already achieve what you were aiming for
Related
I'm not sure I have a syntax error here in my CAML Query or I am misunderstanding how the API works but I have a very large document library in which I am trying to identify items that have had no compliance / retention label set.
My query is trying to return any item in the library that does not have the compliance retention label set to "Test". This is the query but regardless of rowlimit, the console app returns the threshold error... "The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator."
<View Scope='RecursiveAll'><RowLimit>20</RowLimit><Query><Where><Neq><FieldRef Name='_ComplianceTag'/><Value Type='String'>Test</Value></Neq></Where></Query></View>
I have been trying the most basic query and I can confirm this works...
<View Scope='RecursiveAll'><RowLimit>20</RowLimit></View>
But as soon as I add element to the CAML, I get the error. It seems once the query is added it ignores my row limit. The plan is to bring the items back in small batches and apply the label. I've already built a console app which uses the SetComplianceTag to library method. This applies Tags to everything in the library but unfortunately my library is so large we found it missed a few.
Any insights would be much appreciated.
Having read further I now understand the query is executed on the entire list so limits apply regarless of the rowLimit in your CAML. I have now copied and adapted what Piyush K Singh has done in the tutorial linked below. There is a ListItemCollectionPosition object which can handle breaking your API request into batches the size specified in your row limit. I then loop through each batch looking for the items that match my specified conditions...
https://piyushksingh.com/2016/12/04/query-listitems-in-batches-sharepoint-online/
Thanks Piyush!
Currently we are getting all lists in all sub sites using below REST query.
'/_api/Web/Webs?$expand=Lists&select=Title,id,ParentWebUrl'
now we would like to get the all content types that are used in those lists as well. Tried expanding ContentTypes, but could not get it to work.
Can anyone help to get content types belong to those lists? Even if filtering by content type in the REST query it self is fine.
or is there any way to do this in search results of lists?
After searching everywhere and getting no answers here, I've come to the conclusion there is no possibility of doing this way. So, I've used SharePoint Search services to get my results.
querytext = 'path:"your site collection path" AND (ContentClass:"STS_List_GenericList" OR ContentClass:"STS_List_DocumentLibrary")';
Later I decided to add listIDs to the query as I was getting ListIDs from previous Item search.
querytext = '(ListID:"Your list1 ID" OR ListID:"Your list2 ID" OR ListID:"Your listn ID")" AND (ContentClass:"STS_List_GenericList" OR ContentClass:"STS_List_DocumentLibrary")';
The issue was now if the list IDs exceed more than 75, the querytext max character exceeds and no results was received.
After searching a way to increase the maximum limit and finding out there is no way to do it, I divided the list IDs into a set 75 IDs and send separate requests for each set, then combined all together.
You can read it how it was done here Batch Search requests
This is the REST call for a SharePoint Online tenant (you did not specify version). If you iterate your through your list and call this for each list, you can summarize what content types are on your site.
https://YOURTENNANT.sharepoint.com/sites/YOURSITE/_api/lists/getbytitle('YOURLISTNAME')/contenttypes
I am using CtreeCtrl and just struck with a new case which i haven't faced before..
The case is ..
I add more than 2000 items in a particular node .. but i can see only 200+ items under the node..(each time i get different counts displayed) ( ??? )
TreeControl->InsertItem(server,icon,icon,TempParent,TVI_SORT);
So what is wrong here..? Should i need to set any property here..? or Is there any max limit in adding items in the node...??
I add more than 2000 items in a particular node .. but i can see only 200+ items under the node..(each time i get different counts displayed) ( ??? )
There is no property you can set. I just created simple project and added 6 branches, each containing 2500 nodes and have no problem displaying all.
It must be something in your code, that prevents a proper display.
Two choices:
Create simple project and add large amount of items or post your project for download to see what is going on.
I cannot see any other way to help you.
I created two lists in my SharePoint 2010 site. one has 10 items other has 10000 items.
On the small list I can say
http://mysharepoint.com/sites/testsite/_vti_bin/listdata.svc/SmallList?$top=1&$skip=1
no problem it works perfect.
when I say
http://mysharepoint.com/sites/testsite/_vti_bin/listdata.svc/LargeList?$top=1&$skip=1
it throws an error "An error occurred while processing this request"
The internet is full of confustion because most blogs/articles etc talk about "SP2013" where I guess there is a __next in the body of XML which you get from SP.
However I am using SP2010 and this is a PURE client side solution with NO server side object model at all.
Can anyone tell me how to paginate through large lists "specifically" on SP2010.
Please post some working URLs with rest commands.... which you have tested against a large list (please let me know if you need me to provide code which will create a large list of 10K items for you).
You could do it using JavaScript and the WebServices. With the JavaScript API library I created, called SharepointPlus, you'll have to do that (with version 3.0.7):
$SP().list("Name of your list").get({fields:"ID,Title",rowlimit:5000,paging:true},function(data) {
console.log(data.length)
})
You'll have all the items into the 'data' array. You can go thru the array and stop when you reach X elements in a JavaScript loop for example. Like that you should not have an error message due to the large size of your list.
$skip query option is not supported, but $top is.
Workaround, use a mix of $orderby on ID, $filter on ID and $top x items, and loop.
Your query becomes
http://mysharepoint.com/sites/testsite/_vti_bin/listdata.svc/LargeList()?$orderby=Id&$filter=Id gt {lastId}&$top=100
{lastId}=0 on the first run.
Loop the items found, keep track of the "Last Id" and pass it to the next query.
http://mysharepoint.com/sites/testsite/_vti_bin/listdata.svc/LargeList()?$orderby=Id&$filter=Id gt 0&$top=100
http://mysharepoint.com/sites/testsite/_vti_bin/listdata.svc/LargeList()?$orderby=Id&$filter=Id gt 123&$top=100
http://mysharepoint.com/sites/testsite/_vti_bin/listdata.svc/LargeList()?$orderby=Id&$filter=Id gt 345&$top=100
repeat.
Note 1: ID is a column that is always indexed, other
Note 2: This pattern also works with the REST API (LINQ query)
I have around 53,00,000 documents in MarkLogic server and I am building a simple search application. User enters a search term and MarkLogic server searches that term in all the nodes in all the documents and returns the matching documents as the result. I have implemented a custom paging to show results per page. I am showing 10 results per page.
I am using search api for this as:-
import module namespace search="http://marklogic.com/appservices/search" at "/Marklogic/appservices/search/search.xqy";
declare variable $options:=
<options xmlns="http://marklogic.com/appservices/search">
<transform-results apply="raw"/>
</options>;
search:search($p, $options, $noRecFrom, 10)/search:result
where $p is the input from the user $noRecFrom is the number which indicates from where we have to show records. For example for page 1 $noRecFrom will be 1, for page 2 $noRecFrom will be 11, for page 3 $noRecFrom will be 21 and so on. For paging there are hyperlinks to go to First, Next, Prev and Last pages.
To calculate the total number of records returned I am using:-
for $x in search:search($p, $options)
return $x//#total;
While First, Next and Prev hyperlink works perfectly but if someone clicks Last the application stops responding and the query does not show any output. Is it due to the large number of documents in the database or I am implementing it wrongly.
Is there any efficient way for pagination in MarkLogic (for search:search) so that the user can go the Last page without delay in query result for such a large database ?
The way you've implemented it, you're running the search repeatedly in your for loop. And that would indeed be slow.
Instead, you should be calculating a $start parameter based on the #total and number of documents per page, and passing that in as an argument (I think it's the third one) to search:search.
I would also recommend making sure you can run in unfiltered mode. There is good information about optimizing for fast pagination (indexes, etc) on the developer site; the idea is to resolve queries out of indexes to give very good, accurate unfiltered performance.
If you do it that
There is a tutorial on paginated search at http://developer.marklogic.com/learn/2006-09-paginated-search
Once you have resolved the issues mentioned by cwhit above, if you still want to get to the last page of data in a faster manner, you could make your code smart enough to reverse the sort order and pull the correct offset of records.
Here's another tip:
To get better insight to what MarkLogic is doing with search:search, call
search:get-default-options()
to see the starting point for common search applications.