SharePoint Rest call is not returning all fields - python-3.x

Goal: Have python program pull data from SharePoint so we can store on database.
Issue: I am able to connect to share point and return data, but I am not getting all of the fields I can see when hitting the UI page. The UI page I am hitting is in the list on REST call but is a Custom View
Update: Using the renderashtml I was at least able to see some of the data points I am looking for. I would hope there is a better solution than this
Code:
import sharepy
connection = sharepy.connect("https://{site}.sharepoint.com")
r = connection.get("https://{site}.sharepoint.com/{page}/_api/web/Lists/getbytitle('{list_name}')/items")
print(r.content)
print(r.json())
#I have also tried
https://{site}.sharepoint.com/{page}/_api/web/lists('{list_id}')/views('{view_id}')
#I was able to return data as html
https://{site}.sharepoint.com/{page}/_api/web/lists('{list_id}')/views('{view_id}')/renderashtml
Research: I have taken a look at the rest documentation for sharepoint and I am under the impression you cannot return data from a view. The solution I saw was to first hit the view and then generate a list of columns and use that to build a query to search the list. I have tied that and those fields are not available when I pull the list but are in the view.
https://social.msdn.microsoft.com/forums/sharepoint/en-US/a5815727-925b-4ac5-8a46-b0979a910ebb/query-listitems-by-view-through-rest-api
https://msdn.microsoft.com/en-us/library/office/dn531433.aspx#bk_View

Are you trying to get the data from known fields, or discover the names of the fields?
Can you get the desired data by listing the fields in a select?
_api/web/lists/getbytitle('Documents')/items?$select=Title,Created,DateOfBirth
or to get all of the fields:
_api/web/lists/getbytitle('Documents')/items?$select=*

Related

KQL (Keyword not Kusto) Nesting, Document Selecting for an Extranet

Longtime member, been a while since posting. Working on building out an Extranet and am running into a stupidly frustrating issue. First time using SharePoint Online as document repository for external (anonymous) users. In doing so, using Azure permissoning, have the documents split up in repositories on SharePoint based on access level. On top of that I am attempting to display them in Highlighted Content Web part, I am not able to sort them out by location AND type. I have a custom column in each repository that defines what type they are, but when I try to add the AND portion to the KQL it doesn't work. Additionally the internet seems to be massively void of actual documentation of KQL.
(
path:https://domain.sharepoint.com/sites/example/Level%201%20Resources/
OR
path:https://domain.sharepoint.com/sites/example/Level%202%20Resources/
OR
path:https://domain.sharepoint.com/sites/example/Level%203%20Resources/
OR
path:https://domain.sharepoint.com/sites/example/Level%204%20Resources/
OR
path:https://domain.sharepoint.com/sites/example/Level%205%20Resources/
OR
path:https://domain.sharepoint.com/sites/example/Level%206%20Resources/
AND
DocType:"Articles"
)
The above will simply pull all documents from those locations and ignore the AND statement. I have tried renaming it to call on the custom column identifier pulled from the source, and that doesn't work either.
The only real documentation I can find on this is: Here
Which doesn't appear to address filtering based on custom column tags.
EDIT: Reformatted to pull all docs from multiple locations using below, but the nesting portion still isn't working
path:(
"https://domain.sharepoint.com/sites/example/Level%201%20Resources/"
OR
"https://domain.sharepoint.com/sites/example/Level%202%20Resources/"
OR
"https://domain.sharepoint.com/sites/example/Level%203%20Resources/"
OR
"https://domain.sharepoint.com/sites/example/Level%204%20Resources/"
OR
"https://domain.sharepoint.com/sites/example/Level%205%20Resources/"
OR
"https://domain.sharepoint.com/sites/example/Level%206%20Resources/"
)
So the additional issue I was running into was the creation of a column to separate out based on the category of file type (not literal file type). Apparently SPO doesn't like it when you create a list and then reference that list to then filter by via KQL. So I found this morning this.
Apparently the best way to do this is create a custom "Choice" column, allow some time for it to flow and update, and then you can reference it via KQL.

Passing More than 1000 parameters in RESTful api

Our Dashboard have dropdown which consist of more than 8k products and we have such 4-5 dropdowns.
I want to filter data based on this dropdowns.
But if I selecting all products then restful api url is breaking.
Can i use azure bus service or similar message broker service to pass this complex and multiple parameters via service and which then consumed by all apis??
While I do understand what your need is I will suggest to re-think your approach as that will lead to a better user experience.
I will use a first drop down, completely empty, that will fill in options while the user types in whatever product list you have, that way, there is only a group of items that are filled in that the user is interested in.
I will also suggest not to load anything on that drop down until the user has entered, let's say, 5 characters.
After that let the user select an item from the dropdown, and filter the second drop down and continue to use the same technique with all drop downs you have.
Let me know if you have any questions around this approach and I will be more than happy to provide an example if you don't know how to do it.
Edit:
Including samples for load dropdown lists dynamically and also another post with an example on how to return some JSON from an Azure Function that can return the data in a dynamic way like you need.
From what I can see I think it'll be helpful for you to go through several different examples that can bring you progressive knowledge to what you need to achieve, first of all it'll be nice if you know how to load items dynamically on a dropdown list:
That is a nice example, you can have an entry where the user can type
whatever products you have and then they type, lets say, more than 5
characters go and search for that data.
https://www.codebyamir.com/blog/populate-a-select-dropdown-list-with-json
This other example shows you how to return data from an azure function
with the data that you need to show the users. This is not exactly
what you need as you need to receive a parameter with what the user
typed and search in your database for the items that match that search
and return that data in a JSON format so you can use it on your web
page to show the dropdown list. I will suggest to initially hard code
a few items and return those to check that the functionality is there
and once you have that up and running move to get the actual data from
the database.
How to return a JSON object from an Azure Function with Node.js

kibana - programatically return saved search objects and associated data via REST API

I am currently working on an excel export tool for kibana using node.js. Right now I am trying to figure out if it is possible to export the data associated with a saved search within my selected kibana index.
Here is an example of what I am trying to do:
User provides authorization and selects a project within kibana that they have access to.
Once a user has selected a project, any saved searches associated with that project are populated into the UI.
The user selects a saved search, report name, and date range, and submits the form. The application then makes a request to the kibana index and returns the data associated with the selected search and within the given time range.
I have finished the authorization and UI, but I am currently stuck trying to figure out how to return the saved search objects within a specific project. I am also unsure of how to construct the request to the kibana index that would return the data associated with the selected saved search within the given time frame.
Does anyone have any experience with something similar to this? I am also very new to Elasticsearch, is this sort of functionality possible?
Answered by a wonderful Elastic team member here:
https://discuss.elastic.co/t/exporting-saved-search-data/90843

Re-using a list in Node.Js

I'm developing a web scraper in Node JS and I'm trying to figure out the best approach to combine data from one list, with data on another list.
For example:
Step 1: Scrape data from website A
In this step, I scrape some data using cheerio/request and store this in a list, which is then displayed on the screen in a jQuery data table. The data table has a checkbox next to each scraped row of data and the value I have assigned to each checkbox is a "URL".
Step 2: Scrape again based on URLs chosen in checkboxes
In step 2, the app will scrape another website based on the URLs selected in step 1 and will retrieve some data values from these URLs that are scraped.
My dilemma
I wish to use some data values that were scraped in step 1 along with some data values scraped in step 2. However currently in my app, the data from Step 1 has been lost because it's not being saved anywhere.
Since this is a sort of dynamic search whereby a user will search for data, scrape it and then not neccessarily want to see it again, I think saving data into a database would be overkill? So I'm wondering if I should save the list data from step 1 into a session variable and then link them up together again using the URL (in the checkbox) as the key?
Thanks for your help!
Anthony
if you dont want to do saving for these data then pass it as another inputs of your form , try
< input type=hidden value=json.stringify(item)/>
I dont think using database for storage of the scrapped conetent would be an overkill.
The ideal points to note in this process would be.
Use a document store like mongoDB to dump your json data directly. I proposed mongoDB because you get more resources to refer.
Initially open up a db connection in node and your scraping deamon can reuse it for each time when it parses the http result using cheerio and dumps it to db.
Here Once you get the output from http request to your Target URL, the cheerio parse steps should take more time than dumping the data to a db.
I have followed a similar approach for scrapping movie data from imdb and you can find the corresponding code in this repo, if you are interested. I have also used cheerio and its a good selection in my opinion.

Why does owssvr.dll return an empty result for my list?

I am trying to use owssvr.dll and its URL API in SharePoint 2007 to get some list metadata. For most of the lists in my site of interest, I can use a url formatted like so:
http://hts-app1/compounds/_vti_bin/owssvr.dll?XMLDATA=1&List={F987723C-28A4-47D3-83D7-19094B0267DF}
I get back an xml-based list of field metadata for the default view and a list of records in the list. But when I try to use another guid for a separate list, I get an empty response from the web server. It's a 200 response code, but the other headers look like:
Connection:close
Date:Thu, 24 Mar 2011 17:38:46 GMT
MicrosoftSharePointTeamServices:12.0.0.4518
Server:Microsoft-IIS/6.0
X-Powered-By:ASP.NET
I am pretty sure the list guid is valid, since I can match it with the guid for the list I can see in the listedit.aspx page. If I also format my url like so:
http://hts-app1/compounds/_vti_bin/owssvr.dll?XMLDATA=1&List={F987723C-28A4-47D3-83D7-19094B0267DF}&Query=*
with the Query=* parameter, I can get valid xml, but it's just showing everything, and not filtered for my view, and I'd really like to have the view.
I looked in the SharePoint log file, and I couldn't find anything that looked related to this request.
So, how can I debug this? I'd really like to use a simple http GET request to get back this metadata, and owssvr.dll seemed the ideal mechanism.
You can specify a specific view as well. Like this
http://{0}/_vti_bin/owssvr.dll?Cmd=Display&List={1}&view={2}&XMLDATA=TRUE
for example:
https://foo.com/extranet/_vti_bin/owssvr.dll?Cmd=Display&List=%7B7F7AFB9D%2D6D5D%2D4626%2DBD9D%2D085957DB79AB%7D&view=%7B111EC07E%2DF648%2D443B%2D8DE6%2DB53786BE6762%7D&XMLDATA=TRUE
The absolute easiest way to get the list and view guids is to goto edit the view and snatch the list & view info directly from the address bar.
1. That supplying GUID in the querystring of your URL works for one list and does not for other is probably because the lists are on different sites. Verify that the sitename is correct and that GUID is a List ID indeed.
Be wary of the fact that, however, above applies to both SharePoint 2007 and 2010, for SharePoint 2010 additional measures might be required depending on context from which you execute commands on owssvr.dll.
2.
query=* is not a filter parameter; it shows schema as well as data for all available columns in the list, including SharePoint internal columns, which normally you never get to see in ways other than programmatic access.
3.
To get only those columns that are visible in the current default view call display command on owssvr.dll while xmldata parameter is set true; like this:
http://hts-app1/compounds/_vti_bin/owssvr.dll?Cmd=Display&List={F987723C-28A4-47D3-83D7-19094B0267DF}&XMLDATA=TRUE
Again this is for SharePoint 2007 only and depends on execution context.
4.
with the Query=* parameter, I can get
valid xml, but it's just showing
everything, and not filtered for my
view, and I'd really like to have the
view.
Oh but this is not called filtering; filtering is only based on columns values. For this use additional FilterField1 and FilterValue1 parameters.
http://hts-app1/compounds/_vti_bin/owssvr.dll"
+ "?Cmd=Display&List={F987723C-28A4-47D3-83D7-19094B0267DF}"
+ "&XMLDATA=TRUE&FilterField1=YOUR_FILTER_COLUMN_NAME&FilterValue1"
+ YOUR_FILTER_COLUMN_VALUE
Takeoff "(double quotes) and +(plus) and (returns) before testing in browser.

Resources