I am trying to using Microsoft Flow to export a Sharepoint List to Azure Data Lake.
I want it so that anytime a particular online list is changed, its entire contents are loaded into a file in Data Lake. If the file already exists, I want to overwrite it. Can someone please explain how I can go about doing this, I have tried multiple ways, but they are not getting the job done.
Thanks
I was able to get the items in the SharePoint list to near perfection. I will post the Flow here in case anyone in the future needs it.
So what I did is that every 5 minutes I "create" a file in Azure Data Lake which overwrites the file if it exists. The content of the files cannot be blank, so I added a newline to the content. Then I use Get Items to retrieve all the items in the SharePoint List. From there, using an Apply to each loop, I append the content of the current row of the Sharepoint list to the Data Lake file (separated by | and ending with a new line after all the content is added). This works to near perfection, with the only caveat being the newline at the beginning of the file, which I eliminate using PowerQuery.
This is exactly what I needed. If anybody sees a way to make this better, please post so that we can get this to perfection.
Related
Good Day!
I am struggling with the best way to approach this via Power Automate.
Desired Flow: This will be a scheduled flow that would trigger every 10 minutes. Cycle through a SharePoint List that contains a list of subsites within my enterprise along with the path to a specific folder in a specific library. Using the 'Apply to Each' function, identify any NEW files since the last sweep. Copy those files to a different directory in SharePoint (details of which are also included in that same SharePoint list).
I have got all the pieces to this EXCEPT: How to identify the new files since the last sweep.
Can't use the "when a new file is created" trigger, as this listing has close to 300 subsites on it....which would mean 300 different flows.
Appreciate any guidance!
UPDATE:
Getting an error when using the Get Files. Here is the snippet of the flow:
Snippet of the Flow Where error is occuring
Snippet of the SharePoint List
I would use a Get Files (properties only) action with a Filter Query which uses an Created greater than 10 minutes ago.
Below is an example.
Created ge '#{addMinutes(utcNow(), -10)}'
This is for the second part of your question.
After some months I could say I am getting the hang of Microsoft Flow, however I could use some help with the following issue:
In a flow for reporting purposes, a temporary file (.xlsx) is created in a sharepoint folder by means of a template. This temporary file is then filled with rows and info from other sources. So far so good.
I use the body of this newly created and furnished file as an attachment for an e-mail to the chief. However, the attachment came out identical to the (empty) template file, without the rows and furnishing.
Adding a delay of two minutes before attaching and sending the mail solved it for relatively small reports, but this is not ideal as I want it to work regardless of file size. Furthermore I do not understand why it would send an empty (old) version of the temp. file in the first place, as all the furnishing operations should have executed before copying and attaching (the flow is entirely in series).
Sorry for the long story. Does anyone have a more elegant solution than using a Delay-node?
So my problem is quite stupid but I cannot find a way to resolve it. I have one 15 GB file on external SFTP server that I need to copy to my data lake. The thing is that column delimiter is a comma and I have some nested lists as well. So when I am trying to use ADF copy activity, the result looks like that:
And most of my data is gone(as nested structures get cut on the first occurence of comma). So maybe I could ignore delimiter. I have tried to set pipe as a delimiter just to get this whole dataset as one column but this doesnt work either.
Powershell? I have tried different scripts that used to work with smaller files and I am getting an error every time.
I have even tried to upload it manually via Azure Storage Explorer but it fails as well after some time. I am not really sure how to make it work at this point.
Thank you for any advice!
I am not sure if i am going to be able to describe this right but ill give it a go.
We are working on implementing Azure search. At the core level we have searchable PDF documents that we want the text of them added to the index so all of them are searchable.
The initial thought was to just submit that document to the index via the add document rest api. The thinking was that this would be the most simple and quickest path
to getting the text of that document into the index. We also considered using and indexer and just having all the Searchable PDF docs in a blob store and have the indexer
crawl those every 10-15 mins.
We also looked into (based on a recommendation) submitting a standalone JSON file with the text from the PDF in it. Submitting that to the index either via the same add document API or
placing that file in a blob store. Within the JSON document we would need to have document identifiers that provide the index with the location of the PDF so that when that text is found
via search, we can make that clickable and as a result open the PDF.
It seems to me that pushing in the json file with the document add api. Indexing that and when it is part of a search we can use the doc id to link back to it and open it.
For those of you that have used Azure search. How did you implement?
If you're totally sure that only pdf will live on this particular index, then the first approach is faster to implement, since the native indexer can be used for extract the content of the pdf document as well to push it to the index.
Both approaches will work, but for the second one, you would need to extract the pdf yourself using an external tool.
Backgorund:
I am a newbie in the NetSuite world. We are trying to integrate NetSuite with our ERP and I am doing some preliminary research to find out what would be the best option moving ahead. The primary objective of the first task is to download huge volume of data from NetSuite to our end and find alternatives approaches.
I did some research on SuiteScript/SuiteTalk/Analytics and some facts I have come to find and my questions are below:
Custom search can be created and save SuiteScript/SuiteTalk.
This saved search can be invoked via both SuiteScript as well as SuiteTalks
Well have a confusion, is the Saved Search the View, which SuiteAnalytics can access? (NOT MY MAIN QUESTION THOUGH!!).
Using SuiteScript, return of Saved Search execution can be saved on as a flat file, and that file can be moved to File Cabinet. Exposing a REST API using RESTlet, this file can be downloded. [But have not implemented this yet!!]
[MAIN QUESTION] IS IT POSSIBLE TO DO THE SAME, CREATE A FLAT FILE AT NETSUITE END USING SUITTALK? AND ALSO HOW TO DO SAVE/MOVE THE FILE TO FILE CABINET AFTER THAT?
I have not researched more on the topic File Cabinet and how a created file or files here are indexed?
Or Is it better to load whole result set from the SOAP call?
Your comments are highly appreciated!
Thank you!
You can certainly execute a saved search via SuiteTalk. You can also loop through all the results of the saved search and do whatever you'd like with those results, such as create a text file.
The SuiteTalk API also allows for accessing the File Cabinet to create or retrieve files, with limitations on file size.
Suitetalk can be used to create File and move file from a folder to another by changing the folder internalId of the fileObject.
Since you are using the Suitetalk to create/load Saved Search; you are required to create and save the CSV at your end using the search result and then move the file to file cabinet.
Since your objective is to get huge data from NetSuite I would recommend below option:
Use Scheduled script/Map Reduce to build a file and place it in required folder of file cabinet
Using Suitetalk you can extract that file. (Note: You don't need REST do this job. You can get the fileContents and store the result at your end. You cannot directly store the file. You will have to store the filecontents)
Thanks #netsuite-guru and #suite-resources!
So doing some further research, and considering your recommendations, the server side(at NETSUITE) scripting can be done only using SuiteScript to achieve the goal of automating - READ from NetSuite and WRITE to File to FILECABINET!
Also found another good read thread as an option to MapReduce link
But would go with "Scheduled script/Map Reduce" at this time.