Good morning all,
I'm trying to build a logic app, that will upload the files from Azure Blob Storage to Sharepoint. It was quite easy to do when all the files were supposed to be uploaded to one folder. I was asked to seperate them by the name. So, if a file contains 'dog' in the name, it should go to folder 1, but if a file contains 'cat' it should go to a different folder on Sharepoint, from the same blob storage.
I've tried to add a condition to the logic app, that if it's 'true' that 'name' contains 'dog', upload it to folder 1, if false, upload it to folder 2 (there is always a file containing either 'dog' or 'cat'), but it still uploaded all of them to the folder with 'false' result. Basically, when I ran the logic app, all the results were false, so the problem is with the condition itself, but as I'm new to this, I wasn't able to figure out, what exactly is failing. Below is the screenshot of the logic app to upload all the files to one folder, I'm not quite sure where to put the condition (I've tried to place it everywhere, same result) and how to configure it properly.
Working solution to upload everything to one folder
If the left hand side of the condition is the name of the blob, based on what you've said you want, the right hand side should literally be the word Library ... no expressions or anything else.
Your condition, in plain English, says ...
If the name of the blob contains the word "Library", do the true side, else, do the false.
If you want to check for the word Library ignoring case, wrap the blob name in a toLower() expression and set the right hand side to be all lower case, like thus ... library
Related
I am using a Copy Data activity to upload the contents of a database table as .csv files to an SFTP server using a self-hosted Integration Runtime.
The "owners" of the FTP site have pointed out that they are seeing "strange" filenames, i.e. a guid appended to the designated filename. When I look at the uploaded files, however, that suffix is gone.
It appears therefore that the Copy Data activity
(a) creates the file with a guid in the name,
(b) streams the content into the file, and
(c) renames the file at the end.
Can somebody confirm or deny this? Has anybody else seen this behaviour?
(The problem with this is obviously that step (a) triggers some processing in another system, which is a problem for me).
Any help is greatly appreciated.
Thank, Martin
The behavior you are describing indicates you have "Upload with temp file" checked (which I believe is the default) in your SFTP source:
You may uncheck this box if you don't want it or if the server doesn't support it.
I am using Data Factory V2 and have a dataset created that is located in a third-party SFTP. The SFTP uses a SSH key and password. I was successful with creating the connection to the SFTP with the key and password. I can now browse the SFTP within Data Factory, see the only folder on the service and see all the TSV files in that folder.
Naturally, Azure Data Factory asked for the location of the file(s) to import. I use the "Browse" option to select the folder I need, but not the files. I want to use a wildcard for the files.
When I opt to do a *.tsv option after the folder, I get errors on previewing the data. When I go back and specify the file name, I can preview the data. So, I know Azure can connect, read, and preview the data if I don't use a wildcard.
Looking over the documentation from Azure, I see they recommend not specifying the folder or the wildcard in the dataset properties. I skip over that and move right to a new pipeline. Using Copy, I set the copy activity to use the SFTP dataset, specify the wildcard folder name "MyFolder*" and wildcard file name like in the documentation as "*.tsv".
I get errors saying I need to specify the folder and wild card in the dataset when I publish. Thus, I go back to the dataset, specify the folder and *.tsv as the wildcard.
In all cases: this is the error I receive when previewing the data in the pipeline or in the dataset.
Can't find SFTP path '/MyFolder/*.tsv'. Please check if the path exists. If the path you configured does not start with '/', note it is a relative path under the given user's default folder ''. No such file .
Why is this that complicated? What am I missing here? The dataset can connect and see individual files as:
/MyFolder/MyFile_20200104.tsv
But fails when you set it up as
/MyFolder/*.tsv
I use Copy frequently to pull data from SFTP sources. You mentioned in your question that the documentation says to NOT specify the wildcards in the DataSet, but your example does just that. Instead, you should specify them in the Copy Activity Source settings.
In my implementations, the DataSet has no parameters and no values specified in the Directory and File boxes:
In the Copy activity's Source tab, I specify the wildcard values. Those can be text, parameters, variables, or expressions. I've highlighted the options I use most frequently below.
You can specify till the base folder here and then on the Source Tab select Wildcard Path specify the subfolder in first block (if there as in some activity like delete its not present) and *.tsv in the second block.
enter image description here
We are working on a solution of downloading the file posted in share point and movig it to diffrent blob
based on the clinet: xyz,dtf
And her we have same URL below example:
https://ABC.sharepoint.com/teams/ODS/XYZ_Backup/
https://ABC.sharepoint.com/teams/ODS/dtf_Backup/
can we change the folder id and coresponding folder path of blob on fly ,that is parameterize these values for each client something like a lookup table
Also is it possible to change the file name before downloading or after adding a date to it.
I have read the documentation for gdrive here, but I couldn't find a way to do what I want to do. I want to write a bash script to upload automatically a specific folder from my hard drive. The problem is that when I upload it several times, instead of replacing the old folder by the new one, it generates a new folder with the same name.
I could only find the following partial solutions to my problem:
Use update to replace files. The problem with this partial solution is that new files inside the folder could not get uploaded automatically, and I would have to change the bash script every time a new file is produced in the folder that I want to upload.
Erase the folder by its id from google drive and then upload the folder again. The problem here is that whenever I do this, the id of the uploaded folder chagnes, so I couldn't find a way to write a script to do the work.
I am looking for any method that solves my problem. But the precise questions that could help me are:
Is there a way to delete a folder from google drive (using gdrive) by its name instead of by its id?
Is there a way to get the id of a folder by its name? I guess not, since there can be several folders with the same name (but different ids) uploaded. Or am I missing something?
Is there a way to do a recursive update to renew all files that are already inside the folder uploaded on google drive and in addition upload those that are not yet uploaded?
In case it is relevant, I am using Linux Mint 18.1.
Is there a way to delete a folder from google drive (using gdrive) by its name instead of by its id?
Nope. As your next question observes, there can be multiple such folders.
Is there a way to get the id of a folder by its name? I guess not, since there can be several folders with the same name (but different ids) uploaded. Or am I missing something?
You can get the ids (plural) of all folders with a given name.
gdrive list -q "name = 'My folder name' and mimeType='application/vnd.google-apps.folder' and trashed=false"
Is there a way to do a recursive update to renew all files that are already inside the folder uploaded on google drive and in addition upload those that are not yet uploaded?
Yes, but obviously not with a single command. You'll need to write a short script using gdrive list and parse (awk works well) the output.
I am using the Azure Api provided by Microsoft for cloud storage...I am facing an unusual bug while creating the sub folder.
i.e when I create a subfolder within any container it is created easily within seconds. But when I try to create a subfolder again with different name it takes more time as compared to the previous one.
Again I try it is created easily. It means the sub folder no. 1 , 3 ,5 ,7 and so on are created easily and the even no. such as 2, 4 etc sub folders are created with delay.
i.e "Alternate sub folder creation is taking too much time"
Please let me know if there is any solution for this bug...
Are you creating these in blob storage? Folders, strictly speaking don't exist. They are simply a '/' character (or whatever delimiter you specify) are that imbedded in the file names. So you save a file using a path with out implicitly creating the path.
In the documentation, they even refer to these as "vitual folders" because they don't actual exist.