I have a Azure File Copy Task as a part of my build. Some directory needs to be recursively copied to a blob container. Basically, the "cdn" directory from sources should be copied to the cdn blob container.
So, as "Source" for the task, i specified "$/Website/AzureWebsite/www.die.de/cdn-content/cdn/*"
As "Container Name" i specified "cdn".
The task works: My files do get copied. However, after the copying ends, i also see a directory named "$tf" which has various subdirectories with numbers as names. (0, 1, 2, etc.). All of those contain files named "*.gz" or ".rw".
Where is this coming from and how do i get rid of it?
I found this thread: https://developercommunity.visualstudio.com/content/problem/391618/tf-file-is-still-created-in-a-release-delete-all-s.html, the $tf folder is generated when mapping sources for TFVC repository, it's by design. It will create a temp workspace and map the sources first when you queue a build.
If you want to get rid of these, then set the Workspace type to a server workspace, but lose the advantages of local workspaces. See: TFS creates a $tf folder with gigabytes of .gz files. Can I safely delete it? for guidance.
Related
I am working on a script solution to capture status changes of an Azure DevOps repo.
Basically here is what we need:
The script is supposed to create a file list that contains ALL files in an Azure repo, and save it to local drive.
If the newly generated file list has changes, i.e., a file gets removed/added, the script should create a separate file list that records these changes.
Step 2 I can take care of myself.
But since I am not familiar with the DevOps API, could anyone help me on this?
Thank you in advance
For the first question, you can fetch repo to local drive, and list files in this directory.
If you could fetch repo to local drive, you can git log -1 for latest change commit.
I created a Copy Data task in Azure Data Factory which will periodically copy modified files from my file system (self-hosted integration runtime) to an Azure Blob location. That works great when an existing file is modified or when a new file is created in the source, however, it won't delete a file from the blob destination when the corresponding file is deleted from the source file path location - since the file is gone, there is no modified date. Is there a way to keep the source and destination in sync via Azure Data Factory with individually deleted files such as in the scenario described above? Is there a better way to do this? Thanks.
I'm afraid to say Data Factory can't do that with actives, the pipeline only support read the exist file and copy them to sink. And sink side also doesn't support delete a file.
You can achieve that in code level, such as functions or notebook. After the copy finished, build a logic to compare the source and destination files list, delete the file which not exist in source list.
HTH.
Problem: I get an email with a zip file. In that zip are two files. I want to extract one of those files and place it in a folder in ADL.
I've automated this before using logic apps but the zip and extra file is throwing a wrench in the gears here. So far I've managed to get a logic app going to download the zip into a blob container and another logic app to extract the files to another container. Don't know how to proceed from there. Should I be using data factory? I want this automated and to run every week every time I receive an email from a specific sender.
Update:
I am sorry, dont notice your source is ADL, the below steps only need to change the source as ADL is ok. the key is select the Compression type of your source, it will unzip the file for you.
Original Answer:
Create a pipeline,
2.Create a activity.
3.After you create a copy data activity, you need to choose the source and the sink. From your description, you need to unzip a file in a storage container to another container. So, please follow these steps:
And the sink is similar, also choose the azure storage blob and choose the same linked service. Select the container that you want to copy to.
4.Then let's Validate all. If there is no problem, we can publish them.
Now please trigger your pipeline:
6.After that, your zip file will successful unzip and copy to another container.:)
I've been reading about Perforce but haven't found any comprehensive explanation of relationship between workspace and working directory, e.g. how files appear in working directory from workspace, how they are tracked, what inconsistencies are possible between workspace files and working directory files etc.
I come from git background, and so I'm looking for the description of workspace and working directory interaction similar to index and working directory interaction in git.
For comprehensive information, the full Perforce documentation is available online. But here's a basic summary of the terms and concepts:
Perforce is a client-server system. The server tracks changes to files. Developers perform modifications to files using copies of those files on their machines, arrange those modifications into units called changelists, and submit those changelists to the server when they are ready.
All information about files is stored on the server. A client, or workspace, is a set of configuration data about a single copy of versioned files stored on a developer's workstation. For each workspace, the server keeps track of: which files are currently sync'd to that workspace, which pending changelists are being constructed, which user is using those files, in which directory on which computer the workspace resides, etc.
Getting a copy of versioned files into your workspace is called sync; submitting a new changelist with new file versions is called submit. As other users submit modifications of files, your workspace gradually becomes out of date; to bring it up to date you issue the sync command, possibly followed by the resolve command to merge newly-submitted changes to files you have been editing.
There isn't an exact analog of the git index construct. Modified files are copied from the developer's workspace on their workstation to the server by the submit command, and then are stored permanently in the server's archives. A somewhat different workflow is available in Perforce called a shelf. You can construct a changelist with modifications, and then use the shelve command to store that pending changelist on the server in shelved state. Pending and shelved changelists have several similarities to the way that git add allows you to assemble a commit prior to committing it, but there are also a number of differences, starting with the fact that Perforce tracks that information on the server, not in the client.
Perfore does not really have the concept of the "index/staging area" that Git does. Files only exist within the Perforce server (the depot) or in your local working directory.
The workspace is just a bundle of configuration data that tells Perforce what files to put where in your local environment.
It is effectively a "binding" that specifies what files to select from the depot and where to place them in your working directory. This flexibility can be incredibly useful in some circumstances, but it is also an extra level of indirection compared to other VCS's such as Git/Mercurial/Subversion.
If you look at the contents of a workspace definition you will see a "Root:" section that points to a location on your local machine, and a "View:" section that tells Perforce what files you want synced to that location.
The Perforce documentation is a bit confusing with regards to its terminology. In the docs I've seen, "Workspace" refers to BOTH the specific workspace definition that you create (which is a data structure held within the Perforce server) AND the local working directory that the workspace is bound to, fairly interchangably. This is because the whole point of a workspace is to bind to a specific local working directory.
Take a look at example 15 here: Managing files and changelists
The output of the p4 sync command shows that it is writing files to the C: drive. When you "retrieve files from the depot into your client workspace" that DOES actually mean that files get copied into your local working directory.
Let's take a workspace definition for example. The key fields for this explanation are Client, Root, and View, so I'll only show those for clarity:
Client: mysimpleclient
Root: /Users/johndough/myspecialproject
View:
//depot/foo/... //mysimpleclient/...
The Client field defines the name of the workspace. The Root field defines the root of the workspace on the local filesystem. The View field describes how files from the server will be mapped to your workspace files. (This adds some flexibility when compared to Git, but at the cost of a bit of a learning curve). I've used a very simple mapping here, but this could be a lot more complex.
Let's break down the View field. Let's say the server has a file //depot/foo/bar.txt.
When you sync (get latest revision) of this file, Perforce will use the view line to determine where the file lands on your filesystem.
Here is our view again:
//depot/foo/... //mysimpleclient/...
The left side is in depot syntax, the right side is in workspace syntax, and it always starts with //workspace_name. The "..." is a recursive wildcard (kind of like ** with globbing). You can mentally replace the workspace name with the value of the root, to make sense of this.
Remember our root is: /Users/johndough/myspecialproject
So, after replacing "//mysimpleclient" with that value we arrive to:
//depot/foo/... /Users/johndough/myspecialproject/...
Let's use that modified view line to figure out where bar.txt will end up:
//depot/foo/bar.txt /Users/johndough/myspecialproject/bar.txt
This is how it works when you pull files down. When you add new files, the mapping is read in reverse. Let's say you create a file /Users/johndough/myspecialproject/fred/file.txt
Our (mentally modified) view is:
//depot/foo/... /Users/johndough/myspecialproject/...
We interpret this in reverse. Concatenate the relative path of the new file with the root of the workspace:
/Users/johndough/myspecialproject/fred/file.txt
And then do the same for the left side:
//depot/foo/fred/file.txt
Putting it altogether:
//depot/foo/fred/file.txt /Users/johndough/myspecialproject/fred/file.txt
Hopefully this helps. The notion of workspaces is probably the most difficult aspect of getting started with Perforce. Happy coding!
i am using debian. i need to run daily incremental backup for files that are modified since last backup. this is local backup from one disk to another in same system.
directory structure will be same as source
each folder and subfolder has its own compressed archive containing files-only
on first backup, folder structure will be made as source, and files will be archived
on each subsequent backup, newly created folders will be created and new files will be archived to destination. files modified since last backup, will be updated into relevant archive in destination.
is it possible with rsync on any other efficient method?? please help.