I tried to do some research but unable to find the answer.
Let's say you have container A.
at 1pm, container A had the following files:
test1.txt (inside the file is abcdefg)
test2.txt (inside the file is klmnopq)
AzCopy was performed to container A , copying it from storage C to storage D. It started at 1.01pm and finished at 1.05pm.
At 1.02pm, mr.X performed the following:
Add test3.txt inside container A
Modify test1.txt to "defgh"
The question is, what does end up in storage D for container A copy?
the original files? ie test1.txt (inside the file is abcdefg) and test2.txt (inside the file is klmnopq)
or something else?
thanks
The outcome depends on the specific timing of the writes/copies. AzCopy makes sure to guarantee the file integrity by checking the LastModified time of the source during the copy. In other words, whatever is at the source when AzCopy scanning happened will be locked on and copied.
Related
As the title suggests I am trying to copy all files with a specific extension, within a folder structure, to blob storage without recreating the local folder structure;
This works fine when I run the following;
azcopy cp 'H:\folder1\folder2\*.txt' 'https://storage.blob.core.windows.net/folderA/folderB/?saskey'
This copies all *.txt files to /folderB
I have tried many variations of the following;
azcopy.exe cp 'H:\folder1\*\*' 'https://storage.blob.core.windows.net/folderA/folderB/?saskey' --recursive --include-pattern '*.txt'
Regardless of what I try I end up with the following;
/folderA/folderB
/folder1/fileA.txt
/folder2/fileB.txt
I was under the impress that is what the "--recursive" switch was for, but what I am doing is either not supported or my syntax is wrong.
I have read through this;
https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-files#use-wildcard-characters
I could probably script it with something similar to this;
AzCopy - Wildcards In Middle Of Pattern?
But was hoping this was built-in functionality
What you are looking for is not supported. Using --recursive would result in the subdirectory structure of the source retained in the destination. I am not aware of any flag to prevent that.
Actually that helps to avoid conflict. Let's say for example, you have files /folder1/fileA.txt and /folder2/fileA.txt in source. If you try to copy flat in destination (without subpath), that would have caused conflict since both file names are fileA.txt.
How to take backup of file without changing its time-stamp with Ansible playbook? I tried backup=yes but the problem is like it changes the timestamp os the file.
Code:- dest={{item}} state=absent regexp='TLSv1' backup=yes with_items: ('{{certs_dir.stdout_lines}}')
I'm retracting my initial "It can't be done" response - it should be possible by using a series of plays, but it's not very pretty.
If you really need the backup file to keep the time-stamp, you might want to put in an official request on the developer mailing list.
Use the stat module on the initial file to retrieve the file timestamp
Register the backup file name in the return value backup_file from the file or copy module.
Use the command module to call the touch command to set the time of the backup_file to the original time. (The Ansible stat module does not adjust file timestamps.)
I have a process that periodically gets files from a server and copy them with SFTP to a local directory. It should not overwrite the file if it already exists. I know with something like Winston I can automatically rotate the log file when it fills up, but in this case I need a similar functionality to rotate files if they already exist.
An example:
The routine copies a remote file called testfile.txt to a local directory. The next time it's run the same remote file is found and copied. But now I want to rename the first testfile.txt to testfile.txt.0 so it's not overwritten. And so on - after a while I'd have a directory of files with the name testfile.txt.N and the most recent testfile.txt.
What you can do is you can append date and time on the file name that gives every filename a unique name and also helps you archive it.
For example you text.txt can be either 20170202_181921_test.txt or test_20170202_181921.txt
You can use a JavaScript Date Object to get date and time.
P.S show your code of downloading files so that I can add more to that.
I have uploaded with Cloudberry Explorer some files/folders to my Azure container but now I'm gonna change Cloudberry for AzCopy.
What I need is to omit those uploaded files. I don't know if can be done with a AzCopy parameter. the files to be uploaded are stored in a server so doing it manually is impossible due for are thousands of thousands of files/folders.
thanks in advance
As it is documented in azcopy reference
--overwrite string Overwrite the conflicting files and blobs at the destination if this flag is set to true. Possible values include 'true', 'false', 'ifSourceNewer', and 'prompt'. (default "true")
So something like this should work:
azcopy.exe copy "source location" "destination location" --overwrite=false
Use /XO flag in the command. It will not copy/replace old files. Sample command,
AzCopy /Source:C:\myfolder /Dest:https://myaccount.blob.core.windows.net/mycontainer /DestKey:key /XO
If the files uploaded by another tool has different naming convention with the new ones, you could use option /Pattern to upload only the new files,
e.g. old files have naming convention like “abcxxxx”, new files have naming convention like “xyzxxx”, then please specify /Pattern:xyz* to copy the new files only.
Or use option /xo (means exclude old files) to copy new files only, note that AzCopy will compare local files' change time with the 'Last Modified Time' of the destination blobs when you specified option /xo and /xn, please make sure the uploaded old files’ ‘Last modified time’ is same or newer than the their local copies’ change time, otherwise the old files will be uploaded again when you specified option /xo. You can use option /MT to set ‘Last Modified Time’ as same as local copies’ change time during upload.
For more details, please visit http://aka.ms/azcopy
Thanks
I have a file that i want to add to sourcecontrol on linux using cleartool .
I've followed the IBM documentation for this, i've tried this:
cleartool mkelem testScript.sh
I got an error: Can't modify directory "." because it is not checked out.
I also would like to know how can i checkout/checkin files or directories and setting activities.
You need to checkout the parent folder first.
cd /path/to/file/
cleartool mkact newfile
cleartool checkout -c "add file" .
cleartool mkelem testScript.sh
cleartool checkin -nc
The cleartool mkact would work if you are in an UCM view.
It will create and set a new activity, which will record the files and folder you will modify.
Here, the new activity newFile will record the new version of the parent folder, as well as the version 0 and 1 of the file.
You should create separate questions for .. separate questions...
Going back to the original - the reason why it isn't working is, as VonC has pointed out, you haven't checked out the parent of the file. Remember, when you run "cleartool mkelem", you are about to modify the contents of the parent directory (. in this case) by adding a new "pointer" to the element you're now creating. As with everything else in clearcase, when you want to modify the contents of an element, you have to check it out first.
One of ClearCase's greatest strength (and hardest to wrap one's head around) is the concept of an "element", IMO. "Everything" behaves similarly with an element. Making any change to an "element" (file or directory) means you have to check it out first to make that change.
In the case of a file, that's easy to grasp - you're just editing lines in a file. For a directory, it's almost as easy - you can think of a directory as just a list of pointers to data blobs. We make the name of the blob something convenient we can remember (like foo.java or myapplication.cc or README.md). But we can also change the name of the pointer (even though it points to the same data blob) by renaming a file. We can remove the pointer to the blob without impacting the blob itself by using "rmname". That's essentially what "rmname" does.
In ClearCases' case, the mkelem command is a little bit special - it creates the initial datablob, and adds a pointer to that datablob in the current directory (kind of does 2 things at once).