I need to untar hundreds of tar files of a size between 30 and 140GB as soon as they become available. I've created a pool with (currently) 5 "standard a1" single core Linux nodes, " microsoft-azure-batch centos-container 7-4 (latest)"
The batch has an associated storage account. The storage contains the tar files which I wish to untar into that same store. There is also a job called, "untarjob"
Since the batch allows for an association to a store I made an assumption that would be naturally available to each compute node. However, the assumption now is perhaps that isn't true? As such I'm trying to mount the store and run the unix tar command to untar as shown below.
This isn't working:
az batch task create --task-id t80000 --job-id untarjob --command-
line "mount -t cifs //dummy.file.core.windows.net/s01 /tarfiles -o
vers=3.0,username=dummy,password=Pv4<snipp>N8Imu34reseEEe==,dir_mode=0777,
file_mode=0777,sec=ntlmssp ; tar -xvf /tarfiles/80000.tar"
Could you tell me how I could solve this please?
Many thanks
Kevin
First, you probably shouldn't use a container-specific VM image if your tasks don't need containers and need to run directly on the host.
For the issue, wrap your command in a shell invocation:
/bin/sh -c 'sudo mkdir /tarfiles; sudo mount ...; tar ...'
Note the single quotes since you're using double quotes for the --command-line arg. Also, if your command needs to run privileged entirely, consider using user identities to avoid having to use sudo.
Related
i run a mixed windows and linux network with different desktops, notebooks and raspberry pis. i am trying to establish an off-site backup between a local raspberry pi and an remote raspberry pi. both run on dietpi/raspbian and have an external hdd with ntfs to store the backup data. as the data to be backuped is around 800GB i already initially mirrored the data on the external hdd, so that only the new files have to be sent to the remote drive via rsync.
i now tried various combinations of options including --ignore-existing --size-only -u -c und of course combinations of other options like -avz etc.
the problem is: nothing of the above really changes anything, the system tries to upload all the files (although they are remotely existing) or at least a good number of them.
could you give me hint how to solve this?
I do this exact thing. Here is my solution to this task.
rsync -re "ssh -p 1234” -K -L --copy-links --append --size-only --delete pi#remote.server.ip:/home/pi/source-directory/* /home/pi/target-directory/
The options I am using are:
-r - recursive
-e - specifies to utilize ssh / scp as a method to transmit data, a note, my ssh command uses a non-standard port 1234 as is specified by -p in the -e flag
-K - keep directory links
-L - copy links
--copy-links - a duplicate flag it would seem...
--append - this will append data onto smaller files in case of a partial copy
--size-only - this skips files that match in size
--delete - CAREFUL - this will delete local files that are not present on the remote device..
This solution will run on a schedule and will "sync" the files in the target directories with the files from the source directory. to test it out, you can always run the command with --dry-run which will not make any changes at all and only show you what will be transferred and/or deleted...
all of this info and additional details can be found in the rsync man page man rsync
**NOTE: I use ssh keys to allow connection/transfer without having to respond to a password prompt between these devices.
After setting up my Raspberry Pi, I made an image to make reverting to older software states easier. Recently I wanted to do that so I saved the content of my /home/pi folder, formated the sd-card and wrote the image onto it.
So far everything worked fine. Then I tried to simply delete the complete /home/pi folder and replace it with my previously saved folder from the old image. Now it seems like all files are there. But it doesnt boot correctly.
At some point it just stops to boot. I can then use it normally like the terminal, but Desktop is not starting.
So, how can I replace my home directory the right way so I don't make any damage to the system?
edit:
I just tried to do this again.
sudo cp -a /home/pi/fileserver/backup /home/backup
(i mounted a network drive in fileserver. Since network is on windows i assume all permissions are already gone here)
cp -a /home/pi/. /home/original
sudo umount /home/pi/fileserver
rm -r /home/pi/
mv /home/backup /home/pi
sudo chmod -R 755 /home/pi (So far everything still works)
sudo reboot
After reboot it doesnt boot correctly anymore. When I wait long enough I see errors of X Server.
That's quite doubtful approach to archiving the data. First of all, as you mentioned, windows will remove the permission bits. Running chmod -R 755 afterwards has very bad consequences because some programs in order to work require very specific access bits on some files (ssh keys for example). Not to mention that making everything executable is bad for security.
Considering your scenario, you may either
a) backup everything into Tar or Zip archives - this way permissions will be intact
b) Make virtual disk file which will be stored on shared windows drive and mounted to /home/pi
How to do scenario A:
cd /home/pi
tar cvpzf backup.tar.gz .
Copy backup.tar.gz to shared drive
to unpack:
cd /home/pi
tar xpvzf backup.tar.gz
Pros:
One-line backup
Takes small amount of space
Cons:
Packing/unpacking takes time
How to do scenario B:
1) Create a new file to hold the virtual drive volume:
cd /mnt/YourNetworkDriveMountPoint
fallocate -l 500M HomePi.img
dd if=/dev/zero of=HomePi.img bs=1M count=500
mkfs -t ext3 HomePi.img
2) Mount it to home dir
mount -t auto -o loop HomePi.img /home/pi/
500 means the disk will be 500 megabytes in size
This way your whole pi will be saved as a file on windows shared drive, but all the content will be in ext3 so all permissions are preserved.
I suggest you though to keep the current version image file on Pi device itself and the old versions on shared drive. Just copy files over if you need to switch because otherwise if all images are on shared drive then read/write performance will be 100% dependant on network speed.
You can then easily make copies of this file and swap them instantly by unmounting existing image and mounting new one
Pros:
Easy swap between backup versions
Completely transparent process
Cons:
If current image file is on shared drive, performance will be reduced
It will consume considerably more space because all 500 megs will be preallocated.
Pi user must be logged off during image swap for obvious reasons
Now, as for issues with Desktop not displayed, you need to check /var/log/Xorg.0.log for detailed messages. Likely this is caused by messed permissions. I would try to rename/remove your current Xorg settings and cache which are located somewhere in /home/Pi/.config/ (depends on what you're using - XFCE, Gnome, etc.) and let X server recreate them. But again, before doing this please check Xorg.0.log for exact messages - maybe there's another error. If you need any further help please comment to this answer
I'm having a hard time copying files over to my Google Compute Engine. I am using an Ubuntu server on Google Compute Engine.
I'm doing this from my OS X terminal and I am already authorized using gcloud.
local:$ gcloud compute copy-files /Users/Bryan/Documents/Websites/gce/index.php example-instance:/var/www/html --zone us-central1-a
Warning: Permanently added '<IP>' (RSA) to the list of known hosts.
scp: /var/www/html/index.php: Permission denied
ERROR: (gcloud.compute.copy-files) [/usr/bin/scp] exited with return code [1].
insert root# before the instance name:
local:$ gcloud compute copy-files /Users/Bryan/Documents/Websites/gce/index.php root#example-instance:/var/www/html --zone us-central1-a
The reason this doesn't work is that your username does not have permissions on the GCE VM instance and so cannot write to /var/www/html/.
Note that since this question is about Google Compute Engine VMs, you cannot SSH directly to a VM as root, nor can you copy files directly as root, for the same reason: gcloud compute scp uses scp which relies on ssh for authentication.
Possible solutions:
(also suggested by Faizan in the comments) this solution will require two steps every time
use gcloud compute scp --recurse to transfer files/directories where your user can write to, e.g., /tmp or /home/$USER
login to the GCE VM via gcloud compute ssh or via the SSH button on the console and copy using sudo to get proper permissions:
# note: sample command; adjust paths appropriately
sudo cp -r $HOME/html/* /var/www/html
this solution is one step with some prior prep work:
one-time setup: give your username write access to /var/www/html directly; this can be done in several ways; here's one approach:
# make the HTML directory owned by current user, recursively`
sudo chown -R $USER /var/www/html
now you can run the copy in one step:
gcloud compute scp --recurse \
--zone us-central1-a \
/Users/Bryan/Documents/Websites/gce/index.php \
example-instance:/var/www/html
I use a bash script to copy from my local machine to writable directory on the remote GCE machine; then using ssh move the files.
SRC="/cygdrive/d/mysourcedir"
TEMP="~/incoming"
DEST="/var/my-disk1/my/target/dir"
You also need to set GCE_USER and GCE_INSTANCE
echo "=== Pushing data from $SRC to $DEST in two simple steps"
echo "=== 1) Copy to a writable temp directoy in user home"
gcloud compute copy-files "$SRC"/*.* "${GCE_USER}#${GCE_INSTANCE}:$TEMP"
echo "=== 2) Move with 'sudo' to destination"
gcloud compute ssh ${GCE_USER}#${GCE_INSTANCE} --command "sudo mv $TEMP/*.* $DEST"
In my case I don't want to chown the target dir as this causes other problems with other scripts ...
I had the same problem and didn't get it to work using the methods suggested in the other answers. What finally worked was to explicitly send in my "user" when copying the file as indicated in the official documentation. The important part being the "USER#" in
gcloud compute scp [[USER#]INSTANCE:]SRC [[[USER#]INSTANCE:]SRC …] [[USER#]INSTANCE:]DEST
In my case I could initially transfer files by typing:
gcloud compute scp instance_name:~/file_to_copy /local_dir
but after I got the permission denied I got it working by instead typing:
gcloud compute scp my_user_name#instance_name:~/file_to_copy /local_dir
where the username in my case was the one I was logged in to Google Cloud with.
UPDATE
gcloud compute copy-files is deprecated.
Use instead:
$ gcloud compute scp example-instance:~/REMOTE-DIR ~/LOCAL-DIR \ --zone us-central1-a
More info:
https://cloud.google.com/sdk/gcloud/reference/compute/scp
The updated solution for this exact issue (2020)
For the sake of exposition, we have to break the issue in two parts. The "copy-files" command is officially depreciated and we are to use "scp", however both old and new options are limited to certain folders only.
Since we do have access to the /tmp folder, this means we can easily move our distribution files with the preferred "scp" command, as a staging step.
More importantly we also have access to execute scripts, or commands remotely via SSH on the instance which means the limited access is no longer an issue.
Example Time
The first part is to copy the dist folder, and all it's content recursively to the tmp folder to which gloud does give access:
gcloud compute scp --recurse dist user_name#instance:/tmp
The second part leverages the fact that we can run commands remotely via ssh:
gcloud compute ssh user_name#instance --command "sudo bash golive"
(or any other command you may need to execute)
More importantly this also means that we can just copy our distribution files to the final destination using sudo and the "cp" copy function:
gcloud compute ssh user_name#instance --command "sudo cp -rlf /tmp/dist/* /var/www/html/"
This completely eliminates the need to set the permissions first through the ssh terminal.
This is to copy files from remote machine to your machine. And make sure you have ssh setup because this will use default ssh keys.
This worked for me:
gcloud compute scp 'username'#'instance_name':~/source_dir /destination_dir --recurse
This is the generic syntax, so if you want to copy files from your machine to remote machine, you can use this.
--recurse : required to copy directories with other files inside
Syntax: gcloud compute scp 'SOURCE' 'DESTINATION'
NOTE: run it without root
I am having a linux mount on my jenkins build server. After a job in jenkins succeeds, a script is being called which copies the files from workspace to different directories in the mount. Each time I mount the copy operation succeeds but after few hours it fails with I/O error: cannot copy. I have to remount the share again to get this thing going.
Any ideas on the fix? I am struggling for 2 weeks now. I do not want to remount again and again.
Command I used: mount -t cifs -o rw,noperm,username=xyz,password=* //remoteserver/path /local/path.
Thanks
Not sure if this will help you. But, this something that I do for my scripts.
You said that you have a script that copies the files from workspace to the mount. Why don't you add a condition to a script, to check if the mount exists if not remount or something like that.
I need to copy the whole contents of a linux server, but I'm not sure how to do it recursively.
I have a migration script to run on the server itself, but it won't run because the disc is full, so I need something I can run remotely which just gets everything.
I need to copy the whole contents of a linux server, but I'm not sure how to do it recursively.
How about
scp -r root#remotebox:/ your_local_copy
sudo rsync -hxDPavil -H --stats --delete / remote:/backup/
this will copy everything (permissions, owners, timestamps, devices, sockets, hardlinks etc). It will also delete stuff that no longer exists in source. (note that -x indicates to only copy files within the same mountpoint)
If you want to preserve owners but the receiving end is not on the same domain, use --numeric-ids
To automate incremental backup w/snapshots, look at rdiff-backup or rsnapshot.
Also, gnu tar is highly underrated
sudo tar cpf / | ssh remote 'cd /backup && tar xv'