The goal is I want to monitor one directory from different server. For example the remote server is user#host.
I have list.txt that contents list of file that will be moved. And list.txt located in a remote server.
Currently I have this code.
ssh user#host cat /full-path/list.txt |
{
while read line;
do mv user#host:/full-path/$line user#host:/full-path/done/;
done;
}
When I run the code above, error exists. There's no such file or directory.
But when I log in to user#host and cat one file randomly from list.txt, the file exists.
The while loop runs on the local server. You need to put the script in quotes so it's an argument to the ssh command.
... Or a here document, like this:
ssh user#host <<':'
while read line; do
mv /full-path/"$line" /full-path/done/
done </full-path/list.txt
:
... or more succinctly
ssh user#host 'cd /full-path && xargs -a list.txt mv -t done'
Notice also the absence of a useless cat and the local file name resolution (mv would have no idea about the SSH remote path syntax you were trying to use).
Related
I need to download only defined files with wget and ftp.
For example:
1.I retrieve all files using:
echo ls -R | ftp ftp://user:password#host > ./list.txt
2.Then I will parse the result and get a list with absolute paths for each file:
/path/to-the/file-1
/path/to-the/file-2
etc.
3.And now I need to download all files from the result list using wget and ftp.
And I don't want to create a separate FTP session for each file download process.
Please give your advice. Thank you.
Update:
For recursive download I'm using it: wget -r ftp://user:password#host:/ -nH -P /download/path. It works great, but I need to pass a file with a list of remote files for downloading via FTP with one FTP session.
Sorry, I missed the "single session" part when I commented. I think you need to have your script generate a second script to run a single FTP session.
So, your script will not do any FTP itself, it will just write another script that does the transfers. So, it will write a script that does this
ftp -n <SOMEADDRESS> <<EOS
quote USER <USERNAME>
quote PASS <PASSWORD>
bin
get file1 localname1
get file2 localname2
...
get fileN localnameN
quit
EOS
Then it will execute that script, by doing:
bash < thatScript
So your script will look like this:
#!/bin/bash
ScriptName=funkyFTPer
cat - <<END > $ScriptName
ftp -n 192.168.0.1 <<EOS
quote USER freddy
quote PASS frog
END
# Your selection code goes here ***PHNQZ***
echo get file1 localname1 >> $ScriptName
echo get file2 localname2 >> $ScriptName
echo get fileN localnameN >> $ScriptName
echo quit >> $ScriptName
echo EOS >> $ScriptName
echo "Now run bash < $ScriptName"
Then delete the script as it contains your password. Or you can put the password in your .netrc file.
As regards creating directories locally, you can do that in the first script using mkdir -p. The -p has the advantage that it creates all directories in between in one go and doesn't get upset if they already exist.
So, just looking at the area of code where it says ***PHNQZ*** above, let's say your code decides you need file freddy/frog/c.txt, you could do:
remotename="freddy/frog/c.txt"
localdir=${remotename%/*} # Get just directory part using "bash Parameter Substitution"
mkdir -p "$localdir" # make directory and all parts in between
I am trying to use scp to copy large log files from a remote server. However I want only the lines in remote log file that has a string "Fail".
This is how I am doing it currently
scp user#ip:remote_folder/logfile* /localfolder
This copies all the files starting with logfile in remote server to my local folder. The files are pretty large and I need to copy only the lines in those log file, containing the string "Fail" from remote server. Can any body tell me how to do this? Can I use cat or grep command?
Use grep on the remote machine and filter the output into file name and content:
#!/usr/bin/env bash
BASEDIR=~/temp/log
IFS=$'\n'
for match in `ssh user#ip grep -r Fail "remote_folder/logfile*"`
do
IFS=: read file line <<< $match
mkdir -p `dirname $BASEDIR/$file`
echo $line >> $BASEDIR/$file
done
You might want to look at an explanation to IFS in combination with read.
ssh user#ip grep Fail remote_folder/logfile*
I have a shell script that I am using to compare directory contents. The script has to ssh to different servers to get a directory listing. When I run the script below, I am getting the contents of the server that I am logged into's /tmp directory listing and not that of the servers I am trying to ssh to. Could you please tell me what I am doing wrong?
The config file used in the script is as follows (called config.txt):
server1,server2,/tmp
The script is as follows
#!/bin/sh
CONFIGFILE="config.txt"
IFS=","
while read a b c
do
SERVER1=$a
SERVER2=$b
COMPDIR=$c
`ssh user#$SERVER1 'ls -l $COMPDIR'`| sed -n '1!p' >> server1.txt
`ssh user#$SERVER2 'ls -l $COMPDIR'`| sed -n '1!p' >> server2.txt
done < $CONFIGFILE
When I look at the outputs of server1.txt and server2.txt, they are both exactly the same - having the contents of /tmp of the server the script is running on (not server1 or 2). Doing the ssh +dir listing on command line works just fine. I am also getting the error "Pseudo-terminal will not be allocated because stdin is not a terminal". Adding the -t -t to the ssh command isnt helping either
Thank you
I have the back ticks in order to execute the command.
Backticks are not needed to execute a command - they are used to expand the standard output of the command into the command line. Certainly you don't want the output of your ssh commands to be interpreted as commands. Thus, it should work fine without the backticks:
ssh user#$SERVER1 "ls -l $COMPDIR" | sed -n '1!p' >>server1.txt
ssh user#$SERVER2 "ls -l $COMPDIR" | sed -n '1!p' >>server2.txt
(provided that double quotes to allow expansion of $COMPDIR are used).
first you need to generate keys to login to remote without keys
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub remote-host
then try to ssh without pass
ssh remote-host
then try to invoke in your script but first make sanity check
var1=$(ssh remote-host) die "Cannot connect to remote host" unless $var1;
We have a repository of about 3000 MP3 files. Many of these files have our old domain name within their name.
For example: somesong_oldDomainName.mp3
I need to SSH into the site, find all instances of those files and rename them with the new domain name.
Example: somesong_NEWDomainName.mp3
I know the basic SSH commands but not something advanced like this.
Pretty sure it'll be a combination of multiple commands.
Assuming you get an interactive shell when you ssh into your linux server, this might be a possible way:
ssh user#machine-name-or-ip
then you will get some sort of terminal like
user#machine-name:~$
where you enter the commands to execute on that remote machine.
As mentioned in the comments, the answer here might just fit very well:
Bash: Rename small part of multiple files in middle of name
user#machine-name:~$ for i in *.mp3; do mv "$i" "$(echo "$i" | sed 's/_oldDomainName/_NEWDomainName/g')"; done
This assumes, your current directory is the one with all the MP3 files in it.
If you dont want interactivly operate on your files, e.g. because they change very often and you want a script to perform this action, SSH can also execute a command and/or shell script remotely.
To pass the command directly with the SSH call:
SSH error when executing a remote command: "stdin: is not a tty"
To pipe a local shell script into the SSH connection: How to use SSH to run a shell script on a remote machine?
Run a remote shell script via SSH: how to run a script file remotely using ssh
Edit:
Assume you are connected via SSH to your remote machine and have somewhat similar versions of bash and sed, it should work like this:
$ ls
bar_chosefil.mp3 boo_chosefil.mp3 foo_chosefil.mp3
$ for i in *.mp3; do mv $i $(echo $i | sed 's/chosefil/tamasha/g'); done
$ ls
bar_tamasha.mp3 boo_tamasha.mp3 foo_tamasha.mp3
Versions involved:
bash: 4.2.25
sed: 4.2.1
mv: 8.13
Edit 2:
Updated the command to work with blanks in filenames
$ ls
asd chosefil.mp3 bar_chosefil.mp3 boo_chosefil.mp3 foo_chosefil.mp3
$ for i in *.mp3; do mv "$i" "$(echo "$i" | sed 's/chosefil/tamasha/g')"; done
$ ls
asd tamasha.mp3 bar_tamasha.mp3 boo_tamasha.mp3 foo_tamasha.mp3
I have a bash script that basically should work like below:
get build number from user and put it in buildNum var
prep the build on local machine by calling a local script with buildNum as it's argument
sftp the prepped zip file to remote server1
Do this:
ssh -v $server1 <<EOF
rm -rvf path1-on-remote-server1/*
cd path2-on-remote-server1
unzip ../prepped-zip-file-$buildNum.zip
exit
EOF
sftp the prepped zip file to remote server2
The problem i am having is that on the forth step of number 4, $buildNum is not known to the remote server and it fails.
I tried the following two solutions and both failed:
use double quotes "unzip ../prepped-zip-file-$buildNum.zip" which resulted in "unzip ../prepped-zip-file-11.6.zip: Command not found.
tried to get the build number again from the user during the SSH session which failed again by not even waiting for my input and looking for a zip file without the build number at the end of the name, as the var was empty,
i did :
ssh -v $server1 <<EOF
rm -rvf path1-on-remote-server1/*
cd path2-on-remote-server1
echo "enter build num once more: "
read bNum
unzip ../prepped-zip-file-$bNum.zip
exit
EOF
Any suggestions on how to achieve what i am after.
Thanks in advance
Are you sure this doesn't work?
ssh -v $server1 <<EOF
rm -rvf path1-on-remote-server1/*
cd path2-on-remote-server1
unzip ../prepped-zip-file-$buildNum.zip
exit
EOF
When I try it on my machine, with cat instead of ssh -v $server1 for testing, the variable does get substituted into the here-document, just as if the entire document had been on a command line. The remove shell never needs to know there was a variable in the first place.
Actually, though, you may want to give the remote command on the ssh command line rather than redirecting the standard input. This would be more robust in case some of the parts of it unexpectedly decide to try reading from stdin:
ssh -v $server1 "rm -rvf path1-on-remote-server1/*
cd path2-on-remote-server1
unzip ../prepped-zip-file-$buildNum.zip"
(Note that multi-line double-quoted strings are okay with bash).
#!/bin/sh
printf "Enter Build Number: "
read BUILD_NUM
cat << EOF | ssh $server1
hostname
echo "${BUILD_NUM}"
uptime
EOF
That works for me.