I need to use if function to filter out required files only when using SFTP to copy files to my server from a remote server. Here is my try to get the all data inside /filesnew.
#!/bin/bash
files=`sshpass -p 'XXX' sftp -P 2222 User1#10.18.90.12<<EOF
cd /filesnew
ls
EOF`
files=`echo $files|sed "s/.*sftp> ls//"`
(
echo cd /filesnew
for file in $files; do
echo get $file /data/processedfiles/$file
done
) |sshpass -p 'XXX' sftp -P 2222 User1#10.18.90.12
I need to filter out the files which are starting with "USER".
ex:
If($files==*USER*) then
echo get $file /data/processedfiles/$file
Can someone show me how to do this?
Use spaces around operators. Those are all arguments for commands and spaces separate them.
"If" is spelled if (lowercase) in Bash.
Testing a condition is done with [...] in Bash, not with (...).
Filtering is not comparison. Those are completely different operations. Use grep:
... | grep -E -v '^USER'
See: man grep
Related
How do I get the one by one files from SFTP server and move them do another folder in Ubuntu bash script?
#!bin/sh
FOLDER=/home/SFTP/Folder1/
sftp SFTP#ip_address
cd /home/FSTP/Folder1/
for file in "$FOLDER"*
<<EOF
cd /home/local/Folder1
get $file
EOF
mv $file /home/SFTP/Done
done
I know it's not right, but i've tried my best and if anyone can help me, i will appreciate it. Thanks in advance.
OpenSSH sftp is not very powerful client for such tasks. You would have to run it twice. First to collect list of files, use the list to generate list of commands, and execute those in a second run.
Something like this:
# Collect list of files
files=`sftp -b - user#example.com <<EOF
cd /source/folder
ls
EOF`
files=`echo $files|sed "s/.*sftp> ls//"`
# Use the list to generate list of commands for the second run
(
echo cd /source/folder
for file in $files; do
echo get $file
echo rename $file /backup/folder/$file
done
) | sftp -b - user#example.com
Before you run the script on production files, I suggest, you first output the generated command list to a file to check, if the results are as expected.
Just replace the last line with:
) > commands.txt
Maybe use SFTP internal command.
sftp get -r $remote_path $local_path
OR with the -f option to flush files to disk
sftp get -rf $remote_path $local_path
I am attempting to run multiple commands via a bash script on a remote server. specifically, the for loop to be run on the remote server is giving me issues. I suspect it is because I don't know how to properly escape characters or use $().
Below is the code.
ssh (user)#(server) <<EOF
sudo su - (username)
whoami
'for e in $(`ls -lrt /usr/jboss/jbosseap | awk '{print $9}' | grep multichannel`);
do
echo "$e";
done'
Removing user and server names for obvious reasons. Just concentrate on the for loop. when I run that for loop command line (without the $()) its works fine. Just not sure how to nest it in a remote call.
Thanks very much for any and all help!
If you've got a complex script that you're trying to run over ssh you're going to be better off putting that script in a file and piping that file into ssh like:
cat remote_script.sh | ssh user#host
or:
cat remote_script.sh | ssh user#host sudo -u username
And now you don't have to worry about N levels of escaping.
You can run it as below .
here file "list " includes your list of nodes and script should be present in all nodes
for i in $(cat list ) ;do ssh -o StrictHostKeyChecking=no $i "/path/your_script" ;done
I am trying to use scp to copy large log files from a remote server. However I want only the lines in remote log file that has a string "Fail".
This is how I am doing it currently
scp user#ip:remote_folder/logfile* /localfolder
This copies all the files starting with logfile in remote server to my local folder. The files are pretty large and I need to copy only the lines in those log file, containing the string "Fail" from remote server. Can any body tell me how to do this? Can I use cat or grep command?
Use grep on the remote machine and filter the output into file name and content:
#!/usr/bin/env bash
BASEDIR=~/temp/log
IFS=$'\n'
for match in `ssh user#ip grep -r Fail "remote_folder/logfile*"`
do
IFS=: read file line <<< $match
mkdir -p `dirname $BASEDIR/$file`
echo $line >> $BASEDIR/$file
done
You might want to look at an explanation to IFS in combination with read.
ssh user#ip grep Fail remote_folder/logfile*
I am trying to ls the directories and print them out but nothing is being displayed. I am able to SSH and execute the first pwd. However, anything within the the for loop has no output. I know for sure there are directories called event-test- because I've done so manually. I've manually entered the directory (/data/kafka/tmp/kafka-logs/) and ran this piece of code and the correct output appeared so I'm not sure why
manually entered has correct output:
for i in `ls | grep "event-test"`; do echo $i; done;
script:
for h in ${hosts[*]}; do
ssh -i trinity-prod-keypair.pem bc2-user#$h << EOF
sudo bash
cd /data/kafka/tmp/kafka-logs/
pwd
for i in `ls | grep "event-test-"`; do
pwd
echo $i;
done;
exit;
exit;
EOF
done
It is because
`ls | grep "event-test-"`
is executing on your localhost not on remote host. Besides parsing ls is error prone and not even needed. You can do:
for h in "${hosts[#]}"; do
ssh -t -t trinity-prod-keypair.pem bc2-user#$h <<'EOF'
sudo bash
cd /data/kafka/tmp/kafka-logs/
pwd
for i in *event-test-*; do
pwd
echo "$i"
done
exit
exit
EOF
done
When parsing ls it is good practice to do ls -1 to get a prettier list to parse. Additionally, when trying to find files named "event-test-" I would recommend the find command. Since I am not completely sure what you are attempting to do other than list the locations of these "event-test" files I'd recommend something more similar to the following:
for h in "${hosts[#]}"; do ssh trinity-prod-keypair.pem bc2-user#$h -t -t "find /data/kafka/tmp/kafka-logs/ -type f -name *event-test-*;" ; done
This will give you a pretty output of the full path to the file and the file name.
I hope this helps.
We have a repository of about 3000 MP3 files. Many of these files have our old domain name within their name.
For example: somesong_oldDomainName.mp3
I need to SSH into the site, find all instances of those files and rename them with the new domain name.
Example: somesong_NEWDomainName.mp3
I know the basic SSH commands but not something advanced like this.
Pretty sure it'll be a combination of multiple commands.
Assuming you get an interactive shell when you ssh into your linux server, this might be a possible way:
ssh user#machine-name-or-ip
then you will get some sort of terminal like
user#machine-name:~$
where you enter the commands to execute on that remote machine.
As mentioned in the comments, the answer here might just fit very well:
Bash: Rename small part of multiple files in middle of name
user#machine-name:~$ for i in *.mp3; do mv "$i" "$(echo "$i" | sed 's/_oldDomainName/_NEWDomainName/g')"; done
This assumes, your current directory is the one with all the MP3 files in it.
If you dont want interactivly operate on your files, e.g. because they change very often and you want a script to perform this action, SSH can also execute a command and/or shell script remotely.
To pass the command directly with the SSH call:
SSH error when executing a remote command: "stdin: is not a tty"
To pipe a local shell script into the SSH connection: How to use SSH to run a shell script on a remote machine?
Run a remote shell script via SSH: how to run a script file remotely using ssh
Edit:
Assume you are connected via SSH to your remote machine and have somewhat similar versions of bash and sed, it should work like this:
$ ls
bar_chosefil.mp3 boo_chosefil.mp3 foo_chosefil.mp3
$ for i in *.mp3; do mv $i $(echo $i | sed 's/chosefil/tamasha/g'); done
$ ls
bar_tamasha.mp3 boo_tamasha.mp3 foo_tamasha.mp3
Versions involved:
bash: 4.2.25
sed: 4.2.1
mv: 8.13
Edit 2:
Updated the command to work with blanks in filenames
$ ls
asd chosefil.mp3 bar_chosefil.mp3 boo_chosefil.mp3 foo_chosefil.mp3
$ for i in *.mp3; do mv "$i" "$(echo "$i" | sed 's/chosefil/tamasha/g')"; done
$ ls
asd tamasha.mp3 bar_tamasha.mp3 boo_tamasha.mp3 foo_tamasha.mp3