Server to server file transfer port? - linux

I use a bash script(https://github.com/johnnywoof/FTP-Bash-Backup) to Backup my Debian www directory to a ftp server. everything went well but transfer blocked by firewall. The script using separate port every time so i can't unblock by Firewall. is there any way to specify a port ?
The Bash script
# FTP server settings
USERNAME=""
PASSWORD=""
SERVER=""
PORT=21
BACKUPDIR="/"
ndays=7
LOCAL_DIRECTORY="/home"
TEMP_BACKUP_STORE="/tmp"
ENCRYPT_BACKUP=false
AES_PASSWORD_FILE=""
timestamp=$(date --iso)
backup_remote_file_name="$timestamp.tar.gz"
backup_file="$TEMP_BACKUP_STORE/$backup_remote_file_name"
MM=`date --date="$ndays days ago" +%b`
DD=`date --date="$ndays days ago" +%d`
echo "Removing files older than $MM $DD"
listing=`ftp -i -n $SERVER $PORT <<EOMYF
user $USERNAME $PASSWORD
binary
cd $BACKUPDIR
ls
quit
EOMYF`
lista=( $listing )
for ((FNO=0; FNO<${#lista[#]}; FNO+=9));do
# month (element 5), day (element 6) and filename (element 8)
#echo Date ${lista[`expr $FNO+5`]} ${lista[`expr $FNO+6`]} File: ${lista[`expr $FNO+8`]}
if [ ${lista[`expr $FNO+5`]}=$MM ];
then
if [[ ${lista[`expr $FNO+6`]} -lt $DD ]];
then
echo "Removing ${lista[`expr $FNO+8`]}"
ftp -i -n $SERVER $PORT <<EOMYF2
user $USERNAME $PASSWORD
binary
cd $BACKUPDIR
delete ${lista[`expr $FNO+8`]}
quit
EOMYF2
fi
fi
done
echo "Creating backup..."
tar -czf $backup_file $LOCAL_DIRECTORY
if [ "$ENCRYPT_BACKUP" == "true" ]
then
echo "Encrypting backup using OpenSSL..."
output_encrypted_file="$backup_file.enc"
openssl enc -aes-256-cbc -salt -in $backup_file -out $output_encrypted_file -pass file:$AES_PASSWORD_FILE
rm $backup_file
backup_file=$output_encrypted_file
fi
echo "Uploading backup $backup_file ..."
ftp -n -i $SERVER $PORT <<EOF
user $USERNAME $PASSWORD
cd $BACKUPDIR
put $backup_file $backup_remote_file_name
quit
EOF
echo "Deleting temporary files..."
rm $backup_file
echo "Backup complete."

Switch to passive mode:
Insert passive in a new line after user command.
See: Active FTP vs. Passive FTP, a Definitive Explanation

Related

systemd service works different then running the .sh file from user shell trying run a VNC startup script with noVNC

I want to run a .sh file with a systemd service. Let's explain it a little bit. If I go to cd /home/ubuntu I can run ./vnc_startup.sh. This file creates a VNC connection and starts noVNC. Then I go to the browser open the address and login. I can run every command like as example rosrun rviz rviz because I have installed ROS.
If I use this service, it will not work:
cat /etc/systemd/system/novnc.service
[Unit]
After=NetworkManager.service time-sync.target
[Service]
Type=forking
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu
TimeoutStartSec=infinity
TimeoutStopSec=infinity
ExecStartPre=/bin/rm -f /home/ubuntu/no_vnc_startup.log
ExecStartPre=/bin/rm -f /home/ubuntu/vnc_startup.log
ExecStartPre=/bin/rm -f /home/ubuntu/wm.log
ExecStartPre=/bin/rm -f /home/ubuntu/wm_startup.log
ExecStart=/bin/bash -c "source /etc/environment; /home/ubuntu/vnc_startup.sh"
ExecStopPost=/bin/rm -f /home/ubuntu/no_vnc_startup.log
ExecStopPost=/bin/rm -f /home/ubuntu/vnc_startup.log
ExecStopPost=/bin/rm -f /home/ubuntu/wm.log
ExecStopPost=/bin/rm -f /home/ubuntu/wm_startup.log
[Install]
WantedBy=multi-user.target
I also tried it with systemctl --user start novnc.service and put the file inside /usr/lib/systemd/user instead of sudo systemctl start novnc.service and /etc/systemd/system/novnc.service.
Following workaround will work in the noVNC environment: I can open a Terminal. I can see ubuntu#hostname:~$. So it seems to be the right user and I am in the right working directory. Before running as example rosrun rviz rviz I have to run sudo su ubuntu. And then it works. If I had run ./vnc_startup.sh instead of running this script with systemd it works directly without the workaround with sudo su ubuntu.
Hard to explain. I hope you can understand me.
systemctl --user show-environment
HOME=/home/ubuntu
LANG=de_DE
LOGNAME=ubuntu
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
SHELL=/bin/bash
USER=ubuntu
XDG_RUNTIME_DIR=/run/user/1001
The command printenv makes clear that maybe the DISPLAY=:0 is missing, so I tried it with adding
export DISPLAY=:0
xset q
And I also added Environment=XAUTHORITY=/home/ubuntu/.Xauthority:
[Unit]
After=NetworkManager.service time-sync.target
[Service]
Type=forking
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu
Environment=XAUTHORITY=/home/ubuntu/.Xauthority
TimeoutStartSec=infinity
TimeoutStopSec=infinity
ExecStartPre=/bin/rm -f /home/ubuntu/no_vnc_startup.log
ExecStartPre=/bin/rm -f /home/ubuntu/vnc_startup.log
ExecStartPre=/bin/rm -f /home/ubuntu/wm.log
ExecStartPre=/bin/rm -f /home/ubuntu/wm_startup.log
ExecStart=/bin/bash -c "source /etc/environment; export DISPLAY=:0; xset q; /home/ubuntu/vnc_startup.sh"
ExecStopPost=/bin/rm -f /home/ubuntu/no_vnc_startup.log
ExecStopPost=/bin/rm -f /home/ubuntu/vnc_startup.log
ExecStopPost=/bin/rm -f /home/ubuntu/wm.log
ExecStopPost=/bin/rm -f /home/ubuntu/wm_startup.log
[Install]
WantedBy=multi-user.target
Here my log files:
cat no_vnc_startup.log
New 'shlServer01:1 (ubuntu)' desktop is shlServer01:1
Starting applications specified in /home/ubuntu/.vnc/xstartup
Log file is /home/ubuntu/.vnc/shlServer01:1.log
r settings:
- Listen on :6901
- Flash security policy server
- Web server. Web root: /home/ubuntu/noVNC
- No SSL/TLS support (no cert file)
- proxying from :6901 to localhost:5901
Navigate to this URL:
http://shlServer01:6901/vnc.html?host=shlServer01&port=6901
Press Ctrl-C to exit
192.168.0.6 - - [15/Dec/2021 15:16:56] 192.168.0.6: Plain non-SSL (ws://) WebSocket connection
192.168.0.6 - - [15/Dec/2021 15:16:56] 192.168.0.6: Version hybi-13, base64: 'False'
192.168.0.6 - - [15/Dec/2021 15:16:56] 192.168.0.6: Path: '/websockify'
192.168.0.6 - - [15/Dec/2021 15:16:56] connecting to: localhost:5901
cat vnc_startup.log
Killing Xvnc process ID 63164
Xvnc process ID 63164 already killed
cat wm.log
/usr/bin/startxfce4: X server already running on display :0
xfce4-session: Cannot open display: .
▒xfce4-session --help▒ eingeben, um mehr ▒ber die Verwendung zu erfahren.
cat wm_startup.log
------------------ startup of Xfce4 window manager ------------------
No protocol specified
xset: unable to open display ":0"
No protocol specified
xset: unable to open display ":0"
No protocol specified
xset: unable to open display ":0"
No protocol specified
xrdb: Resource temporarily unavailable
xrdb: Can't open display ':0'
No protocol specified
No protocol specified
/usr/bin/startxfce4: X server already running on display :0
xfce4-session: Cannot open display: .
▒xfce4-session --help▒ eingeben, um mehr ▒ber die Verwendung zu erfahren.
What surprises me is that this is not the error. The error messages also come when I start the script from the terminal and then it works.
The vnc_startup.sh looks like following:
#!/bin/bash
### every exit != 0 fails the script
set -e
## print out help
help (){
echo "
OPTIONS:
-w, --wait (default) keeps the UI and the vncserver up until SIGINT or SIGTERM will received
-s, --skip skip the vnc startup and just execute the assigned command.
example: docker run consol/centos-xfce-vnc --skip bash
-d, --debug enables more detailed startup output
e.g. 'docker run consol/centos-xfce-vnc --debug bash'
-h, --help print out this help
Fore more information see: https://github.com/ConSol/docker-headless-vnc-container
"
}
if [[ $1 =~ -h|--help ]]; then
help
exit 0
fi
# should also source /home/ubuntu/generate_container_user
source /home/ubuntu/.bashrc
# add `--skip` to startup args, to skip the VNC startup procedure
if [[ $1 =~ -s|--skip ]]; then
echo -e "\n\n------------------ SKIP VNC STARTUP -----------------"
echo -e "\n\n------------------ EXECUTE COMMAND ------------------"
echo "Executing command: '${#:2}'"
exec "${#:2}"
fi
if [[ $1 =~ -d|--debug ]]; then
echo -e "\n\n------------------ DEBUG VNC STARTUP -----------------"
export DEBUG=true
fi
## correct forwarding of shutdown signal
cleanup () {
kill -s SIGTERM $!
exit 0
}
trap cleanup SIGINT SIGTERM
## write correct window size to chrome properties
/home/ubuntu/chrome-init.sh
## resolve_vnc_connection
VNC_IP=$(hostname -i)
## change vnc password
echo -e "\n------------------ change VNC password ------------------"
# first entry is control, second is view (if only one is valid for both)
mkdir -p "/home/ubuntu/.vnc"
PASSWD_PATH="/home/ubuntu/.vnc/passwd"
if [[ -f $PASSWD_PATH ]]; then
echo -e "\n--------- purging existing VNC password settings ---------"
rm -f $PASSWD_PATH
fi
if [[ $VNC_VIEW_ONLY == "true" ]]; then
echo "start VNC server in VIEW ONLY mode!"
#create random pw to prevent access
echo $(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 20) | vncpasswd -f > $PASSWD_PATH
fi
echo "ubuntu" | vncpasswd -f >> $PASSWD_PATH
chmod 600 $PASSWD_PATH
## start vncserver and noVNC webclient
echo -e "\n------------------ start noVNC ----------------------------"
if [[ $DEBUG == true ]]; then echo "/home/ubuntu/noVNC/utils/launch.sh --vnc localhost:5901 --listen 6901"; fi
/home/ubuntu/noVNC/utils/launch.sh --vnc localhost:5901 --listen 6901 &> /home/ubuntu/no_vnc_startup.log &
PID_SUB=$!
echo -e "\n------------------ start VNC server ------------------------"
echo "remove old vnc locks to be a reattachable container"
vncserver -kill :1 &> /home/ubuntu/vnc_startup.log \
|| rm -rfv /tmp/.X*-lock /tmp/.X11-unix &> /home/ubuntu/vnc_startup.log \
|| echo "no locks present"
echo -e "start vncserver with param: VNC_COL_DEPTH=24, VNC_RESOLUTION=1280x1024\n..."
if [[ $DEBUG == true ]]; then echo "vncserver :1 -depth 24 -geometry 1280x1024"; fi
vncserver :1 -depth 24 -geometry 1280x1024 &> /home/ubuntu/no_vnc_startup.log
echo -e "start window manager\n..."
/home/ubuntu/wm_startup.sh &> /home/ubuntu/wm_startup.log
## log connect options
echo -e "\n\n------------------ VNC environment started ------------------"
echo -e "\nVNCSERVER started on DISPLAY= :1 \n\t=> connect via VNC viewer with $VNC_IP:5901"
echo -e "\nnoVNC HTML client started:\n\t=> connect via http://$VNC_IP:6901/?password=...\n"
if [[ $DEBUG == true ]] || [[ $1 =~ -t|--tail-log ]]; then
echo -e "\n------------------ /home/ubuntu/.vnc/*:1.log ------------------"
# if option `-t` or `--tail-log` block the execution and tail the VNC log
tail -f /home/ubuntu/*.log /home/ubuntu/.vnc/*:1.log
fi
if [ -z "$1" ] || [[ $1 =~ -w|--wait ]]; then
wait $PID_SUB
else
# unknown option ==> call command
echo -e "\n\n------------------ EXECUTE COMMAND ------------------"
echo "Executing command: '$#'"
exec "$#"
fi
The wm_startup.sh looks like this:
#!/usr/bin/env bash
### every exit != 0 fails the script
set -e
echo -e "\n------------------ startup of Xfce4 window manager ------------------"
### disable screensaver and power management
xset -dpms &
xset s noblank &
xset s off &
/usr/bin/startxfce4 --replace > /home/ubuntu/wm.log &
sleep 1
cat /home/ubuntu/wm.log
And it should not be important but the launch.sh file fom noVNC looks like this:
#!/usr/bin/env bash
# Copyright 2016 Joel Martin
# Copyright 2016 Solly Ross
# Licensed under MPL 2.0 or any later version (see LICENSE.txt)
usage() {
if [ "$*" ]; then
echo "$*"
echo
fi
echo "Usage: ${NAME} [--listen PORT] [--vnc VNC_HOST:PORT] [--cert CERT] [--ssl-only]"
echo
echo "Starts the WebSockets proxy and a mini-webserver and "
echo "provides a cut-and-paste URL to go to."
echo
echo " --listen PORT Port for proxy/webserver to listen on"
echo " Default: 6080"
echo " --vnc VNC_HOST:PORT VNC server host:port proxy target"
echo " Default: localhost:5900"
echo " --cert CERT Path to combined cert/key file"
echo " Default: self.pem"
echo " --web WEB Path to web files (e.g. vnc.html)"
echo " Default: ./"
echo " --ssl-only Disable non-https connections."
echo " "
exit 2
}
NAME="$(basename $0)"
REAL_NAME="$(readlink -f $0)"
HERE="$(cd "$(dirname "$REAL_NAME")" && pwd)"
PORT="6080"
VNC_DEST="localhost:5900"
CERT=""
WEB=""
proxy_pid=""
SSLONLY=""
die() {
echo "$*"
exit 1
}
cleanup() {
trap - TERM QUIT INT EXIT
trap "true" CHLD # Ignore cleanup messages
echo
if [ -n "${proxy_pid}" ]; then
echo "Terminating WebSockets proxy (${proxy_pid})"
kill ${proxy_pid}
fi
}
# Process Arguments
# Arguments that only apply to chrooter itself
while [ "$*" ]; do
param=$1; shift; OPTARG=$1
case $param in
--listen) PORT="${OPTARG}"; shift ;;
--vnc) VNC_DEST="${OPTARG}"; shift ;;
--cert) CERT="${OPTARG}"; shift ;;
--web) WEB="${OPTARG}"; shift ;;
--ssl-only) SSLONLY="--ssl-only" ;;
-h|--help) usage ;;
-*) usage "Unknown chrooter option: ${param}" ;;
*) break ;;
esac
done
# Sanity checks
which netstat >/dev/null 2>&1 \
|| die "Must have netstat installed"
netstat -ltn | grep -qs ":${PORT} .*LISTEN" \
&& die "Port ${PORT} in use. Try --listen PORT"
trap "cleanup" TERM QUIT INT EXIT
# Find vnc.html
if [ -n "${WEB}" ]; then
if [ ! -e "${WEB}/vnc.html" ]; then
die "Could not find ${WEB}/vnc.html"
fi
elif [ -e "$(pwd)/vnc.html" ]; then
WEB=$(pwd)
elif [ -e "${HERE}/../vnc.html" ]; then
WEB=${HERE}/../
elif [ -e "${HERE}/vnc.html" ]; then
WEB=${HERE}
elif [ -e "${HERE}/../share/novnc/vnc.html" ]; then
WEB=${HERE}/../share/novnc/
else
die "Could not find vnc.html"
fi
# Find self.pem
if [ -n "${CERT}" ]; then
if [ ! -e "${CERT}" ]; then
die "Could not find ${CERT}"
fi
elif [ -e "$(pwd)/self.pem" ]; then
CERT="$(pwd)/self.pem"
elif [ -e "${HERE}/../self.pem" ]; then
CERT="${HERE}/../self.pem"
elif [ -e "${HERE}/self.pem" ]; then
CERT="${HERE}/self.pem"
else
echo "Warning: could not find self.pem"
fi
# try to find websockify (prefer local, try global, then download local)
if [[ -e ${HERE}/websockify ]]; then
WEBSOCKIFY=${HERE}/websockify/run
if [[ ! -x $WEBSOCKIFY ]]; then
echo "The path ${HERE}/websockify exists, but $WEBSOCKIFY either does not exist or is not executable."
echo "If you intended to use an installed websockify package, please remove ${HERE}/websockify."
exit 1
fi
echo "Using local websockify at $WEBSOCKIFY"
else
WEBSOCKIFY=$(which websockify 2>/dev/null)
if [[ $? -ne 0 ]]; then
echo "No installed websockify, attempting to clone websockify..."
WEBSOCKIFY=${HERE}/websockify/run
git clone https://github.com/novnc/websockify ${HERE}/websockify
if [[ ! -e $WEBSOCKIFY ]]; then
echo "Unable to locate ${HERE}/websockify/run after downloading"
exit 1
fi
echo "Using local websockify at $WEBSOCKIFY"
else
echo "Using installed websockify at $WEBSOCKIFY"
fi
fi
echo "Starting webserver and WebSockets proxy on port ${PORT}"
#${HERE}/websockify --web ${WEB} ${CERT:+--cert ${CERT}} ${PORT} ${VNC_DEST} &
${WEBSOCKIFY} ${SSLONLY} --web ${WEB} ${CERT:+--cert ${CERT}} ${PORT} ${VNC_DEST} &
proxy_pid="$!"
sleep 1
if ! ps -p ${proxy_pid} >/dev/null; then
proxy_pid=
echo "Failed to start WebSockets proxy"
exit 1
fi
echo -e "\n\nNavigate to this URL:\n"
if [ "x$SSLONLY" == "x" ]; then
echo -e " http://$(hostname):${PORT}/vnc.html?host=$(hostname)&port=${PORT}\n"
else
echo -e " https://$(hostname):${PORT}/vnc.html?host=$(hostname)&port=${PORT}\n"
fi
echo -e "Press Ctrl-C to exit\n\n"
wait ${proxy_pid}
Suggesting to simply your /etc/systemd/system/novnc.service service unit with a single script for ExecStart command and a single script for ExecStop
/etc/systemd/system/novnc.service
[Unit]
After=NetworkManager.service time-sync.target
[Service]
Type=forking
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu
Environment=XAUTHORITY=/home/ubuntu/.Xauthority
TimeoutStartSec=infinity
TimeoutStopSec=infinity
ExecStart=/bin/bash -c "/home/ubuntu/servic_vnc_startup.sh"
ExecStop=/bin/bash -c "/home/ubuntu/servic_vnc_shutdown.sh"
[Install]
WantedBy=multi-user.target
/home/ubuntu/servic_vnc_startup.sh
#!\bin\bash
source /home/ubuntu/.bash_profile
source /etc/environment
export DISPLAY=:0
xset q;
rm -f /home/ubuntu/{no_vnc_startup.log,vnc_startup.log,wm.log,wm_startup.log}
/home/ubuntu/vnc_startup.sh
/home/ubuntu/servic_vnc_shutdown.sh
#!\bin\bash
source /home/ubuntu/.bash_profile
source /etc/environment
export DISPLAY=:0
xset q;
pkill -9 -f "/home/ubuntu/vnc_startup.sh"
rm -f /home/ubuntu/{no_vnc_startup.log,vnc_startup.log,wm.log,wm_startup.log}
Debugging
Login as user ubuntu.
Run /home/ubuntu/servic_vnc_startup.sh from command line.
If fails, fix it till it is successful.
Then try running /home/ubuntu/servic_vnc_startup.sh as user noboby:
sudo -u nobody "/home/ubuntu/servic_vnc_startup.sh"
User nobody has no shell and no environment context, as is the systemd service /etc/systemd/system/novnc.service.
If user nobody can run /home/ubuntu/servic_vnc_startup.sh then the /etc/systemd/system/novnc.service service unit can do as well.
Do same testing pattern with /home/ubuntu/servic_vnc_shutdown.sh
Lessons to learn:
Simplify service unit as much as possible.
Pull all scripting to a single shell script. Debug single shell script.
Avoid debugging handling service unit once deployed. Instead debug and modify called script.
Use user nobody user, to debug script to run without environment context and without shell.

Calling multiple shell scripts within a script on different virtual machines

I am trying to create shell scripts which will setup Zookeeper Server in one VM, and its corresponding Zookeeper Clients in different VM's so i written a shell script as below
#!/bin/bash
ZOOKEEPER_SERVER_IP="1.2.3.4"
while read ipaddress zookeepertype number
do
echo -e "Setting up the Zookeepers \n"
echo $ipaddress
if [ "${zookeepertype}" = 'zookeeperserver' ]; then
echo "Setup Zookeeper Server"
#ZOOKEEPER_SERVER_IP = $ipaddress
#echo $ZOOKEEPER_SERVER_IP
#echo $ipaddress
sudo scp -i /home/ubuntu/.ssh/fd -r /home/ubuntu/ZooKeeper_Server_Script.sh ubuntu#$ipaddress:/home/ubuntu/
ssh -i /home/ubuntu/.ssh/fd ubuntu#$ipaddress /home/ubuntu/ZooKeeper_Server_Script.sh
echo "This script is about to run ZooKeeper_Server_Script."
echo "The server script has completed.";
#sleep 30
exit 1
fi
echo -e $ZOOKEEPER_SERVER_IP
if [ $zookeepertype = "zookeeperclient" ] ; then
echo "Setup Zookeeper Client"
echo $ipaddress
sudo scp -i /home/ubuntu/.ssh/fd -r /home/ubuntu/ZooKeeper_Client_Script.sh ubuntu#$ipaddress:/home/ubuntu/
ssh -i /home/ubuntu/.ssh/fd ubuntu#$ipaddress
#mkdir /home/ubuntu/keyfiles
#exit
#sudo scp -i /home/ubuntu/.ssh/fd -r /home/ubuntu/abc/network/test/keyfiles/* ubuntu#$ipaddress:/home/ubuntu/keyfiles
#sudo scp -i /home/ubuntu/.ssh/fd -r /home/ubuntu/abc/test/simple/abc.json ubuntu#$ipaddress:/home/ubuntu/
#ssh -i /home/ubuntu/.ssh/fd ubuntu#$ipaddress
#chmod 777 ZooKeeper_Client_Script.sh
#echo "This script is about to run ZooKeeper_Client_Script."
#sh ./ZooKeeper_Client_Script.sh $ZOOKEEPER_SERVER_IP
echo "The client script has completed."
#exit
fi
#Separating Runhosts File
done < setupZkinput.txt
the input file is
1.2.3.4 zookeeperserver 1
5.6.7.8 zookeeperclient 2
9.10.11.12 zookeeperclient 3
The issue that i am facing is
1) Only the server setup is being done , i.e the script is exiting after the first line
2)Not able to assign the server ip dynamically , in the line ZOOKEEPER_SERVER_IP = $ipaddress
Thanks
What is the default permission you are setting up after copying a file to the server?
remember it required execute permission in order to execute script.

Newly created file becomes 0 kb (data gets overwritten to nothing) on reboot in Linux

I'm having a strange problem that's driving me crazy!
The task in hand is to start one set of files during the first login of "root" user and another set of files during the second login of the same user. I decided to use the ".profile" and ".bashrc" files and to reload the ".bashrc" file towards the end of the task happening during the first login.
During the first login, I create a private key and certificate signing request, and call an API to get the certificate. I store this certificate and private key in a file location and then modify the ".bashrc" to invoke the second set of files, which make use of this certificate and key to authenticate an application to run.
The problem is that the certificate and key are overwritten and become null randomly after the first boot. I've attached the code below for your review.
FIRST SET OF FILES
".profile" script
# .bash_profile
umask 022
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
".bashrc" script
/myFolder/backgroundTask1.sh &
/myFolder/certificateGenerator.sh
backgroundTask1.sh script
pipe=/myFolder/testpipe
if [[ ! -p $pipe ]]; then
mkfifo $pipe
fi
while true
do
## Do some status LED blinking task here
done &
while true
do
if read line < $pipe; then
if [[ "$line" == 'success' ]]; then
## Kill the background LED blinking task created in the above while loop
kill $!
rm $pipe
exit
elif [[ "$line" == 'failed' ]]; then
kill $!
rm $pipe
exit
fi
fi
done
certificateGenerator.sh script
PLEASE NOTE THE LAST FEW LINES WHERE I MODIFY THE BASHRC SCRIPT
Please also note the files /anotherFolder/myKey.key and /anotherFolder/myCert.crt
#!/bin/bash
## Named pipe location for communicating to backgroundTask1
pipe=/myFolder/testpipe
openssl req -new -newkey rsa:2048 -nodes -out certificateSigningRequest.csr -keyout /anotherFolder/myKey.key -subj "/C=myCountry/ST=myState/L=myCity/O=myCompany/OU=myOU/CN=myDevice"
cert_req=$(<$certificateSigningRequest.csr)
## Get AD token from Azure for talking to my custom API hosted on Azure
response=$(curl -o - -s -w "%{http_code}\n" -X POST \
https://login.microsoftonline.com/myCompany.onmicrosoft.com/oauth2/token \
-H 'content-type: application/x-www-form-urlencoded' \
-d 'grant_type=client_credentials&resource=https%3A%2F%2Fmanagement.core.windows.net%2F&client_id=myClientID&client_secret=mySecret')
if [ $?==0 ]; then
status=$(echo $response | tail -c 4)
body=${response::-3}
token=$(echo $body | jq -r '.access_token')
fi
## Send CSR to my custom API to get certificate
response=$(jq -n --arg csr "$cert_req" \
'{
cert: {
csr: $csr
}
}' |
curl -o - -s -w "%{http_code}\n" -X POST \
https://myCustomAPI.azurewebsites.net/api/v1/customEndpoint \
-H "authorization: Bearer $token" \
-H "content-type: application/json" \
-d #-
)
## Parse the response to find out if the request succeeded
if [ $?==0 ]; then
destCertDir=/anotherFolder/myCert.crt
status=$(echo $response | tail -c 4)
body=${response::-3}
cert=$(echo $body | jq -r '.certificate')
if [ "$status" == "$http_success" ]; then
echo "$cert" > "$destCertDir"
## Change .bashrc for next boot
echo '/myFolder/backgroundTask2.sh &' > ~/.bashrc
echo '/myFolder/applicationAuthenticator.sh' >> ~/.bashrc
echo "success" > $pipe
exit
fi
fi
SECOND SET OF FILES
".profile" script
# .bash_profile
umask 022
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
".bashrc" script
/myFolder/backgroundTask2.sh &
/myFolder/applicationAuthenticator.sh
backgroundTask2.sh script
pipe=/myFolder/testpipe2
if [[ ! -p $pipe ]]; then
mkfifo $pipe
fi
while true
do
## Do some status LED blinking task here
done &
while true
do
if read line < $pipe; then
if [[ "$line" == 'success' ]]; then
## Kill the background LED blinking task created in the above while loop
kill $!
rm $pipe
exit
elif [[ "$line" == 'failed' ]]; then
kill $!
rm $pipe
exit
fi
fi
done
applicationAuthenticator.sh script
PLEASE NOTE HOW I MODIFY BASHRC TO STARTUP NORMAL FROM NEXT REBOOT TOWARDS THE END OF THIS SCRIPT
#!/bin/bash
## Named pipe location for communicating to backgroundTask2
pipe=/myFolder/testpipe2
response=$(curl https://myProduct/myCustomAPI.com \
--cert /anotherFoler/myCert.crt --key /anotherFolder/myKey.key \
-H 'Content-Type: application/x-www-form-urlencoded; charset=utf-8' \
-d 'data=xxx')
if [[ $response == 204 ]; then
echo '' > ~/.bashrc
echo "success" > $pipe
exit
else
echo "failed" > $pipe
exit
fi
Problem
Even thought the first set of files create the key and certificate, they are overwritten to NULL after the first reboot.
To make sure that they exist before reboot, I go to the location "/anotherFolder" and check the files physically. They have the full key and certificate before reboot. When I reboot and see that the script fails, the same key and certificate files (which had actual data before reboot) now have NULL values.

Linux/sh: How to list files one by one, compress each (by p7zip without save file on disk) and upload to ftp server (by curl/ncftp)?

Linux/sh: How to list all files one by one in specific folder,
compress each (by p7zip without save file on disk) and
upload to ftp server (by curl/ncftp) with same folder structure?
This script below work perfect but
I don't want to save 7z file on a disk each time. Because I always need to delete them all after uploaded.
I prefer stio from 7zip to curl, how to do that?
#!/bin/sh
FOLDER="/volume3/backup_3/kopia_nas/tmp"
BACKUP_DIR="/volume3/backup_3/kopia_nas/tmp2"
FTP_HOST=""
FTP_USER=""
FTP_PASS=""
FTP_PORT="21"
PASSWORD="abc123"
FTP_FOLDER="/backup2"
#####################################################################
echo "[$(date +'%d-%m-%Y %H:%M:%S')] starting..."
echo ""
/usr/bin/find "${FOLDER}" -type f | while read line; do
# echo "$line" #path+file
# echo "${line##*/}" #file
# echo "${line%/*}" #path
#
/usr/bin/p7zip/7za a "${BACKUP_DIR}${line}.7z" "${line}" -t7z -ms=off -m0=Copy -mhe -mmt -mx0 -p"${PASSWORD}"
curl -s --disable-epsv -v -T "${BACKUP_DIR}${line}.7z" -u "${FTP_USER}:${FTP_PASS}" "ftp://${FTP_HOST}/${FTP_FOLDER}${line%/*}/" --ftp-create-dirs;
#-S -show errors
#-s -silent mode
#-an - no file name
#v- verbose
#/usr/bin/ncftp/ncftpput -m -u -c "${FTP_USER}" -p "${FTP_PASS}" -P "${FTP_PORT}" "${FTP_HOST}" "${FTP_FOLDER}${line%/*}/" "${line##*/}.7z"
# if [ $? -ne 0 ]; then echo "[$(date +'%d-%m-%Y %H:%M:%S')] Upload failed"; fi
done
#rm -rf "${BACKUP_DIR}/" #delete temporary folder
echo ""
echo "[$(date +'%d-%m-%Y %H:%M:%S')] completed..."
exit 0
I try this but it doesn't work for me...
/usr/bin/p7zip/7za a -an -t7z -ms=off -m0=Copy -mhe -mmt -mx0 -so -p"${PASSWORD}" | curl -S --disable-epsv -v -T - -u "${FTP_USER}:${FTP_PASS}" "ftp://${FTP_HOST}/${FTP_FOLDER}${line}/" --ftp-create-dirs;

Unexpected end of file in a shell script, cant find the mistake

HEy there i wrote this little shell script for my pi to upload a picture, but everytime i run the script i get "Unexpected end of file" I does not even show me the first echo.
Thanks for your help :)
raspistill -o snapshot2.jpg
HOST=XXXXX #This is the FTP servers host or IP address.
USER= XXXX #This is the FTP user that has access to the server.
PASS=XXXXX #This is the password for the FTP user.
NOW=$(date +"%c")
echo test
if [ -f work ];
then
echo >> ftp.log "$NOW Script failure"
echo ein prozess arbeitet noch
else
echo beginne upload
touch work
ftp -inv $HOST << EOF
user $USER $PASS
cd /bilder2/
put snapshot2.jpg
bye
echo >> ftp.log "$NOW Upload Success"
rm work
echo erfolgreicher upload
fi
EOF
fi should be placed after EOF, my guess would be that your script should look like:
raspistill -o snapshot2.jpg
HOST=XXXXX #This is the FTP servers host or IP address.
USER= XXXX #This is the FTP user that has access to the server.
PASS=XXXXX #This is the password for the FTP user.
NOW=$(date +"%c")
echo test
if [ -f work ];
then
echo >> ftp.log "$NOW Script failure"
echo ein prozess arbeitet noch
else
echo beginne upload
touch work
ftp -inv $HOST << EOF
user $USER $PASS
cd /bilder2/
put snapshot2.jpg
bye
EOF
echo >> ftp.log "$NOW Upload Success"
rm work
echo erfolgreicher upload
fi

Resources