I have been using VSCode on a Windows laptop for an ssh remote directory on a Linux machine. Things were working fine until an update of VSCode 1.68.0 and now I can't open the remote directory.
On Windows, the output says:
[12:55:24.183] Log Level: 2
[12:55:24.185] remote-ssh#0.82.1
[12:55:24.185] win32 x64
[12:55:24.226] SSH Resolver called for "ssh-remote+ubuntu", attempt 1
[12:55:24.227] "remote.SSH.useLocalServer": false
[12:55:24.227] "remote.SSH.showLoginTerminal": false
[12:55:24.227] "remote.SSH.remotePlatform": {"runon":"linux","pfuntner-kube":"linux","everyday":"linux","kube":"linux","ubuntu":"linux"}
[12:55:24.227] "remote.SSH.path": C:\cygwin64\bin\ssh.exe
[12:55:24.228] "remote.SSH.configFile": c:\users\jpfuntne\.ssh\config
[12:55:24.228] "remote.SSH.useFlock": true
[12:55:24.228] "remote.SSH.lockfilesInTmp": false
[12:55:24.228] "remote.SSH.localServerDownload": auto
[12:55:24.229] "remote.SSH.remoteServerListenOnSocket": false
[12:55:24.229] "remote.SSH.showLoginTerminal": false
[12:55:24.229] "remote.SSH.defaultExtensions": []
[12:55:24.229] "remote.SSH.loglevel": 2
[12:55:24.229] "remote.SSH.enableDynamicForwarding": true
[12:55:24.230] "remote.SSH.enableRemoteCommand": false
[12:55:24.230] "remote.SSH.serverPickPortsFromRange": {}
[12:55:24.230] "remote.SSH.serverInstallPath": {}
[12:55:24.239] SSH Resolver called for host: ubuntu
[12:55:24.239] Setting up SSH remote "ubuntu"
[12:55:24.247] Using commit id "4af164ea3a06f701fe3e89a2bcbb421d2026b68f" and quality "stable" for server
[12:55:24.254] Install and start server if needed
[12:55:24.257] Checking ssh with "C:\cygwin64\bin\ssh.exe -V"
[12:55:24.568] > OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021
[12:55:24.574] Using SSH config file "c:\users\jpfuntne\.ssh\config"
[12:55:24.575] Running script with connection command: "C:\cygwin64\bin\ssh.exe" -T -D 65341 -F "c:\users\jpfuntne\.ssh\config" ubuntu bash
[12:55:24.591] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[12:55:25.421] > WARNING!!!
> READ THIS BEFORE ATTEMPTING TO LOGON
>
> This System is for the use of authorized users only. Individuals
> using this computer without authority, or in excess of their
> authority, are subject to having all of their activities on this
> system monitored and recorded by system personnel. In the course
> of monitoring individuals improperly using this system, or in the
> course of system maintenance, the activities of authorized users
> may also be monitored. Anyone using this system expressly
> consents to such monitoring and is advised that if such
> monitoring reveals possible criminal activity, system personnel
> may provide the evidence of such monitoring to law enforcement
> officials.
>
> Cisco Acceptable Use Policy:
> http://wwwin.cisco.com/c/cec/organizations/security-trust/infosec/policies
> .html]0;C:\WINDOWS\System32\cmd.exe
[12:55:25.421] Got some output, clearing connection timeout
[12:55:26.367] > 28e76cc256c4: running
[12:55:26.405] > Acquiring lock on /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb42
> 1d2026b68f/vscode-remote-lock.ubuntu.4af164ea3a06f701fe3e89a2bcbb421d2026b68f
[12:55:26.430] > Installing to /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d20
> 26b68f...
> 28e76cc256c4%%1%%
> Downloading with wget
[12:55:33.759] > wget download failed
> failed: Connection timed out.
> printenv:
> SHELL=/bin/bash
> PWD=/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f
> LOGNAME=ubuntu
> XDG_SESSION_TYPE=tty
> MOTD_SHOWN=pam
> HOME=/home/ubuntu
> LANG=C.UTF-8
> VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
> SSH_CONNECTION=10.82.235.230 65347 64.102.189.72 22
> XDG_SESSION_CLASS=user
> USER=ubuntu
> SHLVL=0
> XDG_SESSION_ID=5
> XDG_RUNTIME_DIR=/run/user/1000
> SSH_CLIENT=10.82.235.230 65347 22
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
> :/usr/local/games:/snap/bin
> DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> OLDPWD=/home/ubuntu
> _=/usr/bin/printenv
[12:55:33.764] >
> Trigger local server download
> 28e76cc256c4:trigger_server_download
> platform==linux==
> vscodeArch==x64==
> destFolder==/home/ubuntu/.vscode-server/bin/==
> destFolder2==4af164ea3a06f701fe3e89a2bcbb421d2026b68f==
> 28e76cc256c4:trigger_server_download_end
> Waiting for client to transfer server archive...
> Waiting for /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026
> b68f/vscode-scp-done.flag and vscode-server.tar.gz to exist
[12:55:33.765] Got request to download on client for {"platform":"linux","arch":"x64","destFolder":"/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f"}
[12:55:33.765] Downloading VS Code server locally...
[12:55:33.770] >
>
[12:55:34.131] Resolver error: Error: Server returned 404
at v (vscode-file://vscode-app/c:/Users/jpfuntne/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js:622:38476)
at k.download (vscode-file://vscode-app/c:/Users/jpfuntne/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/workbench/workbench.desktop.main.js:623:1592)
[12:55:34.136] ------
On Linux, all I have is:
$ find .vscode-server -type f | xargs -r ls -ld
-rw-rw-r-- 1 ubuntu ubuntu 0 Jun 15 12:55 .vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f/vscode-remote-lock.ubuntu.4af164ea3a06f701fe3e89a2bcbb421d2026b68f
-rw-rw-r-- 1 ubuntu ubuntu 0 Jun 15 12:55 .vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f/vscode-server.tar.gz
$
When I was first having trouble, I did rm -rf .vscode to fix things but it didn't help.
Can anyone help?
I tried to start VSCode again today. At first I thought it had the same problem but it looks like things are working ok now and I got in!
[07:14:44.009] Log Level: 2
[07:14:44.010] remote-ssh#0.82.1
[07:14:44.010] win32 x64
[07:14:44.048] SSH Resolver called for "ssh-remote+ubuntu", attempt 1
[07:14:44.048] "remote.SSH.useLocalServer": false
[07:14:44.049] "remote.SSH.showLoginTerminal": false
[07:14:44.049] "remote.SSH.remotePlatform": {"runon":"linux","pfuntner-kube":"linux","everyday":"linux","kube":"linux","ubuntu":"linux"}
[07:14:44.049] "remote.SSH.path": C:\cygwin64\bin\ssh.exe
[07:14:44.049] "remote.SSH.configFile": c:\users\jpfuntne\.ssh\config
[07:14:44.049] "remote.SSH.useFlock": true
[07:14:44.049] "remote.SSH.lockfilesInTmp": false
[07:14:44.049] "remote.SSH.localServerDownload": auto
[07:14:44.050] "remote.SSH.remoteServerListenOnSocket": false
[07:14:44.050] "remote.SSH.showLoginTerminal": false
[07:14:44.050] "remote.SSH.defaultExtensions": []
[07:14:44.050] "remote.SSH.loglevel": 2
[07:14:44.050] "remote.SSH.enableDynamicForwarding": true
[07:14:44.050] "remote.SSH.enableRemoteCommand": false
[07:14:44.050] "remote.SSH.serverPickPortsFromRange": {}
[07:14:44.050] "remote.SSH.serverInstallPath": {}
[07:14:44.056] SSH Resolver called for host: ubuntu
[07:14:44.056] Setting up SSH remote "ubuntu"
[07:14:44.062] Using commit id "4af164ea3a06f701fe3e89a2bcbb421d2026b68f" and quality "stable" for server
[07:14:44.068] Install and start server if needed
[07:14:44.071] Checking ssh with "C:\cygwin64\bin\ssh.exe -V"
[07:14:44.354] > OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021
[07:14:44.386] Using SSH config file "c:\users\jpfuntne\.ssh\config"
[07:14:44.387] Running script with connection command: "C:\cygwin64\bin\ssh.exe" -T -D 63895 -F "c:\users\jpfuntne\.ssh\config" ubuntu bash
[07:14:44.400] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[07:14:46.375] > WARNING!!!
> READ THIS BEFORE ATTEMPTING TO LOGON
>
> This System is for the use of authorized users only. Individuals
> using this computer without authority, or in excess of their
> authority, are subject to having all of their activities on this
> system monitored and recorded by system personnel. In the course
> of monitoring individuals improperly using this system, or in the
> course of system maintenance, the activities of authorized users
> may also be monitored. Anyone using this system expressly
> consents to such monitoring and is advised that if such
> monitoring reveals possible criminal activity, system personnel
> may provide the evidence of such monitoring to law enforcement
> officials.
>
> Cisco Acceptable Use Policy:
> http://wwwin.cisco.com/c/cec/organizations/security-trust/infosec/policies
> .html]0;C:\WINDOWS\System32\cmd.exe
[07:14:46.375] Got some output, clearing connection timeout
[07:14:47.361] > cb747be286c8: running
> Acquiring lock on /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb42
> 1d2026b68f/vscode-remote-lock.ubuntu.4af164ea3a06f701fe3e89a2bcbb421d2026b68f
> Installing to /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d20
> 26b68f...
> cb747be286c8%%1%%
> Downloading with wget
[07:14:54.685] > wget download failed
> failed: Connection timed out.
> printenv:
> SHELL=/bin/bash
[07:14:54.692] >
> PWD=/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f
> LOGNAME=ubuntu
> XDG_SESSION_TYPE=tty
> MOTD_SHOWN=pam
> HOME=/home/ubuntu
> LANG=C.UTF-8
> VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
> SSH_CONNECTION=10.82.240.153 63898 64.102.189.72 22
> XDG_SESSION_CLASS=user
> USER=ubuntu
> SHLVL=0
> XDG_SESSION_ID=63
> XDG_RUNTIME_DIR=/run/user/1000
> SSH_CLIENT=10.82.240.153 63898 22
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
> :/usr/local/games:/snap/bin
> DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> OLDPWD=/home/ubuntu
> _=/usr/bin/printenv
[07:14:54.695] >
> Trigger local server download
> cb747be286c8:trigger_server_download
> platform==linux==
> vscodeArch==x64==
> destFolder==/home/ubuntu/.vscode-server/bin/==
> destFolder2==4af164ea3a06f701fe3e89a2bcbb421d2026b68f==
> cb747be286c8:trigger_server_download_end
> Waiting for client to transfer server archive...
> Waiting for /home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026
> b68f/vscode-scp-done.flag and vscode-server.tar.gz to exist
>
[07:14:54.697] Got request to download on client for {"platform":"linux","arch":"x64","destFolder":"/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f"}
[07:14:54.698] Downloading VS Code server locally...
[07:14:58.248] Downloaded VS Code server to c:\Users\jpfuntne\AppData\Local\Temp\03ba8324-01be-4f4c-bd89-538da94836ae
[07:14:58.253] Renamed VS Code server to c:\Users\jpfuntne\AppData\Local\Temp\vscode_server_1655378098249\vscode-server.tar.gz
[07:14:58.257] Checking ssh with "C:\cygwin64\bin\ssh.exe -V"
[07:14:58.555] > OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021
[07:14:58.581] Expecting scp to be next to ssh at C:\cygwin64\bin\scp.exe
[07:14:58.582] Testing scp with "C:\cygwin64\bin\scp.exe"
[07:14:59.442] scp exited with code: 1
[07:14:59.442] Got stderr from scp: usage: scp [-346ABCOpqRrsTv] [-c cipher] [-D sftp_server_path] [-F ssh_config]
[-i identity_file] [-J destination] [-l limit]
[-o ssh_option] [-P port] [-S program] source ... target
[07:14:59.443] Using SSH config file "c:\users\jpfuntne\.ssh\config"
[07:14:59.445] Copying file to remote with "C:\cygwin64\bin\scp.exe" -F "c:\users\jpfuntne\.ssh\config" "vscode-server.tar.gz" "vscode-scp-done.flag" ubuntu:"/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b68f"
[07:14:59.447] Using cwd: file:///c%3A/Users/jpfuntne/AppData/Local/Temp/vscode_server_1655378098249
[07:14:59.447] Terminal shell path: C:\WINDOWS\System32\cmd.exe
[07:15:00.744] > WARNING!!!
> READ THIS BEFORE ATTEMPTING TO LOGON
>
> This System is for the use of authorized users only. Individuals
> using this computer without authority, or in excess of their
> authority, are subject to having all of their activities on this
> system monitored and recorded by system personnel. In the course
> of monitoring individuals improperly using this system, or in the
> course of system maintenance, the activities of authorized users
> may also be monitored. Anyone using this system expressly
> consents to such monitoring and is advised that if such
> monitoring reveals possible criminal activity, system personnel
> may provide the evidence of such monitoring to law enforcement
> officials.
>
> Cisco Acceptable Use Policy:
> http://wwwin.cisco.com/c/cec/organizations/security-trust/infosec/policies
> .html]0;C:\WINDOWS\System32\cmd.exe
[07:15:01.620] > vscode-server.tar.gz 0% 0 0.0KB/s --:-- ETA
[07:15:02.630] > vscode-server.tar.gz 32% 17MB 17.0MB/s 00:02 ETA
[07:15:03.645] > vscode-server.tar.gz 66% 36MB 17.1MB/s 00:01 ETA
[07:15:04.691] > vscode-server.tar.gz 100% 54MB 17.4MB/s 00:03
[07:15:04.705] >
[07:15:04.722] > vscode-scp-done.flag 0% 0 0.0KB/s --:-- ETA
[07:15:04.736] > vscode-scp-done.flag 100% 9 0.7KB/s 00:00
[07:15:06.150] "Copy server to host" terminal command done
[07:15:06.702] > Found flag and server on host
> cb747be286c8%%2%%
> tar --version:
[07:15:06.717] >
> tar (GNU tar) 1.30
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Written by John Gilmore and Jay Fenlason.
[07:15:08.979] > Checking /home/ubuntu/.vscode-server/.4af164ea3a06f701fe3e89a2bcbb421d2026b68f.l
> og and /home/ubuntu/.vscode-server/.4af164ea3a06f701fe3e89a2bcbb421d2026b68f.pid
> for a running server
[07:15:09.019] > Starting server with command... /home/ubuntu/.vscode-server/bin/4af164ea3a06f701
> fe3e89a2bcbb421d2026b68f/bin/code-server --start-server --host=127.0.0.1 --accep
> t-server-license-terms --enable-remote-auto-shutdown --port=0 --telemetry-level
> all &> "/home/ubuntu/.vscode-server/.4af164ea3a06f701fe3e89a2bcbb421d2026b68f.
> log" < /dev/null
> printenv:
> SHELL=/bin/bash
> PWD=/home/ubuntu
> LOGNAME=ubuntu
> XDG_SESSION_TYPE=tty
> MOTD_SHOWN=pam
> HOME=/home/ubuntu
> LANG=C.UTF-8
> VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
> SSH_CONNECTION=10.82.240.153 63898 64.102.189.72 22
> XDG_SESSION_CLASS=user
> USER=ubuntu
> SHLVL=0
> XDG_SESSION_ID=63
> XDG_RUNTIME_DIR=/run/user/1000
> SSH_CLIENT=10.82.240.153 63898 22
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
> :/usr/local/games:/snap/bin
> DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> OLDPWD=/home/ubuntu/.vscode-server/bin/4af164ea3a06f701fe3e89a2bcbb421d2026b
> 68f
> _=/usr/bin/printenv
> rm: cannot remove '/home/ubuntu/.vscode-server/.4af164ea3a06f701fe3e89a2bcbb421d
> 2026b68f.token': No such file or directory
[07:15:09.029] >
> Spawned remote server: 60287
[07:15:09.036] >
> Waiting for server log...
[07:15:09.052] >
[07:15:09.080] > Waiting for server log...
[07:15:09.146] >
> *
> * Reminder: You may only use this software with Visual Studio family products,
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
>
[07:15:09.171] > cb747be286c8: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==35689==
> osReleaseId==ubuntu==
> arch==x86_64==
> tmpDir==/run/user/1000==
> platform==linux==
> unpackResult==success==
> didLocalDownload==1==
> downloadTime====
> installTime==2258==
> extInstallTime====
> serverStartTime==122==
> connectionToken==1a1a1a1a-1a1a-11a1-aa1a-1111a1111a1a==
[07:15:09.184] > cb747be286c8: end
[07:15:09.184] Received install output:
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==35689==
osReleaseId==ubuntu==
arch==x86_64==
tmpDir==/run/user/1000==
platform==linux==
unpackResult==success==
didLocalDownload==1==
downloadTime====
installTime==2258==
extInstallTime====
serverStartTime==122==
connectionToken==1a1a1a1a-1a1a-11a1-aa1a-1111a1111a1a==
[07:15:09.185] Remote server is listening on 35689
[07:15:09.185] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":35689},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/1000","platform":"linux","connectionToken":"1a1a1a1a-1a1a-11a1-aa1a-1111a1111a1a"},"installTime":2258,"serverStartTime":122,"didLocalDownload":true,"installUnpackCode":"success"}
[07:15:09.193] Starting forwarding server. localPort 63909 -> socksPort 63895 -> remotePort 35689
[07:15:09.195] Forwarding server listening on 63909
[07:15:09.195] Waiting for ssh tunnel to be ready
[07:15:09.257] [Forwarding server 63909] Got connection 0
[07:15:09.260] Tunneled 35689 to local port 63909
[07:15:09.260] Resolved "ssh-remote+ubuntu" to "127.0.0.1:63909"
[07:15:09.272] ------
[07:15:09.300] [Forwarding server 63909] Got connection 1
[07:15:09.650] [Forwarding server 63909] Got connection 2
On Linux:
$ find .vscode-server -type f | wc
1798 1798 196423
$
I know that to do port forwarding, the command is ssh -L. I also use other options to decorate it. So for example, a final full command may look like this ssh -fCNL *:10000:127.0.0.1:10001 127.0.0.1. And everything just works after entering password.
Then, because there is not only one port need to be forwarded, I decide to leave the job to shell script and use expect(tcl) to provide passwords(all the same).
Although without a deep understanding of expect, I managed to write the code with the help of Internet. The script succeeds spawning ssh and provides correct password. But I end up finding there is no such process when I try to check using ps -ef | grep ssh and netstat -anp | grep 10000.
I give -v option to ssh and the output seems to be fine.
So where is the problem? I have searched through Internet but most of questions are not about port forwarding. I'm not sure whether it is proper to use expect while I just want to let script automatically provide password.
Here the script.
#!/bin/sh
# Port Forwarding
# set -x
## function definition
connection ()
{
ps -ef | grep -v grep | grep ssh | grep $1 | grep $2 > /dev/null
if [ $? -eq 0 ] ; then
echo "forward $1 -> $2 done"
exit 0
fi
# ssh-keygen -f "$HOME/.ssh/known_hosts" -R "127.0.0.1"
/usr/bin/expect << EOF
set timeout 30
spawn /usr/bin/ssh -v -fCNL *:$1:127.0.0.1:$2 127.0.0.1
expect {
"yes/no" {send "yes\r" ; exp_continue}
"password:" {send "1234567\r" ; exp_continue}
eof
}
catch wait result
exit [lindex \$result 3]
EOF
echo "expect ssh return $?"
echo "forward $1 -> $2 done"
}
## check expect available
which expect > /dev/null
if [ $? -ne 0 ] ; then
echo "command expect not available"
exit 1
fi
login_port="10000"
forward_port="10001"
## check whether the number of elements is equal
login_port_num=$(echo ${login_port} | wc -w)
forward_port_num=$(echo ${forward_port} | wc -w)
if [ ${login_port_num} -ne ${forward_port_num} ] ; then
echo "The numbers of login ports and forward ports are not equal"
exit 1
fi
port_num=${login_port_num}
## provide pair of arguments to ssh main function
index=1
while [ ${index} -le ${port_num} ] ; do
login_p=$(echo ${login_port} | awk '{print $'$index'}')
forward_p=$(echo ${forward_port} | awk '{print $'$index'}')
connection ${login_p} ${forward_p}
index=$((index + 1))
done
Here the output from script
spawn /usr/bin/ssh -v -fCNL *:10000:127.0.0.1:10001 127.0.0.1
OpenSSH_7.2p2 Ubuntu-4ubuntu2.10, OpenSSL 1.0.2g 1 Mar 2016
...
debug1: Next authentication method: password
wang#127.0.0.1's password:
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (password).
Authenticated to 127.0.0.1 ([127.0.0.1]:22).
debug1: Local connections to *:10000 forwarded to remote address 127.0.0.1:10001
debug1: Local forwarding listening on 0.0.0.0 port 10000.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on :: port 10000.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions#openssh.com
debug1: forking to background
expect ssh return 0
forward 10000 -> 10001 done
This should work for you:
spawn -ignore SIGHUP ssh -f ...
UPDATE:
Another workaround is:
spawn bash -c "ssh -f ...; sleep 1"
UPDATE 2 (a bit explanation):
ssh -f calls daemon() to make itself a daemon. See ssh.c in the souce code:
/* Do fork() after authentication. Used by "ssh -f" */
static void
fork_postauth(void)
{
if (need_controlpersist_detach)
control_persist_detach();
debug("forking to background");
fork_after_authentication_flag = 0;
if (daemon(1, 1) == -1)
fatal("daemon() failed: %.200s", strerror(errno));
}
daemon() is implemented like this:
int
daemon(int nochdir, int noclose)
{
int fd;
switch (fork()) {
case -1:
return (-1);
case 0:
break;
default:
_exit(0);
}
if (setsid() == -1)
return (-1);
if (!nochdir)
(void)chdir("/");
if (!noclose && (fd = open(_PATH_DEVNULL, O_RDWR, 0)) != -1) {
(void)dup2(fd, STDIN_FILENO);
(void)dup2(fd, STDOUT_FILENO);
(void)dup2(fd, STDERR_FILENO);
if (fd > 2)
(void)close (fd);
}
return (0);
}
There's a race condition (not sure if its the correct term for here) between _exit() in the parent process and setsid() in the child process. Here _exit() would always complete first since "the function _exit() terminates the calling process immediately" and setsid() is much more heavy weight. So when the parent process exits, setsid() is not effective yet and the child process is still in the same session as the parent process. According to the apue book (I'm referring to the 2005 edition, Chapter 10: Signals), SIGHUP "is also generated if the session leader terminates. In this case, the signal is sent to each process in the foreground process group."
In brief:
Expect allocates a pty and runs ssh on the pty. Here, ssh would be running in a new session and be the session leader.
ssh -f calls daemon(). The parent process (session leader) calls _exit(). At this time, the child process is still in the session so it'll get SIGHUP whose default behavior is to terminate the process.
How the workarounds works:
The nohup way (spawn -ignore SIGHUP) is to explicitly ask the process to ignore SIGHUP so it'll not be terminated.
For bash -c 'sshh -f ...; sleep 1', bash would be the session leader and sleep 1 in the end prevents the session leader from exiting too soon. So after sleep 1, the child ssh process's setsid() has already done and child ssh is already in a new process session.
UPDATE 3:
You can compile ssh with the following modification (in ssh.c) and verify:
static int
my_daemon(int nochdir, int noclose)
{
int fd;
switch (fork()) {
case -1:
return (-1);
case 0:
break;
default:
// wait a while for child's setsid() to complete
sleep(1);
// ^^^^^^^^
_exit(0);
}
if (setsid() == -1)
return (-1);
if (!nochdir)
(void)chdir("/");
if (!noclose && (fd = open(_PATH_DEVNULL, O_RDWR, 0)) != -1) {
(void)dup2(fd, STDIN_FILENO);
(void)dup2(fd, STDOUT_FILENO);
(void)dup2(fd, STDERR_FILENO);
if (fd > 2)
(void)close (fd);
}
return (0);
}
/* Do fork() after authentication. Used by "ssh -f" */
static void
fork_postauth(void)
{
if (need_controlpersist_detach)
control_persist_detach();
debug("forking to background");
fork_after_authentication_flag = 0;
if (my_daemon(1, 1) == -1)
// ^^^^^^^^^
fatal("my_daemon() failed: %.200s", strerror(errno));
}
I have two servers ,
SERVER1: the first is installed with Redhat 7.7
SERVER2: the second server is installed with Redhat 6.10
The problem can be described in below 4 point:
1- I can ssh from SERVER1 to SERVER 2 and vice versa
as below:
> SERVER1,root,root # ssh SERVER2
>
> Last login: Sat May 16 02:06:48 2020 from x.x.x.x
>
> Agent pid 103002
>
> SERVER2,root,root #
To be noted that there is a new file is created under /tmp
> SERVER2,root,root # ll /tmp/ssh-tuqA103001
>
> total 0
>
> srw------- 1 root root 0 May 21 11:45 agent.103001
>
> SERVER2,root,root #
also ssh command ommits output Agent pid 103002 as above
each time a new file is created with a new number
2- I can ssh from SERVER2 to SERVER1 as below:
> SERVER2,root,root # ssh SERVER1
> Last login: Thu May 21 11:13:45 2020
> from x.x.x.x
> SERVER1,root,root #
to be noted that ssh does not omit the agent pid nor create any files under /tmp
3- I can scp any file from SERVER2 to SERVER1 as below:
> SERVER2,root,root # scp -rp test.sh SERVER1:/tmp
> test.sh 100% 470 0.5KB/s 00:00
> SERVER2,root,root#
4- But I can't transfer any file from SERVER1 to SERVER2 as below:
> SERVER1,root,root # scp -rp foo SERVER2:/tmp
> Agent pid 107142
> SERVER1,root,root #
I have captured the debug log for this scp failure as below:
SERVER1,root,root # scp -rpvvv foo SERVER2:/tmp
Executing: program /usr/bin/ssh host SERVER2, user (unspecified), command scp -v -r -p -t /tmp
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 49: Applying options for *
debug2: resolving "SERVER2" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to SERVER2 [SERVER2] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to SERVER2:22 as 'root'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from SERVER2
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01#openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01#openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ssh-ed25519-cert-v01#openssh.com,ssh-dss-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc#lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc#lysator.liu.se
debug2: MACs ctos: hmac-md5,hmac-sha1,umac-64#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160#openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5,hmac-sha1,umac-64#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160#openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib#openssh.com
debug2: compression stoc: none,zlib#openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64#openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64#openssh.com compression: none
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<3072<8192) sent
debug3: receive packet: type 31
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug2: bits set: 1577/3072
debug3: send packet: type 32
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug3: receive packet: type 33
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:Pcht7p5vfFNXT6fZIcf8m7qG3xdMViWiIVKGcaHV/xg
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 135.183.142.155
debug1: Host '135.183.142.155' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug2: bits set: 1542/3072
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /root/.ssh/id_rsa (0x55ebe8e53f10)
debug2: key: /root/.ssh/id_dsa (0x55ebe8e54300)
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug2: key: /root/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:0)
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:y9b1WKq87QvObEImsrh2qzj4Xucf4Zcq2Zz9uCEWfiw
debug3: sign_and_send_pubkey: RSA SHA256:y9b1WKq87QvObEImsrh2qzj4Xucf4Zcq2Zz9uCEWfiw
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to SERVER2 ([SERVER2]:22).
debug2: fd 4 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions#openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env GUESTFISH_INIT
debug3: Ignored env HOSTNAME
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env HISTSIZE
debug3: Ignored env SSH_CLIENT
debug3: Ignored env QTDIR
debug3: Ignored env QTINC
debug3: Ignored env SSH_TTY
debug3: Ignored env QT_GRAPHICSSYSTEM_CHECKED
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env ENV
debug3: Ignored env GUESTFISH_PS1
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env GUESTFISH_OUTPUT
debug3: Ignored env PS1
debug3: Ignored env HISTCONTROL
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env QTLIB
debug3: Ignored env CVS_RSH
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env LESSOPEN
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env GUESTFISH_RESTORE
debug3: Ignored env HISTFILE
debug3: Ignored env OLDPWD
debug3: Ignored env _
debug1: Sending command: scp -v -r -p -t /tmp
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
Agent pid 107427
debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug3: send packet: type 96
debug2: channel 0: input drain -> closed
ionnoclab2,root,root # debug2: channel 0: write failed
debug2: channel 0: close_write
debug2: channel 0: send eow
debug3: send packet: type 98
debug2: channel 0: output open -> closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
debug3: send packet: type 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 3312, received 2880 bytes, in 0.1 seconds
Bytes per second: sent 41262.8, received 35880.7
debug1: Exit status 0
SERVER1,root,root #**
What do you think the issue is? and how to solve it?
I developed this expect script, TELNET_TEST.expect to test a TELNET connection on a remote machine.
This script should connect via telnet on a target machine, wait for the login prompt, send the password and then exit.
This script does work and you can see in example 1 that the script does successfully login via telnet then exit, but something very confusing is going on, (to me).
Why do I get an exit status 1? I believe I should be getting an exit of status 0...
Please let me know why I am getting an exit of status 1? Also, what would I need to change in my script in order to get the exit code I am anticipating?
My expect script:
more TELNET_TEST.expect
#!/usr/bin/expect --
set LOGIN [lindex $argv 0]
set PASSWORD [lindex $argv 1]
set IP [lindex $argv 2]
set timeout 20
spawn telnet -l $LOGIN $IP
expect -re "(Password:|word:)"
send $PASSWORD\r
expect -re "(#|>)"
send exit\r
expect {
timeout {error "incorrect password"; exit 1}
eof
}
catch wait result
set STATUS [ lindex $result 3 ]
exit $STATUS
EXAMPLE1
Running the expect script from my Linux machine I get an exit status 1 even though the telnet login is ok.
./var/TELNET_TEST.expect root pass123 198.23.234.12
.
spawn telnet -l root pass123
Trying 198.23.234.12...
Connected to 198.23.234.12.
Escape character is '^]'.
Digital UNIX (machine1001) (ttyp0)
login: root
Password:
Last login: Mon Jul 14 16:40:15 from 198.23.234.12
Digital UNIX V4.0F (Rev. 1229); Wed Nov 23 15:08:48 IST 2005
****************************************************************************
Wide Area Networking Support V3.0-2 (ECO 3) for Digital UNIX is installed.
You have new mail.
machine1001> Connection closed by foreign host.
[root#LINUX_XOR]# echo $?
1
I see that in the transcript of you session:
machine1001> Connection closed by foreign host.
Exit code 1 is the exit code for "Connection closed by foreign host". That is the "correct" code when the connection is closed by the "other side" (in that case, in response to your exit command).
As far as I can tell, if you want an exit code of 0, you need to enter command mode in your telnet client and send the quit command. That way, the connection is closed by the client not by the foreign host. But is this really more "normal" than the other way?
From the sources of GNU telnet (inetutils-1.9), in the file commands.c:
int
tn (int argc, char *argv[])
{
....
.... many many lines of code here
....
close (net);
ExitString ("Connection closed by foreign host.\n", 1);
return 0;
}
and (utilities.c):
void
ExitString (char *string, int returnCode)
{
SetForExit ();
fwrite (string, 1, strlen (string), stderr);
exit (returnCode);
}
Below is a small bash script. The expected output after connecting to the server is to print Hello World and in the next line the current month and year - like Jan 2014. For using VARIABLE1 in the 'here document', I need dollar expansion, so the terminating character ~ is not quoted.
VARIABLE1="World"
ssh username#server.domain.com <<~
echo "Hello $VARIABLE1"
COMMAND1=`date +%b`
COMMAND2=$(date +%Y)
echo "$COMMAND1 $COMMAND2"
~
The actual output that I get is this :
Pseudo-terminal will not be allocated because stdin is not a terminal.
Password:
Hello World
When ssh is run in verbose mode, here is last part of the output:
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: *****
debug1: Trying private key: *****
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive).
debug1: Final hpn_buffer_size = *****
debug1: HPN Disabled: 0, HPN Buffer Size: *****
debug1: channel 0: new [client-session]
debug1: Enabled Dynamic Window Scaling
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
Hello World
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 1552, received 2360 bytes, in 0.1 seconds
Bytes per second: sent 17253.2, received 26235.6
debug1: Exit status 0
Could somebody point out the bug here? Why doesn't it print Jan 2014?
There is some information here about how to force a tty, however, your script should work if you escape your characters properly:
VARIABLE1="World"
ssh username#server.domain.com <<EOF
echo Hello $VARIABLE1
COMMAND1=\`date +%b\`
COMMAND2=\$(date +%Y)
echo "\$COMMAND1 \$COMMAND2"
EOF
If you want to force the tty:
ssh -t -t user#domain.com <<EOF
echo Hello $VARIABLE1
COMMAND1=\`date +%b\`
COMMAND2=\$(date +%Y)
echo "\$COMMAND1 \$COMMAND2"
exit
EOF
Alternatively, you could just send the script as an argument to ssh:
ssh user#domain.com "echo Hello $VARIABLE1;COMMAND1=\`date +%b\`; COMMAND2=\$(date +%Y); echo \$COMMAND1 \$COMMAND2"
EDIT: If you want to disable the message about the Pseudo-terminal allocation use "-T":
VARIABLE1="World"
ssh -T username#server.domain.com <<EOF
echo Hello $VARIABLE1
COMMAND1=\`date +%b\`
COMMAND2=\$(date +%Y)
echo "\$COMMAND1 \$COMMAND2"
EOF