I am creating an ssh tunnel using subprocess.Popen. However to successfully create this tunnel I am using a Yubikey which requires a pin to release the keys for successful authentication, built in with ssh config. Code below is as far as I can get.
def launch_tunnel(self):
try:
enterpin = getpass.getpass()
bytepin = str.encode(enterpin)
launchtunnel = subprocess.Popen('ssh tunnel command',
shell=True,
stdout=subprocess.PIPE,
stdin=subprocess.PIPE
stderr=subprocess.PIPE).communicate(input=bytepin)
except Exception as e:
print(e)
When I run it I get the following 2 prompts.
Password:
Enter PIN for 'PIV_II (PIV Card Holder pin)':
First one being getpass.getpass() and the second one being another process which requires the Yubikey Pin. It's clear that .communicate() is not working here and from what I can tell it's because the ssh process spawns another process(the pin prompt) that requires the pin for ssh authentication.
Is there anyway to set the pin prior using something like getpass and pass it directly to the 2nd process. Currently this 2nd process(pin prompt) is interrupting the rest of my application so I would like to control it?
I found part of the solution. Using communicate() without anything in () halts the process at the prompt so I do not need to implement getpass before launching subprocess. It has given me another issue but I will take that to a new thread.
launchtunnel = subprocess.Popen(checktunnelexists[1], shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
launchtunnel.communicate()[0]
Related
I have python script that use scp to transfer some files to other remote hosts.
try:
out = subprocess.check_output(f"scp filename.txt user1#host1:/home/user1/", stderr=subprocess.STDOUT, timeout=15)
print(out)
except subprocess.TimeoutExpired as e:
print(e.output)
// Handle hostkey verification error
Now some times, its possible that the remote server isnt authenticated and part of known hosts files, in which case the script just forever stays in waiting state for user input with Are you sure you want to continue connecting (yes/no/[fingerprint])?
What i want is to be able to catch this scenario, so i can handle host key verification.
I tried adding timeout and catch the exception but instead of timing out it still stays in user prompt state.
For some unknown reasons ,my browser open test pages of my remote server very slowly. So I am thinking if I can reconnect to the browser after quitting the script but don't execute webdriver.quit() this will leave the browser opened. It is probably kind of HOOK or webdriver handle.
I have looked up the selenium API doc but didn't find any function.
I'm using Chrome 62,x64,windows 7,selenium 3.8.0.
I'll be very appreciated whether the question can be solved or not.
No, you can't reconnect to the previous Web Browsing Session after you quit the script. Even if you are able to extract the Session ID, Cookies and other session attributes from the previous Browsing Context still you won't be able to pass those attributes as a HOOK to the WebDriver.
A cleaner way would be to call webdriver.quit() and then span a new Browsing Context.
Deep Dive
There had been a lot of discussions and attempts around to reconnect WebDriver to an existing running Browsing Context. In the discussion Allow webdriver to attach to a running browser Simon Stewart [Creator WebDriver] clearly mentioned:
Reconnecting to an existing Browsing Context is a browser specific feature, hence can't be implemented in a generic way.
With internet-explorer, it's possible to iterate over the open windows in the OS and find the right IE process to attach to.
firefox and google-chrome needs to be started in a specific mode and configuration, which effectively means that just
attaching to a running instance isn't technically possible.
tl; dr
webdriver.firefox.useExisting not implemented
Yes, that's actually quite easy to do.
A selenium <-> webdriver session is represented by a connection url and session_id, you just reconnect to an existing one.
Disclaimer - the approach is using selenium internal properties ("private", in a way), which may change in new releases; you'd better not use it for production code; it's better not to be used against remote SE (yours hub, or provider like BrowserStack/Sauce Labs), because of a caveat/resource drainage explained at the end.
When a webdriver instance is initiated, you need to get the before-mentioned properties; sample:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.google.com/')
# now Google is opened, the browser is fully functional; print the two properties
# command_executor._url (it's "private", not for a direct usage), and session_id
print(f'driver.command_executor._url: {driver.command_executor._url}')
print(f'driver.session_id: {driver.session_id}')
With those two properties now known, another instance can connect; the "trick" is to initiate a Remote driver, and provide the _url above - thus it will connect to that running selenium process:
driver2 = webdriver.Remote(command_executor=the_known_url)
# when the started selenium is a local one, the url is in the form 'http://127.0.0.1:62526'
When that is ran, you'll see a new browser window being opened.
That's because upon initiating the driver, the selenium library automatically starts a new session for it - and now you have 1 webdriver process with 2 sessions (browsers instances).
If you navigate to an url, you'll see it is executed on that new browser instance, not the one that's left from the previous start - which is not the desired behavior.
At this point, two things need to be done - a) close the current SE session ("the new one"), and b) switch this instance to the previous session:
if driver2.session_id != the_known_session_id: # this is pretty much guaranteed to be the case
driver2.close() # this closes the session's window - it is currently the only one, thus the session itself will be auto-killed, yet:
driver2.quit() # for remote connections (like ours), this deletes the session, but does not stop the SE server
# take the session that's already running
driver2.session_id = the_known_session_id
# do something with the now hijacked session:
driver.get('https://www.bing.com/')
And, that is it - you're now connected to the previous/already existing session, with all its properties (cookies, LocalStorage, etc).
By the way, you do not have to provide desired_capabilities when initiating the new remote driver - those are stored and inherited from the existing session you took over.
Caveat - having a SE process running can lead to some resource drainage in the system.
Whenever one is started and then not closed - like in the first piece of the code - it will stay there until you manually kill it. By this I mean - in Windows for example - you'll see a "chromedriver.exe" process, that you have to terminate manually once you are done with it. It cannot be closed by a driver that has connected to it as to a remote selenium process.
The reason - whenever you initiate a local browser instance, and then call its quit() method, it has 2 parts in it - the first one is to delete the session from the Selenium instance (what's done in the second code piece up there), and the other is to stop the local service (the chrome/geckodriver) - which generally works ok.
The thing is, for Remote sessions the second piece is missing - your local machine cannot control a remote process, that's the work of that remote's hub. So that 2nd part is literally a pass python statement - a no-op.
If you start too many selenium services on a remote hub, and don't have a control over it - that'll lead to resource drainage from that server. Cloud providers like BrowserStack take measures against this - they are closing services with no activity for the last 60s, etc, yet - this is something you don't want to do.
And as for local SE services - just don't forget to occasionally clean up the OS from orphaned selenium drivers you forgot about :)
OK after mixing various solutions shared on here and tweaking I have this working now as below. Script will use previously left open chrome window if present - the remote connection is perfectly able to kill the browser if needed and code functions just fine.
I would love a way to automate the getting of session_id and url for previous active session without having to write them out to a file during hte previous session for pick up...
This is my first post on here so apologies for breaking any norms
#Set manually - read/write from a file for automation
session_id = "e0137cd71ab49b111f0151c756625d31"
executor_url = "http://localhost:50491"
def attach_to_session(executor_url, session_id):
original_execute = WebDriver.execute
def new_command_execute(self, command, params=None):
if command == "newSession":
# Mock the response
return {'success': 0, 'value': None, 'sessionId': session_id}
else:
return original_execute(self, command, params)
# Patch the function before creating the driver object
WebDriver.execute = new_command_execute
driver = webdriver.Remote(command_executor=executor_url, desired_capabilities={})
driver.session_id = session_id
# Replace the patched function with original function
WebDriver.execute = original_execute
return driver
remote_session = 0
#Try to connect to the last opened session - if failing open new window
try:
driver = attach_to_session(executor_url,session_id)
driver.current_url
print(" Driver has an active window we have connected to it and running here now : ")
print(" Chrome session ID ",session_id)
print(" executor_url",executor_url)
except:
print("No Driver window open - make a new one")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=myoptions)
session_id = driver.session_id
executor_url = driver.command_executor._url
Without getting into why do you think that leaving an open browser windows will solve the problem of being slow, you don't really need a handle to do that. Just keep running the tests without closing the session or, in other words, without calling driver.quit() as you have mentioned yourself. The question here though framework that comes with its own runner? Like Cucumber?
In any case, you must have some "setup" and "cleanup" code. So what you need to do is to ensure during the "cleanup" phase that the browser is back to its initial state. That means:
Blank page is displayed
Cookies are erased for the session
I am trying to create a script that will run commands over my 1000 Cisco devices.
The device model is: Cisco Sx220 Series Switch Software, Version 1.1.4.1
The issue is that there is some kind of strange behavior for some of those Cisco devices.
When I am trying to login with regular SSH (PUTTY) with the correct credentials we are first getting 'Authentication Failure' and after 1 seconds I am getting the User Password Prompt again, typing the same credentials again is giving me a successful login.
The problem is that when I am trying to connect using my script (uses ParallelSSHClient), the connection drops after getting the authentication failure message and not able to enter the credentials again since it is getting the exception and terminal the program.
I am looking for a way to enter those credentials manual by connecting to the machine, getting the Authentication Failure message and ignoring it, recognizing that the current prompt has the User or Password appears on screen and then send it manually.
I look for this kind of procedure anywhere but without any luck.
Does ParallelSSHClient has this feature?
If Paramiko has it, I am willing to move to Paramiko.
Thanks :)
try:
client = ParallelSSHClient(hosts=ip_list, user=user, password=password)
except Exception as err:
print("There was an issue with connecting to the machine")
command_output = client.run_command(command)
Here is the accrual error that I am getting:
pssh.exceptions.AuthenticationException: ('Authentication error while connecting to %s:%s - %s', '172.31.255.10', 22, AuthenticationException('Password authentication failed',))
For some unknown reasons ,my browser open test pages of my remote server very slowly. So I am thinking if I can reconnect to the browser after quitting the script but don't execute webdriver.quit() this will leave the browser opened. It is probably kind of HOOK or webdriver handle.
I have looked up the selenium API doc but didn't find any function.
I'm using Chrome 62,x64,windows 7,selenium 3.8.0.
I'll be very appreciated whether the question can be solved or not.
No, you can't reconnect to the previous Web Browsing Session after you quit the script. Even if you are able to extract the Session ID, Cookies and other session attributes from the previous Browsing Context still you won't be able to pass those attributes as a HOOK to the WebDriver.
A cleaner way would be to call webdriver.quit() and then span a new Browsing Context.
Deep Dive
There had been a lot of discussions and attempts around to reconnect WebDriver to an existing running Browsing Context. In the discussion Allow webdriver to attach to a running browser Simon Stewart [Creator WebDriver] clearly mentioned:
Reconnecting to an existing Browsing Context is a browser specific feature, hence can't be implemented in a generic way.
With internet-explorer, it's possible to iterate over the open windows in the OS and find the right IE process to attach to.
firefox and google-chrome needs to be started in a specific mode and configuration, which effectively means that just
attaching to a running instance isn't technically possible.
tl; dr
webdriver.firefox.useExisting not implemented
Yes, that's actually quite easy to do.
A selenium <-> webdriver session is represented by a connection url and session_id, you just reconnect to an existing one.
Disclaimer - the approach is using selenium internal properties ("private", in a way), which may change in new releases; you'd better not use it for production code; it's better not to be used against remote SE (yours hub, or provider like BrowserStack/Sauce Labs), because of a caveat/resource drainage explained at the end.
When a webdriver instance is initiated, you need to get the before-mentioned properties; sample:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.google.com/')
# now Google is opened, the browser is fully functional; print the two properties
# command_executor._url (it's "private", not for a direct usage), and session_id
print(f'driver.command_executor._url: {driver.command_executor._url}')
print(f'driver.session_id: {driver.session_id}')
With those two properties now known, another instance can connect; the "trick" is to initiate a Remote driver, and provide the _url above - thus it will connect to that running selenium process:
driver2 = webdriver.Remote(command_executor=the_known_url)
# when the started selenium is a local one, the url is in the form 'http://127.0.0.1:62526'
When that is ran, you'll see a new browser window being opened.
That's because upon initiating the driver, the selenium library automatically starts a new session for it - and now you have 1 webdriver process with 2 sessions (browsers instances).
If you navigate to an url, you'll see it is executed on that new browser instance, not the one that's left from the previous start - which is not the desired behavior.
At this point, two things need to be done - a) close the current SE session ("the new one"), and b) switch this instance to the previous session:
if driver2.session_id != the_known_session_id: # this is pretty much guaranteed to be the case
driver2.close() # this closes the session's window - it is currently the only one, thus the session itself will be auto-killed, yet:
driver2.quit() # for remote connections (like ours), this deletes the session, but does not stop the SE server
# take the session that's already running
driver2.session_id = the_known_session_id
# do something with the now hijacked session:
driver.get('https://www.bing.com/')
And, that is it - you're now connected to the previous/already existing session, with all its properties (cookies, LocalStorage, etc).
By the way, you do not have to provide desired_capabilities when initiating the new remote driver - those are stored and inherited from the existing session you took over.
Caveat - having a SE process running can lead to some resource drainage in the system.
Whenever one is started and then not closed - like in the first piece of the code - it will stay there until you manually kill it. By this I mean - in Windows for example - you'll see a "chromedriver.exe" process, that you have to terminate manually once you are done with it. It cannot be closed by a driver that has connected to it as to a remote selenium process.
The reason - whenever you initiate a local browser instance, and then call its quit() method, it has 2 parts in it - the first one is to delete the session from the Selenium instance (what's done in the second code piece up there), and the other is to stop the local service (the chrome/geckodriver) - which generally works ok.
The thing is, for Remote sessions the second piece is missing - your local machine cannot control a remote process, that's the work of that remote's hub. So that 2nd part is literally a pass python statement - a no-op.
If you start too many selenium services on a remote hub, and don't have a control over it - that'll lead to resource drainage from that server. Cloud providers like BrowserStack take measures against this - they are closing services with no activity for the last 60s, etc, yet - this is something you don't want to do.
And as for local SE services - just don't forget to occasionally clean up the OS from orphaned selenium drivers you forgot about :)
OK after mixing various solutions shared on here and tweaking I have this working now as below. Script will use previously left open chrome window if present - the remote connection is perfectly able to kill the browser if needed and code functions just fine.
I would love a way to automate the getting of session_id and url for previous active session without having to write them out to a file during hte previous session for pick up...
This is my first post on here so apologies for breaking any norms
#Set manually - read/write from a file for automation
session_id = "e0137cd71ab49b111f0151c756625d31"
executor_url = "http://localhost:50491"
def attach_to_session(executor_url, session_id):
original_execute = WebDriver.execute
def new_command_execute(self, command, params=None):
if command == "newSession":
# Mock the response
return {'success': 0, 'value': None, 'sessionId': session_id}
else:
return original_execute(self, command, params)
# Patch the function before creating the driver object
WebDriver.execute = new_command_execute
driver = webdriver.Remote(command_executor=executor_url, desired_capabilities={})
driver.session_id = session_id
# Replace the patched function with original function
WebDriver.execute = original_execute
return driver
remote_session = 0
#Try to connect to the last opened session - if failing open new window
try:
driver = attach_to_session(executor_url,session_id)
driver.current_url
print(" Driver has an active window we have connected to it and running here now : ")
print(" Chrome session ID ",session_id)
print(" executor_url",executor_url)
except:
print("No Driver window open - make a new one")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=myoptions)
session_id = driver.session_id
executor_url = driver.command_executor._url
Without getting into why do you think that leaving an open browser windows will solve the problem of being slow, you don't really need a handle to do that. Just keep running the tests without closing the session or, in other words, without calling driver.quit() as you have mentioned yourself. The question here though framework that comes with its own runner? Like Cucumber?
In any case, you must have some "setup" and "cleanup" code. So what you need to do is to ensure during the "cleanup" phase that the browser is back to its initial state. That means:
Blank page is displayed
Cookies are erased for the session
I'm currently working on my web interface for git. Accessing git itself by child_process.spawn. Everything is fine while there is simple "command -> response" mechanism, but I cannot understand what should I do with command prompts (git fetch asks for password for example). Hypothetically there is some event fired, but I don't know what to listen to. All I see is "git_user#myserver's password: _" in command line where node.js process itself is running.
It would be great to redirect this request into my web application, but is it even possible?
I've tried to listen on message, data, pipe, end, close, readable at all streams (stdout, stdin, stderr), but no one fires on password prompt.
Here is my working solution (without mentioned experiments):
var out="";
var err="";
var proc=spawn(exe,cmd);
proc.on("exit",function(exitCode){
});
proc.stdout.on("data",function(data){
out+=data;
});
proc.stderr.on("data",function(data){
err+=data;
});
proc.on("close",function(code){
if(!code)func(out);
else return errHandler(err);
});
Can you please help me with my investigations?
UPDATE
Current situation: on my GIT web interface there is a button "FETCH" (as an example, for simple "git fetch"). When I press it, http request is generated and being sent to node.js server created by http.createServer(callback).listen(8080). callback function receives my request and creates child_process.spawn('git',['-C','path/to/local/repo','fetch']). All this time I see only loading screen on my web interface, but if I switch to command line window where node script is running I will see a password prompt. Now let's pretend that I can't switch window to console, because I work remotely.
I want to see password prompt on my web interface. It would be very easy to achieve if, for instance, child_process would emit some event on child.stdin (or somewhere else) when prompting for user input. In that case I would send string "Come on, dude, git wants to know your password! Enter it here: _______" back to web client (by response.end(str)), and will keep on waiting for the next http connection with client response, containing desired password. Then simply child.stdin.write(pass) it to git process.
Is this solution possible? Or something NOT involving command line with parent process.
UPDATE2
Just tried to attach listeners to all possible events described in official documentation: stdout and stderr (readable, data, end, close, error), stdin (drain, finish, pipe, unpipe, error), child (message, exit, close, disconnect, message).
Tried the same listeners on process.stdout, process.stderr after piping git streams to it.
Nothing fires on password request...
The main reason why your code wont work is because you only find out what happened with your Git process after is what executed.
The major reason to use spawn is beacause the spawned process can be configured, and stdout and stderr are Readable streams in the parent process.
I just tried this code out and it worked pretty good. Here is an example of spawning a process to perform a git push. However, as you may know git will ask you for username and password.
var spawn = require('child_process').spawn;
var git = spawn('git', ['push', 'origin', 'master']);
git.stderr.on('data', function(data) {
// do something with it
});
git.stderr.pipe(process.stderr);
git.stdout.pipe(process.stdout);
Make a local git repo and setup things so that you can do the above push command. However, you can really do any git command.
Copy this into a file called git_process.js.
Run with node git_process.js
Don't know if this would help but I found the only way to intercept the prompts from child processes was to set the detached option to true when you spawn a new child process.
Like you I couldn't find any info on prompts from child process in node on the interwebs. One would suspect it should go to stdout and then you would have to write to stdin. If I remember correctly you may find the prompt being sent to stderr.
Its a bit amazing to me that others haven't had this problem. Maybe we just doing it wrong.