Python3 subprocess can't run command "la" or "ll" - python-3.x

completeProcess = subprocess.run("la")
This is my code to run commands via python, and it raise this error:
Traceback (most recent call last):
File "script_senior.py", line 171, in <module>
completeProcess = subprocess.run("la") # doesn't capture output
File "/usr/lib/python3.5/subprocess.py", line 693, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'la'
My OS is ubuntu, and when enter "la" in terminal, it will list files(include hidden files) in current working directory. and "ll" cmd will list file details. but I can't use them in python's subprocess api
How can I achieve this function? Is there any common api to run commands or just the python didn't recognize "la"

la is an alias for 'ls -A'. That comes as a default shell alias in systems like Ubuntu. If you wanna run it just use:
completeProcess = subprocess.run(["ls","-A"])
and if you are looking for the stdout you can use something like:
completeProcess = subprocess.run(["ls","-A"] , stdout=subprocess.PIPE)
print(completeProcess.stdout.decode())

Related

How to specify local shell for Fabric2/Paramiko/Invoke?

When trying to create a fabric2.Connection, Paramiko tries to invoke a local /bin/bash command:
$ fab2 db-shell
Traceback (most recent call last):
File "/nix/store/m2iyj18cifr4a1rvpfgphg7kfgsf2pj2-python3.9-fabric2-2.7.1/bin/.fab2-wrapped", line 9, in <module>
sys.exit(program.run())
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/program.py", line 384, in run
self.execute()
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/program.py", line 566, in execute
executor.execute(*self.tasks)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/executor.py", line 129, in execute
result = call.task(*args, **call.kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/tasks.py", line 127, in __call__
result = self.body(*args, **kwargs)
File "/home/username/project/fabfile.py", line 645, in db_shell
bastion_connection().run(
File "/home/username/project/fabfile_utils.py", line 141, in bastion_connection
conn = Connection(
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/fabric2/connection.py", line 403, in __init__
self.ssh_config = self.config.base_ssh_config.lookup(host)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 223, in lookup
options = self._lookup(hostname=hostname)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 250, in _lookup
or self._does_match(
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 389, in _does_match
passed = invoke.run(exec_cmd, hide="stdout", warn=True).ok
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/__init__.py", line 48, in run
return Context().run(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/context.py", line 95, in run
return self._run(runner, command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/context.py", line 102, in _run
return runner.run(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 380, in run
return self._run_body(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 431, in _run_body
self.start(command, self.opts["shell"], self.env)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 1291, in start
self.process = Popen(
File "/nix/store/0zzvjh5gnz0ny7ckilzyn9hmg5lypszf-python3-3.9.13/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/nix/store/0zzvjh5gnz0ny7ckilzyn9hmg5lypszf-python3-3.9.13/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/bin/bash'
I've not been able to find anything in the docs about this yet.
I'm running fabric2==2.7.1, invoke==1.6.0, and paramiko==2.8.0 on NixOS stable 22.11.
Invoke currently defaults to /bin/bash, which is a known issue, since operating systems like Alpine and NixOS don't have that path. Fortunately we can specify a command in $PATH rather than manually looking it up, so adding the following to the Connection call does the trick: config=invoke.Config(overrides={"shell": "bash"}).

subprocess.check_output failing within Docker

Similiar questions have been asked before, but the solutions for these questions don't resolve my own issue. I have a repo that relies on a base repo, whereby a connection is made to a neo4j database. My tests for the base repo work. The repo I have (transforms files) has already proven to work locally. When I run the repo in Docker I get a FileNotFoundError which is being thrown out from the base repo:
Base repo module
def get_docker_host_ip(self):
z = subprocess.check_output(['ip', '-4', 'route', 'list', 'match', '0/0'])
z = z.decode()[len('default via '):]
return z[: z.find(' ')]
The traceback:
Traceback (most recent call last):
File "/opt/app/src/myscript.py", line 252, in <module>
myModule = My_Module(args)
File "/opt/app/src/myscript.py", line 45, in __init__
self.neo4jadapter = Neo4jAdapter()
File "/opt/app/base/base/neo4j_adapter.py", line 19, in __init__
neo4j_host = self.get_docker_host_ip()
File "/opt/app/base/base/neo4j_adapter.py", line 51, in get_docker_host_ip
z = subprocess.check_output(['ip', '-4', 'route', 'list', 'match', '0/0'])
File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
**kwargs).stdout
File "/usr/lib/python3.7/subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ip': 'ip'
For subprocess.check_output, I have also added shell=True as recommended on other posts, but it doesn't make a different:
z = subprocess.check_output(['ip', '-4', 'route', 'list', 'match', '0/0'], shell=True)
This Traceback only occurs when I run the script in Docker. Where am I going wrong?
Thanks to #DavidMaze's comment, the ip tool needs to be installed via Docker, for example:
RUN apt-get install iproute2

Sometimes throws Bad file descriptor error, sometimes doesn't

The following piece of code was working fine for a year, of late it intermittently started failing with following error, wonder if any OS or hardware change is causing this
import subprocess
def bash_command(cmd):
process = subprocess.Popen(cmd, shell=True, executable="/bin/bash")
process.wait()
bash_command("bash myscript.sh")
The error message is:
Traceback (most recent call last): File
"/Users/ishandutta2007/Documents/Projects/p2/shell_way/all_in_one_refill.py",
line 68,
bash myscript.sh File "/Users/ishandutta2007/Documents/Projects/p2/shell_way/all_in_one_refill.py",
line 31, in bash_command
process = subprocess.Popen(cmd, shell=True, executable="/bin/bash") File
"/Users/ishandutta2007/.pyenv/versions/3.6.0/lib/python3.6/subprocess.py",
line 707, in __init__
restore_signals, start_new_session) File "/Users/ishandutta2007/.pyenv/versions/3.6.0/lib/python3.6/subprocess.py",
line 1326, in _execute_child
raise child_exception_type(errno_num, err_msg) OSError: [Errno 9] Bad file descriptor

How to use date command inside Popen?

I cannot use parameters when invoking date command inside Popen.
I am using Ubuntu 18.04.2 LTS, Python 3.6.7. I am trying to just print date by invokin date command from shell, unfortunately I got errors which says
Traceback (most recent call last):
File "./123.py", line 15, in <module>
print_date()
File "./123.py", line 9, in print_date
MyOut = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: "date '+%S'": "date '+%S'"
Code:
#!/usr/bin/python3
import subprocess
import time
import os
cmd = "date '+%S'"
def print_date():
MyOut = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout,stderr = MyOut.communicate()
print(stdout)
print(stderr)
os.system(cmd)
print_date()
Could anyone explain what went wrong, and why Python tries to find file "date '+%S'" ?
It works when we change cmd to cmd = "date", so what is correct way of passing '+%S' ?
The normal way to pass a command to Popen() is with a list. Try this:
cmd = ["date" '+%S']

CNTK on Jupyter notebook kernal not working

I am trying to get Jupyter Notebook to work with my Azure data Science VM and the kernel in not working. I installes CNTK using the shellbash and it works well from the command like. However when I pull it up using Anaconda or with the command: "jupyter notebook" I get this error:
Traceback (most recent call last):
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site- packages\notebook\base\handlers.py", line 503, in wrapper
result = yield gen.maybe_future(method(self, *args, **kwargs))
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site- packages\tornado\gen.py", line 1015, in run
value = future.result()
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\gen.py", line 1021, in run
yielded = self.gen.throw(*exc_info)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\notebook\services\sessions\handlers.py", line 75, in post
type=mtype))
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\gen.py", line 1015, in run
value = future.result()
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\gen.py", line 1021, in run
yielded = self.gen.throw(*exc_info)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 79, in create_session
kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\gen.py", line 1015, in run
value = future.result()
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\gen.py", line 1021, in run
yielded = self.gen.throw(*exc_info)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 92, in start_kernel_for_session
self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\gen.py", line 1015, in run
value = future.result()
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\tornado\gen.py", line 285, in wrapper
yielded = next(result)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\notebook\services\kernels\kernelmanager.py", line 87, in start_kernel
super(MappingKernelManager, self).start_kernel(**kwargs)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\jupyter_client\multikernelmanager.py", line 110, in start_kernel
km.start_kernel(**kwargs)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\jupyter_client\manager.py", line 243, in start_kernel
**kw)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\jupyter_client\manager.py", line 189, in _launch_kernel
return launch_kernel(kernel_cmd, **kw)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\site-packages\jupyter_client\launcher.py", line 123, in launch_kernel
proc = Popen(cmd, **kwargs)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py35\lib\subprocess.py", line 1224, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
Any advice on how to get the python kernel to work with CNTK in the notebook?
I opened up the terminal to the CNTK-py35 directory and entered the following code and the kernel started working got it working:
conda create -n py35 python=3.5
activate py35
conda install notebook ipykernel
ipython kernel install --user --name=python3.5
Then I opened jupyter notebook from inside Anaconda and ran
import CNTK
Need a bit more information to give you meaningful advice, as I cannot see it from the screen dump.
Are you able to activate CNTK-py35 environment
If you are able to activate the environment, could you please start the python kernel from within the activated environment
If you are successful with that from within the python shell are you able to run import CNTK
If that works, that means your CNTK installation is good.
Now you can go the command prompt, from within the activate environment can you try starting the jupyter notebook environment.
You can find more information along these lines for the python setup, at this link: https://github.com/Microsoft/CNTK/wiki/Setup-Windows-Python

Resources