subprocess.check_output failing within Docker - python-3.x

Similiar questions have been asked before, but the solutions for these questions don't resolve my own issue. I have a repo that relies on a base repo, whereby a connection is made to a neo4j database. My tests for the base repo work. The repo I have (transforms files) has already proven to work locally. When I run the repo in Docker I get a FileNotFoundError which is being thrown out from the base repo:
Base repo module
def get_docker_host_ip(self):
z = subprocess.check_output(['ip', '-4', 'route', 'list', 'match', '0/0'])
z = z.decode()[len('default via '):]
return z[: z.find(' ')]
The traceback:
Traceback (most recent call last):
File "/opt/app/src/myscript.py", line 252, in <module>
myModule = My_Module(args)
File "/opt/app/src/myscript.py", line 45, in __init__
self.neo4jadapter = Neo4jAdapter()
File "/opt/app/base/base/neo4j_adapter.py", line 19, in __init__
neo4j_host = self.get_docker_host_ip()
File "/opt/app/base/base/neo4j_adapter.py", line 51, in get_docker_host_ip
z = subprocess.check_output(['ip', '-4', 'route', 'list', 'match', '0/0'])
File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
**kwargs).stdout
File "/usr/lib/python3.7/subprocess.py", line 472, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ip': 'ip'
For subprocess.check_output, I have also added shell=True as recommended on other posts, but it doesn't make a different:
z = subprocess.check_output(['ip', '-4', 'route', 'list', 'match', '0/0'], shell=True)
This Traceback only occurs when I run the script in Docker. Where am I going wrong?

Thanks to #DavidMaze's comment, the ip tool needs to be installed via Docker, for example:
RUN apt-get install iproute2

Related

How to specify local shell for Fabric2/Paramiko/Invoke?

When trying to create a fabric2.Connection, Paramiko tries to invoke a local /bin/bash command:
$ fab2 db-shell
Traceback (most recent call last):
File "/nix/store/m2iyj18cifr4a1rvpfgphg7kfgsf2pj2-python3.9-fabric2-2.7.1/bin/.fab2-wrapped", line 9, in <module>
sys.exit(program.run())
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/program.py", line 384, in run
self.execute()
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/program.py", line 566, in execute
executor.execute(*self.tasks)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/executor.py", line 129, in execute
result = call.task(*args, **call.kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/tasks.py", line 127, in __call__
result = self.body(*args, **kwargs)
File "/home/username/project/fabfile.py", line 645, in db_shell
bastion_connection().run(
File "/home/username/project/fabfile_utils.py", line 141, in bastion_connection
conn = Connection(
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/fabric2/connection.py", line 403, in __init__
self.ssh_config = self.config.base_ssh_config.lookup(host)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 223, in lookup
options = self._lookup(hostname=hostname)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 250, in _lookup
or self._does_match(
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/paramiko/config.py", line 389, in _does_match
passed = invoke.run(exec_cmd, hide="stdout", warn=True).ok
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/__init__.py", line 48, in run
return Context().run(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/context.py", line 95, in run
return self._run(runner, command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/context.py", line 102, in _run
return runner.run(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 380, in run
return self._run_body(command, **kwargs)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 431, in _run_body
self.start(command, self.opts["shell"], self.env)
File "/nix/store/20k85mfdkbrj5w1pq0d7dzagygfip70h-python3-3.9.13-env/lib/python3.9/site-packages/invoke/runners.py", line 1291, in start
self.process = Popen(
File "/nix/store/0zzvjh5gnz0ny7ckilzyn9hmg5lypszf-python3-3.9.13/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/nix/store/0zzvjh5gnz0ny7ckilzyn9hmg5lypszf-python3-3.9.13/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/bin/bash'
I've not been able to find anything in the docs about this yet.
I'm running fabric2==2.7.1, invoke==1.6.0, and paramiko==2.8.0 on NixOS stable 22.11.
Invoke currently defaults to /bin/bash, which is a known issue, since operating systems like Alpine and NixOS don't have that path. Fortunately we can specify a command in $PATH rather than manually looking it up, so adding the following to the Connection call does the trick: config=invoke.Config(overrides={"shell": "bash"}).

Can't install django with pipenv although it is installed and identified by the command prompt

I've been trying to set up a virtual environment for django using pipenv. I've installed pipenv and configured the path properly. When I run the following command
pipenv --version
I get
pipenv, version 2022.9.8
Which implies that pipenv is working.
But when I attempt to install django with the following command
pipenv install django
I get the following error.
C:\Users\Saifia>pipenv install django
Traceback (most recent call last):
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 620, in parse_executable
result_version = get_python_version(path)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\utils.py", line 97, in get_python_version
c = subprocess.Popen(version_cmd, **subprocess_kwargs)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1420, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts\pipenv.exe\__main__.py", line 7, in <module>
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\click\core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\cli\options.py", line 57, in main
return super().main(*args, **kwargs, windows_expand_args=False)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\click\core.py", line 1053, in main
rv = self.invoke(ctx)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\click\core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\click\core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\click\core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\click\decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\click\core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\cli\command.py", line 236, in install
do_install(
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\core.py", line 2056, in do_install
ensure_project(
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\core.py", line 526, in ensure_project
ensure_virtualenv(
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\core.py", line 447, in ensure_virtualenv
python = ensure_python(project, three=three, python=python)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\core.py", line 336, in ensure_python
path_to_python = find_a_system_python(python)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\core.py", line 302, in find_a_system_python
return next(iter(finder.find_all_python_versions()), None)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 307, in find_all_python_versions
python_version_dict = getattr(self.system_path, "python_version_dict", {})
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 115, in system_path
self._system_path = self.create_system_path()
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\pythonfinder.py", line 77, in create_system_path
return pyfinder_path.SystemPath.create(
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 696, in create
instance = instance._run_setup()
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 247, in _run_setup
new_instance = new_instance._setup_windows()
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\path.py", line 425, in _setup_windows
windows_finder = WindowsFinder.create()
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\windows.py", line 149, in create
return cls()
File "<attrs generated init pythonfinder.models.windows.WindowsFinder>", line 13, in __init__
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\windows.py", line 116, in get_versions
versions[py_version.version_tuple[:5]] = base_dir
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 367, in __getattribute__
result = super(PythonVersion, self).__getattribute__(key)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 432, in version_tuple
self.patch,
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 377, in __getattribute__
instance_dict = self.parse_executable(executable)
File "C:\Users\Saifia\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pipenv\vendor\pythonfinder\models\python.py", line 622, in parse_executable
raise ValueError("Not a valid python path: %r" % path)
ValueError: Not a valid python path: 'C:/Users/Saifia/anaconda3/Scripts/python.exe'
I've been trying to figure this out the whole day and have tried various things but I'm new this and I can't figure out my mistake.
Any help will be greatly appreciated!
I was using virtualenv for virtualization and had to use pipenv to follow a tutorial and could not get pipenv to work. I had to uninstall virtualenv and pipenv and reinstall pipenv for it to work. I am still using pipenv but I do not like it virtualenv works better for me.
I think that the key to your problem is the last message in the traceback:
"ValueError: Not a valid python path: 'C:/Users/Saifia/anaconda3/Scripts/python.exe'"
Check and see if this file exists at this location.
If not, your default python version isn't set correctly in pipenv.
Deactivate any virtual environment and check your python version before you install pipenv.

Python3 subprocess can't run command "la" or "ll"

completeProcess = subprocess.run("la")
This is my code to run commands via python, and it raise this error:
Traceback (most recent call last):
File "script_senior.py", line 171, in <module>
completeProcess = subprocess.run("la") # doesn't capture output
File "/usr/lib/python3.5/subprocess.py", line 693, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'la'
My OS is ubuntu, and when enter "la" in terminal, it will list files(include hidden files) in current working directory. and "ll" cmd will list file details. but I can't use them in python's subprocess api
How can I achieve this function? Is there any common api to run commands or just the python didn't recognize "la"
la is an alias for 'ls -A'. That comes as a default shell alias in systems like Ubuntu. If you wanna run it just use:
completeProcess = subprocess.run(["ls","-A"])
and if you are looking for the stdout you can use something like:
completeProcess = subprocess.run(["ls","-A"] , stdout=subprocess.PIPE)
print(completeProcess.stdout.decode())

Migration error at startup for Jupyter Notebook

Installed Anaconda3 (Anaconda Navigator 1.8.7) for Mac OS X (10.13.5) as part of a JuliaPro (0.6.3) install. Every time I try to launch Jupyter Notebook (Terminal or Navigator GUI), I get this:
Prestbury:~ me$ jupyter notebook Traceback (most recent call last):
File "/anaconda3/bin/jupyter-notebook", line 11, in
sys.exit(main()) File "/anaconda3/lib/python3.6/site-packages/jupyter_core/application.py",
line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs) File
"/anaconda3/lib/python3.6/site-packages/traitlets/config/application.py",
line 657, in launch_instance
app.initialize(argv) File "", line 2, in initialize File
"/anaconda3/lib/python3.6/site-packages/traitlets/config/application.py",
line 87, in catch_config_error
return method(app, *args, **kwargs) File "/anaconda3/lib/python3.6/site-packages/notebook/notebookapp.py", line
1501, in initialize
super(NotebookApp, self).initialize(argv) File "", line 2, in initialize File
"/anaconda3/lib/python3.6/site-packages/traitlets/config/application.py",
line 87, in catch_config_error
return method(app, *args, **kwargs) File "/anaconda3/lib/python3.6/site-packages/jupyter_core/application.py",
line 242, in initialize
self.migrate_config() File "/anaconda3/lib/python3.6/site-packages/jupyter_core/application.py",
line 168, in migrate_config
migrate() File "/anaconda3/lib/python3.6/site-packages/jupyter_core/migrate.py", line
247, in migrate
with open(os.path.join(env['jupyter_config'], 'migrated'), 'w') as f: PermissionError: [Errno 13] Permission denied:
'/Users/me/.jupyter/migrated'
I have completely removed Anaconda3 and performed a fresh install after also installing nodejs. The problem persists. Any recommendation please?

Cannot load jupyter-notebook from terminal in Ubuntu 14.04

After installing the latest version of Anaconda, and following post-installation steps, I am not able to load jupyter notebook on my system, Ubuntu 14.04.
I get the following error message:
Traceback (most recent call last):
File "./bin/jupyter-notebook", line 6, in <module>
sys.exit(notebook.notebookapp.main())
File "/home/carlos/anaconda3/lib/python3.5/site-packages/jupyter_core /application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/home/carlos/anaconda3/lib/python3.5/site-packages/traitlets/config/application.py", line 588, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/home/carlos/anaconda3/lib/python3.5/site-packages/traitlets/config/application.py", line 74, in catch_config_error
return method(app, *args, **kwargs)
File "/home/carlos/anaconda3/lib/python3.5/site-packages/notebook/notebookapp.py", line 1023, in initialize
self.init_webapp()
File "/home/carlos/anaconda3/lib/python3.5/site-packages/notebook/notebookapp.py", line 882, in init_webapp
xheaders=self.trust_xheaders)
File "/home/carlos/anaconda3/lib/python3.5/site-packages/tornado/util.py", line 221, in __new__
instance.initialize(*args, **init_kwargs)
File "/home/carlos/anaconda3/lib/python3.5/site-packages/tornado/httpserver.py", line 155, in initialize
read_chunk_size=chunk_size)
File "/home/carlos/anaconda3/lib/python3.5/site-packages/tornado/tcpserver.py", line 112, in __init__
self.ssl_options['certfile'])
ValueError: certfile "jupnb.pem" does not exist
The certfile actually exists, in the directory ~/.jupyter/
I haven't been lucky googling the solution. Any comments?
Thanks!
Carlos
edit ~/.jupyter/jupyter_notebook_config.py and check the correct paths,
e.g. for ~/.jupyter/jupnb.key and ~/.jupyter/jupnb.pem:
import os
c.NotebookApp.keyfile = os.path.expanduser('~') + '/.jupyter/jupnb.key'
c.NotebookApp.certfile = os.path.expanduser('~') + '/.jupyter/jupnb.pem'

Resources