AWS CLI in CMD and PowerShell - python-3.x

On Win 10 I've installed Anaconda (via chocolatey) and the AWS CLI. I can successfully issue commands via the AWS CLI as long as I'm in one of the following:
an Anaconda prompt
a Git Bash prompt
the powershell terminal inside of visual studio code
However, when I try to issue commands from a "straight-up" CMD or PowerShell window, then I get errors:
C:\>aws s3 ls
Traceback (most recent call last):
File "c:\Users\barias\AppData\Roaming\Python\Python37\Scripts\aws.cmd", line 50, in <module>
import awscli.clidriver
File "C:\Users\barias\AppData\Roaming\Python\Python37\site-packages\awscli\clidriver.py", line 17, in <module>
import botocore.session
File "C:\tools\Anaconda3\lib\site-packages\botocore\session.py", line 30, in <module>
import botocore.credentials
File "C:\tools\Anaconda3\lib\site-packages\botocore\credentials.py", line 34, in <module>
from botocore.config import Config
File "C:\tools\Anaconda3\lib\site-packages\botocore\config.py", line 16, in <module>
from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
File "C:\tools\Anaconda3\lib\site-packages\botocore\endpoint.py", line 22, in <module>
from botocore.awsrequest import create_request_object
File "C:\tools\Anaconda3\lib\site-packages\botocore\awsrequest.py", line 26, in <module>
import botocore.utils
File "C:\tools\Anaconda3\lib\site-packages\botocore\utils.py", line 31, in <module>
import botocore.httpsession
File "C:\tools\Anaconda3\lib\site-packages\botocore\httpsession.py", line 7, in <module>
from urllib3.util.ssl_ import (
ImportError: cannot import name 'ssl' from 'urllib3.util.ssl_' (C:\tools\Anaconda3\lib\site-packages\urllib3\util\ssl_.py)
I am not a Python programmer, so it is not self-evident what this error means.
A side commentary on my Anaconda install:
I am surprised to see Anaconda installed in C:\tools. I think that is because (1) my chocolatey script which installed Anaconda was executed inside that folder and (2) the Anaconda chocolatey package perhaps was designed to install in the current working directory? Regardless, I don't think this problem is a result of where Anaconda is installed.
I also found that I had to manually put the python exe onto the path. I'm not sure why the Anaconda install didn't do that for me.
As it turns out, I can successfully issue terraform commands from CMD or PowerShell to that same AWS account. I would have thought if the AWS CLI wasn't working, then terraform would be hosed as well. Weird.
None of the prompts (Git Bash, Anaconda, CMD, PowerShell) have PYTHONPATH OR PYTHONHOME environment variables set. The Anaconda path has these environment variables:
CONDA_DEFAULT_ENV=base
CONDA_EXE=C:\tools\Anaconda3\Scripts\conda.exe
CONDA_PREFIX=C:\tools\Anaconda3
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=C:\tools\Anaconda3\python.exe
CONDA_SHLVL=1
I tried setting these in the CMD window, but it did not fix the problem. Also, the Git Bash prompt successfully handles the AWS CLI...and it doesn't have any environment variables configured regarding python or "conda".
Any ideas of why I'm getting this error? Remember, it works fine as long as I execute commands from an Anaconda prompt.

Related

class YAMLObject(metaclass=YAMLObjectMetaclass): invalid syntax for docker-compose up command

On my Linux system two python versions are present 2.7 and 3.6. To use python 3.6. in my PYTHONPATH in .profile file I have kept /usr/local/lib64/python3.6/site-packages and /usr/local/lib/python3.6/site-packages at the top. and there is no entry of python 2.7 path. In my sys.path variable I can see the python2.7 path entries. (dont know how they appear)
now the issue is when I use docker-compose up command it gives me below error. docker-compose version is 1.29.2
Traceback (most recent call last):
File "/usr/bin/docker-compose", line 7, in <module>
from compose.cli.main import main
File "/usr/lib/python2.7/site-packages/compose/cli/main.py", line 22, in <module>
from ..bundle import get_image_digests
File "/usr/lib/python2.7/site-packages/compose/bundle.py", line 12, in <module>
from .config.serialize import denormalize_config
File "/usr/lib/python2.7/site-packages/compose/config/__init__.py", line 6, in <module>
from .config import ConfigurationError
File "/usr/lib/python2.7/site-packages/compose/config/config.py", line 13, in <module>
import yaml
File "/usr/local/lib64/python3.6/site-packages/yaml/__init__.py", line 284
class YAMLObject(metaclass=YAMLObjectMetaclass):
^
SyntaxError: invalid syntax
If I add python 2.7 site package path at the top of PYTHONPATH docker-compose works but other programs start failing. I dont want to use python 2.7 how can I use only python 3.6.
Please suggest.

ModuleNotFoundError: No module named 'six' - python 3.8.5

When trying to open jupyter notebook from my terminal in Visual Studio Code, I keep getting the error below. I have tried uninstalling and reinstalling six using conda install but the same issue persists. It was working fine yesterday so I don't know what happened between now and then. I'm using Python 3.8.5 in my virtual environment. Any ideas what could be the issue?
(dreams) C:\Users\jacks\Documents\dreams_project>jupyter notebook
Traceback (most recent call last):
File "C:\Users\jacks\Anaconda3\Scripts\jupyter-notebook-script.py", line 6, in <module>
from notebook.notebookapp import main
File "C:\Users\jacks\Anaconda3\lib\site-packages\notebook\notebookapp.py", line 83, in <module>
from .services.kernels.kernelmanager import MappingKernelManager, AsyncMappingKernelManager
File "C:\Users\jacks\Anaconda3\lib\site-packages\notebook\services\kernels\kernelmanager.py", line 18, in <module>
from jupyter_client.session import Session
File "C:\Users\jacks\Anaconda3\lib\site-packages\jupyter_client\session.py", line 41, in <module>
from jupyter_client.jsonutil import extract_dates, squash_dates, date_default
File "C:\Users\jacks\Anaconda3\lib\site-packages\jupyter_client\jsonutil.py", line 10, in <module>
from dateutil.parser import parse as _dateutil_parse
File "C:\Users\jacks\Anaconda3\lib\site-packages\dateutil\parser\__init__.py", line 2, in <module>
from ._parser import parse, parser, parserinfo, ParserError
File "C:\Users\jacks\Anaconda3\lib\site-packages\dateutil\parser\_parser.py", line 42, in <module>
import six
ModuleNotFoundError: No module named 'six'
I had the same issue, six was showing as installed by conda and loading fine when imported manually in python. The base version of Jupyter Notebook would load, but Jupyter Notebook wouldn't load in my virtual environment.
Running pip3 install six from a terminal with my environment activated seemed to solve the issue but I don't know why!
Also make sure the path to Conda in the VS Python extension settings is correct.

Docker, Python, and "import logging"

I have trying to build a docker image/container that is Python 3.8.2 and that will run a specific script when ran. I can get it to build but when I try and run it breaks.
Specifically, I am running into a problem where:
...
import grpc
...
gives the error:
Traceback (most recent call last):
File "/project1/script1.py", line 4, in <module>
import grpc
File "/usr/local/lib/python3.8/site-packages/grpc/__init__.py", line 23, in <module>
from grpc._cython import cygrpc as _cygrpc
File "src/python/grpcio/grpc/_cython/cygrpc.pyx", line 27, in init grpc._cython.cygrpc
File "/usr/local/lib/python3.8/asyncio/__init__.py", line 8, in <module>
from .base_events import *
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 18, in <module>
import concurrent.futures
File "/usr/local/lib/python3.8/concurrent/futures/__init__.py", line 8, in <module>
from concurrent.futures._base import (FIRST_COMPLETED,
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 42, in <module>
LOGGER = logging.getLogger("concurrent.futures")
AttributeError: module 'logging' has no attribute 'getLogger'
HOWEVER,
if i use the
docker run -it <image name> sh
to run the image, if I try to run pip in the Docker containers command line I get the same error.
Here is the Dockerfile used to create the image:
FROM python:3.8.2
#Build the Python Environments
COPY requirements.txt /tmp/
RUN pip install -r /tmp/requirements.txt
RUN pip install "obspy==1.2.1"
#COPY project1
COPY project1 /
#Update PYTHON PATH to include project1
ENV PYTHONPATH "${PYTHONPATH}:/project1"
#Run the script1.py script
CMD python /project1/script1.py
All of this runs fine on my windows box in a Python 3.8.2 environment that contains all the packages in requirements.txt
Anyone have any suggestions on what is going on?
May be try with a different python version image.
Try mentioning the specific version of the dependency in requirements.txt file
Is there any python file named logging.py or module folder named logging in your project? If so try give it another name.

Setting up Hydrogen and Atom with Anaconda managing python installation

I have added my python 3 executable to the system PATH (against the advice of Anaconda) to try and get Hydrogen (and really any Atom extension/plugin) to run lines or blocks of code in Atom. The 'scripts' atom plugin appears to work (I select some code and enter ctrl-shift-b), but I'd love to use more of the features in Hydrogen. When I execute (for example):
print('hello world')
I get the following error:
Python 3
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec)
File "C:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code exec(code, run_globals)
File "C:\ProgramData\Anaconda3\lib\site-packages\ipykernel_launcher.py", line 15, in from ipykernel import kernelapp as app
File "C:\ProgramData\Anaconda3\lib\site-packages\ipykernel__init__.py", line 2, in from .connect import
File "C:\ProgramData\Anaconda3\lib\site-packages\ipykernel\connect.py", line 18, in import jupyter_client File "C:\ProgramData\Anaconda3\lib\site-packages\jupyter_client__init__.py", line 4, in from .connect import
File "C:\ProgramData\Anaconda3\lib\site-packages\jupyter_client\connect.py", line 23, in import zmq
File "C:\ProgramData\Anaconda3\lib\site-packages\zmq__init__.py", line 47, in from zmq import backend
File "C:\ProgramData\Anaconda3\lib\site-packages\zmq\backend__init__.py", line 40, in reraise(*exc_info)
File "C:\ProgramData\Anaconda3\lib\site-packages\zmq\utils\sixcerpt.py", line 34, in reraise raise value
File "C:\ProgramData\Anaconda3\lib\site-packages\zmq\backend__init__.py", line 27, in _ns = select_backend(first)
File "C:\ProgramData\Anaconda3\lib\site-packages\zmq\backend\select.py", line 27, in select_backend mod = import(name, fromlist=public_api)
File "C:\ProgramData\Anaconda3\lib\site-packages\zmq\backend\cython__init__.py", line 6, in from . import (constants, error, message, context,
ImportError: DLL load failed: The specified module could not be found.
I have reinstalled the package, I've tried using the Anaconda power shell and normal prompt to install and load Atom. My only guess is that its having trouble launching a kernel, or am I supposed to launch one and then connect?
End goal:
run code block and it works.
To use a Conda env as a kernel in Hydrogen you must register the env using ipykernel, e.g.,
conda activate myenv
python -m ipykernel install --user
This creates an entry for the kernel in a default user-level location that is generically visible to any Jupyter instances run by the user (such as Hydrogen). It is recommended to also include a --name NAME flag to distinguish your different envs. Please refer to the python -m ipykernel install -h for more options.
Also, note that the minimum requirement for using a Conda env as a kernel is to install ipykernel. And, yeah, clean up the PATH so that it conforms to Conda best practices - there should be no need for manually editing.

Tensorflow Object Detection API / ImportError: cannot import name 'keypoint_box_coder_pb2'

I followed instructions (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md) to install the Tensorflow Object Detection
API. I use Anaconda on Windows 10 and all of the steps in the instructions seemed to complete OK.
But when trying to run python object_detection/builders/model_builder_test.py to test the installation, I get: ImportError: cannot import name 'keypoint_box_coder_pb2'
What am I missing?
(tensorflow) C:\Users\User\Documents\GitHub\models\research>python
object_detection/builders/model_builder_test.py Traceback (most recent
call last): File "object_detection/builders/model_builder_test.py",
line 21, in
from object_detection.builders import model_builder File "C:\Users\User\New\Anaconda3\envs\tensorflow\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\builders\model_builder.py",
line 18, in
from object_detection.builders import box_coder_builder File "C:\Users\User\New\Anaconda3\envs\tensorflow\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\builders\box_coder_builder.py",
line 21, in
from object_detection.protos import box_coder_pb2 File "C:\Users\User\New\Anaconda3\envs\tensorflow\lib\site-packages\object_detection-0.1-py3.5.egg\object_detection\protos\box_coder_pb2.py",
line 17, in ImportError: cannot import name
'keypoint_box_coder_pb2'
I got it to work. What I did was:
I ran:
protoc --python_out=. .\object_detection\protos\anchor_generator.proto .\object_detection\protos\argmax_matcher.proto .\object_detection\protos\bipartite_matcher.proto .\object_detection\protos\box_coder.proto .\object_detection\protos\box_predictor.proto .\object_detection\protos\eval.proto .\object_detection\protos\faster_rcnn.proto .\object_detection\protos\faster_rcnn_box_coder.proto .\object_detection\protos\grid_anchor_generator.proto .\object_detection\protos\hyperparams.proto .\object_detection\protos\image_resizer.proto .\object_detection\protos\input_reader.proto .\object_detection\protos\keypoint_box_coder.proto .\object_detection\protos\losses.proto .\object_detection\protos\matcher.proto .\object_detection\protos\mean_stddev_box_coder.proto .\object_detection\protos\model.proto .\object_detection\protos\optimizer.proto .\object_detection\protos\pipeline.proto .\object_detection\protos\post_processing.proto .\object_detection\protos\preprocessor.proto .\object_detection\protos\region_similarity_calculator.proto .\object_detection\protos\square_box_coder.proto .\object_detection\protos\ssd.proto .\object_detection\protos\ssd_anchor_generator.proto .\object_detection\protos\string_int_label_map.proto .\object_detection\protos\train.proto
What was missing was: .\object_detection\protos\keypoint_box_coder.proto
Then ran:
python setup.py build
python setup.py install
Then changed:
nets to research.slim.net in faster_rcnn_inception_resnet_v2_feature_extractor.py and faster_rcnn_resnet_v1_feature_extractor.py (in the models folder)
Ran:
python object_detection/builders/model_builder_test.py
Got:
Ran 11 tests in 0.195s
OK
It looks like you have not run
export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim # (from tensorflow/models/research folder)
in the same terminal before launching the test, have you ? If you've chosen the .bashrc option, you probably need to reload your bshrc filewith . ~/.bashrc

Resources