Run scrapyd in Python 3.6 - python-3.x

I've been looking around and I can't seem to find an answer on how to run scrapyd in Python 3 and above. When I run it it keeps defaulting to python 2.7, though I recall reading in the docs or elsewhere that scrapyd supports python3:
https://scrapyd.readthedocs.io/en/stable/news.html#id1
Running on an AWS Ubuntu 18.04 server. What am I doing wrong? How can I change to Python 3.4 and above?
I'm having problems because I'm using abstract inheritance in my spiders which python 2.7 doesn't support.
Edit: I'm able to run Scrapyd in Python 3 using a virtual environment, but how would you do it without one?

Simplest solution is use a virtual enviroment.
Since you are using Ubuntu and you can make it work using a venv I assume you installed scrapy using pip instead of pip3. Since Py2.7 is the stardard reference to python in Ubuntu, when running pip it will execute pip for py2 instead of py3.
Another way would be uninstalling pip uninstall scrapy and installing it again using pip3 pip3 install scrapy. This should fix the reference to scrapy. If by any chance you also use scrapy in py2.7 and therefore can't uninstall, then you have another reason to use venv.

Related

Do I have different python versions and which one I am working with?

I am confused about different python versions on my Computer and in Anaconda.
First: I use an iMac (OSX Catalina).I sometimes use the command line (Terminal on my iMac) directly and sometimes I use Anaconda Navigator (for example when I want to work with Jupyter Notebooks).
When I type python3 --version, I get the answer that I have python 3.8.0 installed. But recently, I installed the latest pip version with pip3 --user --upgrade pip. The answer I got was:
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 20.0.2 from /Users/Andrea_5K/Library/Python/3.7/lib/python/site-packages/pip (python 3.7)
I wondered why in this message python 3.7 is referenced.
When I check my Library/Python folder in my user folder I see that I have a 3.7 folder and a 3.8 folder as well. Is there something wrong? Is pip 20.0.2 now installed on python 3.7 although I use python 3.8? And what does the WARNING message mean?

virtualenv couldn't find python interpreter path?

my computer have python3.4, python3.5, python3.6, python3.7, but now i want to use python virtualenv to create a env , but it show path not found?
The image in here:
You need to provide the full path to Python, for example:
virtualenv.exe -p C:\Python37\python.exe
For git-bash the command should be slightly different:
virtualenv.exe -p /c/Python37/python.exe
First of all having multiple versions of Python is not advised especially if you cant manage them. The issue you are getting stems from the installation of the Virtualenv package. You need to use the python version that has Virtualenv in the site-packages. Examine all the site packages for the 3 versions then use the one that has virtualenv alternatively you can uninstall all the 3 versions and reinstall one I'd advise on Python 3.6 then install virtualenv via pip and try again.

How to install stuff like Requests and BeautifulSoup to use in Python?

I am an extreme beginner with Python and its libraries and installation in general. I want to make an extremely simple google search web scraping tool. I was told to use Requests and BeautifulSoup. I have installed python3 on my Mac by using brew install python3 and I am wondering how to get those two libraries
I googled around and many results said that by doing brew install python3 it will automatically install pip so I can use something like pip install requests but it says pip: command not found.
by running python3 --version it says Python 3.7.4
Since you're running with Python3, not Python (which usually refers to 2.7), you should try using pip3.
pip on the other hand, is the package installer for Python, not Python3.

Why virtualenv shows all packages installed and do not install modules in virtualenv?

Question
I am not sure why when virtualenv active, pip freeze stills showing all modules when it suppose NOT to. What am I doing wrong?
On the terminal
I tried two ways to create a virtual env:
virtualenv my-virtualenv
virtualenv --no-site-packages my-virtualenv
then activate it
source my-virtualenv/bin/activate
Results
Both show all packages (when they suppose not to). Running command pip freeze I get:
(my-virtualenv)$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
actionlib==1.11.9
angles==1.9.11
bondpy==1.7.19
camera-calibration==1.12.20
camera-calibration-parsers==1.11.12
catkin==0.7.6
cv-bridge==1.12.4
diagnostic-analysis==1.9.2
diagnostic-common-diagnostics==1.9.2
diagnostic-updater==1.9.2
dynamic-reconfigure==1.5.48
gazebo-plugins==2.5.13
gazebo-ros==2.5.13
...
rosnode==1.12.7
rosparam==1.12.7
rospy==1.12.7
rosservice==1.12.7
rostest==1.12.7
rostopic==1.12.7
rosunit==1.13.5
roswtf==1.12.7
tf2-ros==0.5.16
topic-tools==1.12.7
xacro==1.11.2
Running
(my-virtualenv)$ which python
/home/user/.../my-virtualenv/bin/python
(my-virtualenv)$ python -V
Python 2.7.12
and
(my-virtualenv)$ which python3
/usr/bin/python3
(my-virtualenv)$ python3 -V
Python 3.5.2
Hence, I even can use python3 when virtualenv is activated but it uses the pc installed module. Moreover, new installed packages are install in the pc and not the virtualenv and I can't install python3 in the vitualenv because it "exists" already (but in the pc).
I was having the same issue and it was because, somehow, Python configuration of ROS was generating it.
I solved it removing the source commands of ROS from the ~/.bashrc. E.g.:
source /opt/ros/melodic/setup.bash
source $HOME/ROS/aslam_ws/devel/setup.bash
By the way, I realised that, before changing anything, creating the environment from PyCharm works properly too (when in the terminal I still had the issue).
There must be a better solution but I haven't found it yet.

Does TensorFlow 1.9 support Python 3.7

I'm hesitating whether to downgrade to Python 3.6 or install a new version of TensorFlow.
Does TensorFlow 1.9 support Python 3.7?
I was able to install Tensorflow 1.12.0 with Python 3.7 on MacOS, with the following command.
sudo python3 -m pip install --upgrade https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.12.0-py3-none-any.whl
Not yet. It seems there are some variables named "async", which has become a keyword in 3.7. Shouldn't be too difficult to fix, but still a problem.
Source: https://github.com/tensorflow/tensorflow/issues/20444
Probably not yet.
First of all, you will probably get a SyntaxError: invalid syntax because some parameters and variables at the pywrap_tensorflow_internal.py module have the name async which is a reserved keyword in Python 3.7.0 (see also this post).
However, you can solve this by simply changing the name of all these (Ctrl + R and Replace All) from async to for example async1.
The problem is that then you will probably get a ImportError: No module named '_pywrap_tensorflow_internal' which perhaps may be fixed for any previous version before Python 3.7.0 as this post on StackOverflow suggests but personally I could not fix it on my laptop with Python 3.7.0 .
Consequently, I simply downgraded to Python 3.6.6 and now Tensorflow is working fine.
There is a whl built here. However in my run it was cpu version.
https://www.lfd.uci.edu/~gohlke/pythonlibs/
as of Aug 26tH, there is an update on the tensorflow github site
https://github.com/tensorflow/tensorflow/pull/21202
Tensorflow 1.13.1 now supports Python 3.7.
According to the comments on github
see
https://github.com/tensorflow/tensorflow/issues/20517#issuecomment-457185528
by member av8ramit
"We are aiming to try and have Windows and Ubuntu python binaries by rc2 or the official."
the estimated release date for 2019-02-03
however on
https://github.com/winpython/winpython/issues/687
one can read from a winpython contributor:
"Tensorflow-2 is 2019, Tensorflow-1 on Python-3.7 is still not in sight"
Finally Worked for me!
On Python 3.6.4 (even 3.6.5 failed for me)
Install Python3.6.4 from here:
https://www.python.org/downloads/mac-osx/
If you're facing issues with Python pkgs that you already had on you older version just revert it using this post:
python location on mac osx
I found this to work after searching for a while.
As Tensorflow only supports Python 3.6 as of now, you can install a different version of python alongside your standard one. Here are the steps I followed:
Download the Python3.6 tgz file from the official website (eg. Python-3.6.6.tgz)
Unpack it with tar -xvzf Python-3.6.6.tgz
cd Python-3.6.6
run ./configure
run make altinstall to install it (install vs altinstall explanation here Difference in details between "make install" and "make altinstall")
You'll normally find your new python install under /usr/local/bin. Now you can create a new virtualenv specifying the python version with:
virtualenv --python=python3.6 env3.6
Get into the virtualenv running the command source env3.6/source/bin/activate.
Install tensorflow with the classic pip install tensorflow
Profit
First of all,
*conda create -n yourenvname python=3.6
*conda activate yourenvname
*pip install tensorflow
then it will work like rock.

Resources