EDIT: I believe this traceback stems from some sort of issue with the dependencies. Using pip to upgrade the packages didn't work but I created a new folder and installed from scratch and that worked out
I'm a Python novice so I'm struggling to debug an AWS Lambda I'm writing.
I've narrowed it down to this line of code s3_client = botoSession.resource('s3'), which is giving a long traceback with Syntax Error: invalid syntax. The botoSession variable is just for the credentials - botoSession = boto3.session.Session(aws_access_token, aws_secret_access_token).
I've also tried s3_client = boto3.client('s3'), s3_client = boto3.resource('s3'), s3_client = botoSession.resource('s3').
When I used botoSession.client('ses', region) I had no issues sending emails.
I found Error: client = boto3.client('s3') | AWS Elastic Beanstalk Worker Environment which appeared to be a similar issue, but it appeared to be fairly old and I wasn't able to figure out what the solution was. I tried adding
import sys
sys.path = [p for p in sys.path if not p.endswith('futures-3.0.3-py3.4.egg')]
to the top of my file which didn't seem to work.
The entire traceback is as follows:
Traceback (most recent call last):
File "smartsheetExporter.py", line 45, in <module>
s3_client = botoSession.resource('s3')
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/session.py", line 389, in resource
aws_session_token=aws_session_token, config=config)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/session.py", line 836, in create_client
client_config=config, api_version=api_version)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/client.py", line 65, in create_client
cls = self._create_client_class(service_name, service_model)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/client.py", line 90, in _create_client_class
base_classes=bases)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/utils.py", line 61, in _handler
module = import_module(module)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/utils.py", line 52, in import_module
__import__(name)
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/s3/inject.py", line 15, in <module>
from boto3.s3.transfer import create_transfer_manager
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/boto3/s3/transfer.py", line 127, in <module>
from s3transfer.exceptions import RetriesExceededError as \
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/s3transfer/__init__.py", line 134, in <module>
import concurrent.futures
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/concurrent/futures/__init__.py", line 8, in <module>
from concurrent.futures._base import (FIRST_COMPLETED,
File "/Users/nihar/LocalDocs/PythonPractice/Smartsheet-Emailer-Lambda/concurrent/futures/_base.py", line 381
raise exception_type, self._exception, self._traceback
^
SyntaxError: invalid syntax
Whenever strange things are happening, it's always a good idea to update things:
sudo pip install pip --upgrade
sudo pip install boto --upgrade
sudo pip install boto3 --upgrade
sudo pip install awscli --upgrade
If you're using Python 3, try pip3 instead of pip.
I just had this same issue with boto3 and ended up having to downgrade the Python version that my lambda was running from Python 3.6 to Python 2.7. If you're using Serverless Framework for this, your serverless.yml file looks like this.
provider:
name: aws
runtime: python3.6
memorySize: 3008
cool_function:
name: cool-function
description: This lambda goes and performs magic.
handler: cool_function.lambda_handler
runtime: python2.7
- schedule:
rate: rate(4 hours)
timeout: 180
I had the same problem with python 3.6 and AWS Lambda.
I found another answer that helped me here.
You should use futures==2.2.0
If you are running your code on an Amazon EC2 instance with a Role assigned to the instance, then you only need this:
import boto3
s3_client = boto3.client('s3')
s3_resource = boto3.resource('s3') # Pick whichever is wish to use
If you are not on an Amazon EC2 instance, this works:
import boto3
session = boto3.Session(aws_access_key_id='AKIAxxx',aws_secret_access_key='yyy')
s3_client = session.client('s3')
s3_resource = session.resource('s3')
Of course, you should never put your credentials in the code file. Instead, put them in a credentials file (easiest is via aws configure) or in Environment Variables. That way, they won't be copied into any code repository.
See: Boto3 credentials
Related
I tried a few different codes but they all seem to be giving me the same error.
import discord
client = discord.Client()
client.run("TheBotTokenzzzInQuotes")
And it's giving me this error
File "<ipython-input-1-1f31c2ad1160>", line 1, in <module>
runfile('C:/Users/Lenovo/Desktop/bot.py', wdir='C:/Users/Lenovo/Desktop')
File "C:\Users\Lenovo\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)
File "C:\Users\Lenovo\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/Lenovo/Desktop/bot.py", line 4, in <module>
client.run("token")
File "C:\Users\Lenovo\Anaconda3\lib\site-packages\discord\client.py", line 637, in run
_cleanup_loop(loop)
File "C:\Users\Lenovo\Anaconda3\lib\site-packages\discord\client.py", line 97, in _cleanup_loop
loop.close()
File "C:\Users\Lenovo\Anaconda3\lib\asyncio\selector_events.py", line 83, in close
raise RuntimeError("Cannot close a running event loop")
RuntimeError: Cannot close a running event loop
I have my own discord bot that I run, and the client variable for mine doesn't look anything like that if I'm being honest.
If you are not using "Discord.py Rewrite", that could be a possible reason for a failure.
Another possible reason(although very unlikely) is that you do not have aiohttp installed(which is the dependency asyncio needs for Discord.py Rewrite), you can install it by using the following command:
pip install aiohttp
If you would like an example bot to look at, I have listed one below that is directly from the developer of discord.py Rewrite(the version that is currently supported for python).
https://github.com/Rapptz/discord.py/blob/master/examples/basic_bot.py
If you were wondering what the beginning of my code looked like, I have it below this:
import discord
from discord.ext import commands
from discord.utils import get
client = commands.Bot(command_prefix = '>')
I'm running this script on vagrant based Linux system, It's a simple basic code for running flask app, I'm getting this error:
raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied
Here is the script:
#!/usr/bin python3
from flask import Flask
app = Flask(__name__)
#app.route('/')
#app.route('/hello')
def helloworld():
return 'Hello and Welcome to flask'
if __name__ == '__main__':
app.debug = True
app.run(host='0.0.0.0', port=5000)
And the error I got:
vagrant#vagrant:/vagrant$ python3 project.py
* Serving Flask app "project" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
Traceback (most recent call last):
File "project.py", line 14, in <module>
app.run(host='0.0.0.0', port=5000)
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 943, in run run_simple(host, port, self, **options)
File "/usr/local/lib/python3.5/dist-packages/werkzeug/serving.py", line 988, in run_simple
run_with_reloader(inner, extra_files, reloader_interval, reloader_type)
File "/usr/local/lib/python3.5/dist-packages/werkzeug/_reloader.py", line 332, in run_with_reloader
sys.exit(reloader.restart_with_reloader())
File "/usr/local/lib/python3.5/dist-packages/werkzeug/_reloader.py", line 176, in restart_with_reloader
exit_code = subprocess.call(args, env=new_environ, close_fds=False)
File "/usr/lib/python3.5/subprocess.py", line 557, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
PermissionError: [Errno 13] Permission denied
I had the same problem, it turns out that the port needs to be greater than 5000:
from flask import Flask, request, abort, jsonify
app = Flask(__name__)
#app.route('/getSquare', methods=['POST'])
def get_square():
pass
return jsonify({'answer': num ** 2})
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5001, debug=True) # make sure port is > 5000
Maybe it's a permission error since Vagrant is trying to run your flask application and doesn't have the necessary executable permission?
I say this because of this part of your stack trace:
PermissionError: [Errno 13] Permission denied
and appears to be failing on this line of code: app.run(host='0.0.0.0', port=5000)
Perhaps you could give executable permissions on your directory containing your flask project and see if that helps resolve your issue:
chmod -R 755 /path/to/directory
Hopefully that helps!
Since you're playing in a VM that you built using Vagrant, you must have done something like
vagrant#vagrant:/vagrant$ sudo pip3 install flask
first. Possibly you installed some other packages. Depending on what base OS you're using, using pip (or pip3) to install at the system level (even in a VM) can cause problems.
After wrestling with my base install getting mangled a few time, I now always use virtualenv inside of VMs so that all of the package installs are local.
I'll bet that
vagrant#vagrant:/vagrant$ sudo apt-get install -y python-virtualenv
vagrant#vagrant:/vagrant$ virtualenv --python=python3 venv
vagrant#vagrant:/vagrant$ venv/bin/pip install flask
vagrant#vagrant:/vagrant$ ven/bin/python project.py
will give you a better result.
So, here's a file I made (flaskblog.py):
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "<h1>Home Page</h1>"
Here's how I first ran it:
$ export FLASK_APP=flaskblog.py
$ flask run
Here's how I ran it in debug mode:
$ export FLASK_APP=flaskblog.py
$ export FLASK_DEBUG=1
$ flask run
Now I want to run the application directly using python. I first updated the .py file:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "<h1>Home Page</h1>"
if __name__ == "__main__":
app.run()
This is the command I used to run the python file:
$ python3 flaskblog.py
It worked fine. Now I want to run the application in debug mode. So, I updated the file:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "<h1>Home Page</h1>"
if __name__ == "__main__":
app.run(debug=True) #Added ("debug=True") here
Command used to run the file:
$ python3 flaskblog.py
Here's the error:
* Serving Flask app "flaskblog" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
Traceback (most recent call last):
File "flaskblog.py", line 9, in <module>
app.run(debug=True)
File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 943, in run
run_simple(host, port, self, **options)
File "/usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 988, in run_simple
run_with_reloader(inner, extra_files, reloader_interval, reloader_type)
File "/usr/local/lib/python3.6/dist-packages/werkzeug/_reloader.py", line 332, in run_with_reloader
sys.exit(reloader.restart_with_reloader())
File "/usr/local/lib/python3.6/dist-packages/werkzeug/_reloader.py", line 176, in restart_with_reloader
exit_code = subprocess.call(args, env=new_environ, close_fds=False)
File "/usr/lib/python3.6/subprocess.py", line 267, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/XXX/XXX/XXX/XXX/XXX/XXX/XXX/XXX/Flask_Blog/flaskblog.py'
I just used "XXX" instead of the actual directories. Any help will be appreciated!
PS: All the code is from this video: https://www.youtube.com/watch?v=MwZwr5Tvyxo&list=PL-osiE80TeTs4UjLw5MM6OjgkjFeUxCYH
It looks like Flask is trying to run ./flaskblog.py directly for some reason, rather than with the python binary (python3 flaskblog.py), which is not working since flaskblog.py isn't executable.
So just add the following line (shebang) at the top of flaskblog.py
#!/usr/bin/env python3
...and make the file executable:
chmod +x flaskblog.py
Then try again, either with python3 flaskblog.py or directly as ./flaskblog.py.
I simply changed the permissions on the .py file from 775 to 664, essentially removing the 'x' from the permissions.
Changed it from:
-rwxrwxr-x 1 ubuntu ubuntu
to:
-rw-rw-r-- 1 ubuntu ubuntu
I have a AWS EC2 instance containing a CentOS (atleast that's what I think) and I followed this link(How to yum install Node.JS on Amazon Linux) to install nodejs on my instance (the second answer).The thing is I am able to install it successfuly and even able to run the following commands
node --version
v8.11.1
npm --version
5.6.0
node -e "console.log('Running Node.js ' + process.version)"
Running Node.js v8.11.1
But I have a python file that runs a node command through python's
subprocess module.This the python file below
import json
import yaml
import subprocess
def output_filter(json_data, exp):
json_data = json.dumps(json_data)
result = subprocess.check_output(['node', '/home/ec2-user/api-demo/jstest3.js', json_data, exp])
dict = yaml.load(result.decode("utf-8"))
filtered_output = json.dumps(dict)
return json.loads(filtered_output)
Below is my jstest3.js file
var args = process.argv.slice(2);
//console.log(args[0]);
var jsonata = require("jsonata");
var data = JSON.parse(args[0]);
var expression = jsonata(args[1]);
var result = expression.evaluate(data);
console.log(result);
I basically uses a nodejs library called jsonata to do some JSON parsing.I am able to use execute this python file in my local machine (I have an Ubuntu 16.10 x86_64) without any issues.But when I execute this in my aws instance, this is the error I get
result = subprocess.check_output(['node', '/home/ec2-user/api-demo/jstest3.js', json_data, exp])
File "/usr/lib64/python3.5/subprocess.py", line 316, in check_output
**kwargs).stdout
File "/usr/lib64/python3.5/subprocess.py", line 383, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib64/python3.5/subprocess.py", line 676, in __init__
restore_signals, start_new_session)
File "/usr/lib64/python3.5/subprocess.py", line 1289, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'node'
Why am I getting this error despite node being installed in my aws instance as shown above?What am I doing wrong?
Ah! For some reason the AWS instance couldn't keep up with all the installations I did.So once I restarted the instance by going to the AWS console -> restart instance, all the errors were suddenly gone and it started processing my nodejs requests.
Environment:
Python 3.2.3 (using virtualenv)
Pyramid 1.4
pyramid_mongodb scaffold
After installing myproject using pyramid_mongodb scaffold I ran python setup.py test -q and it's failing with below errors.
running build_ext
Traceback (most recent call last):
File "setup.py", line 33, in <module>
""",
File "/usr/lib/python3.2/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.2/distutils/dist.py", line 917, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.2/distutils/dist.py", line 936, in run_command
cmd_obj.run()
File "/root/App/Big3/lib/python3.2/site-packages/distribute-0.6.24-py3.2.egg/setuptools /command/test.py", line 137, in run
self.with_project_on_sys_path(self.run_tests)
File "/root/App/Big3/lib/python3.2/site-packages/distribute-0.6.24-py3.2.egg/setuptools /command/test.py", line 117, in with_project_on_sys_path
func()
File "/root/App/Big3/lib/python3.2/site-packages/distribute-0.6.24-py3.2.egg/setuptools /command/test.py", line 146, in run_tests
testLoader = loader_class()
File "/usr/lib/python3.2/unittest/main.py", line 123, in __init__
self.parseArgs(argv)
File "/usr/lib/python3.2/unittest/main.py", line 191, in parseArgs
self.createTests()
File "/usr/lib/python3.2/unittest/main.py", line 198, in createTests
self.module)
File "/usr/lib/python3.2/unittest/loader.py", line 132, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python3.2/unittest/loader.py", line 132, in <listcomp>
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/python3.2/unittest/loader.py", line 91, in loadTestsFromName
module = __import__('.'.join(parts_copy))
File "/root/App/Big3/Lime/lime/__init__.py", line 1, in <module>
from pyramid.config import Configurator
File "/root/App/Big3/lib/python3.2/site-packages/pyramid-1.4.1-py3.2.egg/pyramid/config /__init__.py", line 10, in <module>
from webob.exc import WSGIHTTPException as WebobWSGIHTTPException
File "/root/App/Big3/lib/python3.2/site-packages/WebOb-1.2.3-py3.2.egg/webob/exc.py", line 1115, in <module>
from paste import httpexceptions
File "/root/App/Big3/lib/python3.2/site-packages/Paste-1.7.5.1-py3.2.egg/paste /httpexceptions.py", line 634
except HTTPException, exc:
^
SyntaxError: invalid syntax
I understand the error, that Paste is not python3 compatible. I also know how to fix it but that would essentially mean porting Paste to python3 (which is something I don't want to do), so can anyone tell what I can do?
From the error stack I see that webob/exc.py is doing from paste import httpexceptions but when I checked the code I see that the import is under a try except block (without raising any error in except), so I even tried the test after removing paste from the lib but then when I run the test, I see that the setup.py is installing paste again
running test
Checking .pth file support in .
/root/App/Big3/bin/python -E -c pass
Searching for Paste>=1.7.1
I checked .pth files and removed reference to paste and then started re-installation of project but somehow it still sees paste as required
Installed /root/App/Big3/Myproject
Processing dependencies for Myproject==0.0
Searching for Paste>=1.7.1
Reading http://pypi.python.org/simple/Paste/
My setup.py file is same as this
Can someone tell me where is this paste dependency coming into my project.
I didn't intend to answer my own question but since I have made changes which are working for me, I thought I will share it here (assuming that there would be other folks wanting to have pyramid_mongodb scaffold work on python3)
Changes in development. ini
Removed
[pipeline:main]
pipeline =
egg:WebError#evalerror
{{project}}
Changed
[app:{{project}}] to [app:main]
Added (optional)
pyramid.includes =
pyramid_debugtoolbar
Changed server (from paste to waitress)
[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543
Changes in Setup.py
changed requires from
requires = ['pyramid', 'WebError', 'pymongo']
to
requires = ['pyramid', 'pyramid_debugtoolbar', 'pymongo', 'uwsgi', 'waitress']
It's important to remove webError
The application is now working...