Hello,
I am using CherryPy to host the gui of an application that takes json files from qualtrics and drops them in a mysql server.
The code seems to work for most surveys but for some I get the following error:
Traceback (most recent call last):
File "C:\Users\jam66\AppData\Local\Programs\Python\Python37-32\lib\site-packages\cherrypy\_cprequest.py", line 627, in respond
self._do_respond(path_info)
File "C:\Users\jam66\AppData\Local\Programs\Python\Python37-32\lib\site-packages\cherrypy\_cprequest.py", line 686, in _do_respond
response.body = self.handler()
File "C:\Users\jam66\AppData\Local\Programs\Python\Python37-32\lib\site- packages\cherrypy\lib\encoding.py", line 264,
in __call__ct.params['charset'] = self.find_acceptable_charset()
File "C:\Users\jam66\AppData\Local\Programs\Python\Python37-32\lib\site- packages\cherrypy\lib\encoding.py", line 173, in find_acceptable_charset
if encoder(self.default_encoding):
File "C:\Users\jam66\AppData\Local\Programs\Python\Python37-32\lib\site-packages\cherrypy\lib\encoding.py", line 114, in encode_string
for chunk in self.body:
TypeError: 'bool' object is not iterable
Any help on beginning to understand this issues is appreciated
My guess is that some of your exposed methods are returning a boolean. You have to return a string or an iterable. Unless you are using the json tool, in that case the dictionary to string is handled by the tool.
As a way to debug it, just print or log the value that would will be returned, verify the type with the type function.
Related
Im working with a django project(im pretty new to django) and running into an issue passing a model object between my view and a celery task.
I am taking input from a form which contains several ModelChoiceField fields and using the selected object in a celery task. When I queue the task(from the post method in the view) using someTask.delay(x, y, z) where x, y and z are various objects from the form ModelChoiceFields I get the error object of type <someModelName> is not JSON serializable.
That said, if I create a simple test function and pass any of the same objects from the form into the function I get the expected behavior and the name of the object selected in the form is logged.
def test(object):
logger.debug(object.name)
I have done some poking based on the above error and found django serializers which allows for a workaround by serializing the object using serializers.serialize('json', [template]), in the view before passing it to the celery task.
I can then access the object in the celery task by using template = json.loads(template)[0].get('fields') to access its required bits as a dictionary -- while this works, it does seem a bit inelegant and I wanted to see if there is something I am missing here.
Im obviously open to any feedback/guidance here however my main questions are:
Why do I get the object...is not JSON serializable error when passing a model object into a celery task but not when passing to my simple test function?
Is the approach using django serializers before queueing the celery task considered acceptable/correct or is there a cleaner way to achieve this goal?
Any suggestions would be greatly appreciated.
Traceback:
I tried to post the full traceback here as well however including that caused the post to get flagged as 'this looks like spam'
Internal Server Error: /build/
Traceback (most recent call last):
File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/serialization.py", line 49, in _reraise_errors
yield
File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/serialization.py", line 220, in dumps
payload = encoder(data)
File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 65, in dumps
return _dumps(s, cls=cls or _default_encoder,
File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/kombu/utils/json.py", line 55, in default
return super().default(o)
File "/usr/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Template is not JSON serializable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/tech/sandbox_project/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
response = get_response(request)
Add this lines to settings.py
# Project/settings.py
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
Then instead of passing object, send JSON with id/pk if you're using a model instance call the task like this..
test.delay({'pk': 1})
Django model instance is not available in celery environment, as it runs in a different process
How you can get the model instance inside task then? Well, you can do something like below -
def import_django_instance():
"""
Makes django environment available
to tasks!!
"""
import django
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'Project.settings')
django.setup()
# task
#shared_task(name="simple_task")
def simple_task(data):
import_django_instance()
from app.models import AppModel
pk = data.get('pk')
instance = AppModel.objects.get(pk=pk)
# your operation
I have a problem getting an object from the mongodb instance. If I search for this object with pymongo interface, everything is fine - object can be found. If try to do the very same thing with pymodm - it fails with error.
Here is what I'm doing:
from pymodm import connect, MongoModel, fields
from pymongo import MongoClient
class detection_object(MongoModel):
legacy_id = fields.IntegerField()
client = MongoClient(MONGODB_URI)
db = client[MONGODB_DEFAULT_SCHEME]
collection = db['detection_object']
do = collection.find_one({'legacy_id': 1437424})
print(do)
connect(MONGODB_URI)
do = detection_object.objects.raw({'legacy_id': 1437424}).first()
print(do)
The first print outputs this: {'_id': ObjectId('5c4099dcffa4fb11494d983d'), 'legacy_id': 1437424}. However, during the execution of this command: do = detection_object.objects.raw({'legacy_id': 1437424}).first() interpreter fails with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/pymodm/queryset.py", line 127, in first
return next(iter(self.limit(-1)))
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/konsof01/PycharmProjects/testthisfuckingshit/settings.py", line 29, in <module>
do = detection_object.objects.raw({'legacy_id': 1437424}).first()
File "/usr/local/lib/python3.7/site-packages/pymodm/queryset.py", line 129, in first
raise self._model.DoesNotExist()
__main__.DoesNotExist
How can this be? I'm trying to query the very same object, with the same connection and collection. Any ideas, please?
you could try it as follows:
detection_object.objects.raw({'legacy_id': "1437424"} ).first()
probably the legacy_id is stored as string.
Othewise, make sure the db name is present at the end of the MONGO_URI as it is underlined in the docs.
Each document in your 'detection_object' collection requires to have '_cls' attribute. The string value stored in this attribute should be
__main__.classname
(class name according to your code is detection_object).
For example a document in your database needs to look like this:
{'_id': ObjectId('5c4099dcffa4fb11494d983d'), 'legacy_id': 1437424, '_cls': '__ main __.detection_object'}
Writing an analytics script for facebook chats with python. The library I'm using (fbchat) sometimes returns None type objects in place of messages. I thought I could catch this with a simple
if message is None:
continue
in my loop through my message list. However, I'm still getting exceptions due to the object being None type. What am I doing wrong here?
Code snippet:
for message in messages:
#sometimes the fbchat library doesn't return a message
if message is None:
continue
#strip the line endings to not mess up the csv
message.text.replace('\n', ' ')
Exact exception:
Traceback (most recent call last):
File "collect.py", line 58, in <module>
message.text.replace('\n', ' ')
AttributeError: 'NoneType' object has no attribute 'replace'
Checking for message.text is None resolves the issue and is what I should have done in the first place.
I want to let Scrapy crawl local html files but am stuck because the header lacks the Content-type field. I've followed the tutorial here: Use Scrapy to crawl local XML file - Start URL local file address So basically, I am pointing scrapy to local urls, such as file:///Users/felix/myfile.html
However, scrapy will crash then, since it looks like (on MacOS) the resulting response object does not contain the required field Content-type.
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 /Users/felix/IdeaProjects/news-please/newsplease/__init__.py
[scrapy.core.scraper:158|ERROR] Spider error processing <GET file:///Users/felix/IdeaProjects/news-please/newsplease/0a2199bdcef84d2bb2f920cf042c5919> (referer: None)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scrapy/utils/defer.py", line 102, in iter_errback
yield next(it)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
return (_set_referer(r) for r in result or ())
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "/Users/felix/IdeaProjects/news-please/newsplease/crawler/spiders/download_crawler.py", line 33, in parse
if not self.helper.parse_crawler.content_type(response):
File "/Users/felix/IdeaProjects/news-please/newsplease/helper_classes/parse_crawler.py", line 116, in content_type
if not re.match('text/html', response.headers.get('Content-Type').decode('utf-8')):
AttributeError: 'NoneType' object has no attribute 'decode'
Someone suggested to run a simple http server, see Python Scrapy on offline (local) data but that is not an option, mainly because of the overhead caused by running another server.
I need to use scrapy in the first place, as we have a larger framework that uses scrapy. We plan to add the functionality to crawl from local files to that framework. However, since there are several questions on SO on how to crawl from local files (see previous links), I assume this problem is of general interest.
You can actually fork news-please or change scrapy to always return True in the function def content_type(self, response) in newsplease/helper_classes/parse_crawler.py if it is from local storage.
The new file will look like this:
def content_type(self, response):
"""
Ensures the response is of type
:param obj response: The scrapy response
:return bool: Determines wether the response is of the correct type
"""
if response.url.startswith('file:///'):
return True
if not re.match('text/html', response.headers.get('Content-Type').decode('utf-8')):
self.log.warn("Dropped: %s's content is not of type "
"text/html but %s", response.url,
response.headers.get('Content-Type'))
return False
else:
return True
I'm trying to download one message using the GMail API. Below is my traceback:
pdiracdelta#pdiracdelta-Laptop:~/GMail Metadata$ ./main.py
<oauth2client.client.OAuth2Credentials object at 0x7fd6306c4d30>
False
Traceback (most recent call last):
File "./main.py", line 105, in <module>
main()
File "./main.py", line 88, in main
service = discovery.build('gmail', 'v1', http=http)
File "/usr/lib/python3/dist-packages/oauth2client/util.py", line 137, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/lib/python3/dist-packages/googleapiclient/discovery.py", line 197, in build
resp, content = http.request(requested_url)
File "/usr/lib/python3/dist-packages/oauth2client/client.py", line 562, in new_request
redirections, connection_type)
File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1138, in request
headers = self._normalize_headers(headers)
File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1106, in _normalize_headers
return _normalize_headers(headers)
File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 194, in _normalize_headers
return dict([ (key.lower(), NORMALIZE_SPACE.sub(value, ' ').strip()) for (key, value) in headers.items()])
File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 194, in <listcomp>
return dict([ (key.lower(), NORMALIZE_SPACE.sub(value, ' ').strip()) for (key, value) in headers.items()])
TypeError: sequence item 0: expected str instance, bytes found
And below is a snippet of code which produces the credential object and boolean print just before the Traceback. It confirms that the credentials object is valid and is being used as suggested by Google:
credentials = get_credentials()
print(credentials)
print(str(credentials.invalid))
http = credentials.authorize(httplib2.Http())
service = discovery.build('gmail', 'v1', http=http)
What is going wrong here? It seems to me that I am not at fault, since the problem can be traced back to service = discovery.build('gmail', 'v1', http=http) which uses nothing but valid information (implying one of the packages used further in the stack cannot handle this valid information). Is this a bug, or am I doing something wrong?
UPDATE: it seems that the _normalize_headers function has now been patched. Updating your python version should fix the problem (I'm using 3.6.7 now).
Solved with help from Padraic Cunningham, who identified the problem as an encoding issue. I solved this problem by applying .decode('utf-8') to the header keys and values (headers is a dict) if they are bytes-type objects (which are apparently UTF-8 encoded) and transforming them into python3 strings. This is probably due to some python2/3 mixing in the google API.
The fix also includes changing all code from google API examples to python3 code (e.g. exception handling) but most importantly my workaround involves editing /usr/lib/python3/dist-packages/httplib2/__init__.py at lines 193-194, redefining the _normalize_headers(headers) function as:
def _normalize_headers(headers):
for key in headers:
# if not encoded as a string, it is ASSUMED to be encoded as UTF-8, as it used to be in python2.
if not isinstance(key, str):
newkey = key.decode('utf-8')
headers[newkey] = headers[key]
del headers[key]
key = newkey
if not isinstance(headers[key], str):
headers[key] = headers[key].decode('utf-8')
return dict([ (key.lower(), NORMALIZE_SPACE.sub(value, ' ').strip()) for (key, value) in headers.items()])
WARNING: this workaround is obviously quite dirty as it involves editing files from the httplib2 package. If someone finds a better fix, please post it here.