Problems loading images from bytes in Flutter/Dart - python-3.x

I am building an application with Flutter. I am using using a mix of newer technologies that I am struggling to piece together.
My actual issue is that I cannot get images to load in the app when using the following methods:
Insert test data into MongoDB using Mongoengine, images are inserted into GridFS using this method.
Query GraphQL server for data and retrieve data and receive images from GridFS in form of bytes but in a string - e.g "b'/9j/4AAQSkZJRgABAQEASABIAAD/4V0X .... '"
Use that bytes string to load images in app with something like Image.memory()
But the images fail to load with an error: type String is not a subtype of type Uint8List. So I tried to convert the string of bytes from a String to raw bytes by doing:
List<int> bytesList = imageData['image']['data'].codeUnits;
Uint8List thumbImageBytes = Uint8List.fromList(bytesList);
I get the following exception:
I/flutter ( 4303): ══╡ EXCEPTION CAUGHT BY IMAGE RESOURCE SERVICE ╞══════
I/flutter ( 4303): The following _Exception was thrown while resolving an image:
I/flutter ( 4303): Exception: Could not instantiate image codec.
I have no idea what I can do to fix this, I cannot seem to find anything by googling etc. It would seem there is no information available for this exact scenario except for this S.O question which is what I have tried to do. I have also tried all the methods sugested in comments, followed the suggested links and tried all available combinations of answers, comments, everything.
My set up is as follows;
Main app: Flutter/Dart
API Server: Python/Flask/Mongoengine based GraphQL/Graphene API
Backend Database: MongoDB
The Python side;
A Mongoengine Document model:
class Product(Document):
meta = {'collection': 'product'}
name = StringField(unique=True)
price = FloatField()
sale_price = FloatField()
description = StringField()
image = FileField()
thumb = FileField()
created_at = DateTimeField(default=datetime.utcnow)
edited_at = DateTimeField()
# user = ReferenceField(User)
def __repr__(self):
return f'<Product Model::name: {self.name}>'
A Graphene schema for the model:
class ProductAttribute:
name = graphene.String()
price = graphene.Float()
sale_price = graphene.Float()
description = graphene.String()
image = Upload()
thumb = graphene.String()
created_at = graphene.DateTime()
edited_at = graphene.DateTime()
class Product(MongoengineObjectType):
"""Product node."""
class Meta:
model = ProductModel
interfaces = (graphene.relay.Node,)
class CreateProductInput(graphene.InputObjectType, ProductAttribute):
"""Arguments to create a product."""
pass
class CreateProduct(graphene.Mutation):
"""Create a product."""
product = graphene.Field(lambda: Product, description="Product created by this mutation.")
class Arguments:
input = CreateProductInput()
image = Upload(required=True)
def mutate(self, info, image, input):
data = utils.input_to_dictionary(input)
data['created_at'] = datetime.utcnow()
data['edited_at'] = datetime.utcnow()
print(data)
product = ProductModel(**data)
product.save()
return CreateProduct(product=product)
My base Graphene schema:
class Query(graphene.ObjectType):
"""Query objects for GraphQL API."""
node = graphene.relay.Node.Field()
single_product = graphene.relay.Node.Field(schema_product.Product)
all_products = MongoengineConnectionField(schema_product.Product)
class Mutations(graphene.ObjectType):
createProduct = schema_product.CreateProduct.Field()
schema = graphene.Schema(query=Query, types=[schema_product.Product], mutation=Mutations)

It's very suspicious that:
Your binary data is stored in a String (this is usually wrong).
Your String happens to be composed entirely of printable characters.
It's therefore likely that you're getting back binary data that's been encoded to a printable string. A common encoding is base64, and sure enough, when I tried to base64-encode a few different types of images, I see that base64-encoding JPEG files generates a string that starts off with /9j/4AAQSk, just like the string you get.
You are definitely getting back base64-encoded data. If you aren't doing the encoding yourself, then something is automatically doing the encoding for you, and likely there's a symmetric mechanism to decode it for you. If not, you'll need to explicitly base64-decode your String to get back binary data. You can use dart:convert to decode it for you.

Related

Can you make sure only one object related to another object has a certain field set?

I have a model called Video, and it has related objects on another model called Label. Example here:
class Video(models.Model):
pass
class Label(models.Model):
video = models.ForeignKey(Video, related_name="labels")
current = models.NullBooleanField()
I need to be able to find the current label on a video by doing something like my_video.labels.filter(current=True), and this query should only ever return one label, so only one label on the video should have that field set to True.
Is there a way of ensuring this on the model/db?
Thanks
EDIT: The answer given below has achieved exactly this. Adding some django tests below for anyone else reading as some proof:
class TestLabelIntegrity(TestCase):
def test_a_video_can_have_only_one_current_label(self):
video = Video.objects.create()
label_1 = Label.objects.create(
video=video,
current=True
)
with self.assertRaises(IntegrityError):
label_2 = Label.objects.create(
video=video,
current=True
)
def test_two_different_videos_can_each_have_current_layers(self):
""" No assertions needed, just need to make sure no integrity errors are raised"""
video_1 = Video.objects.create()
label_1 = Label.objects.create(
video=video_1,
current=True
)
video_2 = Video.objects.create()
label_2 = Label.objects.create(
video=video_2,
current=True
)
I believe you can solve this using UniqueConstraint.
Using this, you can restrict that a Video only have a single label that current == True
You can define the UniqueConstraint in the models Meta.
You’ll get a database integrity error on save() if the condition fails.
See the documentation for this here:
https://docs.djangoproject.com/en/4.0/ref/models/constraints/
class Label(models.Model):
...
class Meta:
constraints = [
models.UniqueConstraint(
fields=["current", "video"],
condition=Q(current=True),
name="unique_current_label",
),
]

Passing base64 .docx to docx.Document results in BadZipFile exception

I'm writing an Azure function in Python 3.9 that needs to accept a base64 string created from a known .docx file which will serve as a template. My code will decode the base64, pass it to a BytesIO instance, and pass that to docx.Document(). However, I'm receiving an exception BadZipFile: File is not a zip file.
Below is a slimmed down version of my code. It fails on document = Document(bytesIODoc). I'm beginning to think it's an encoding/decoding issue, but I don't know nearly enough about it to get to the solution.
from docx import Document
from io import BytesIO
import base64
var = {
'template': 'Some_base64_from_docx_file',
'data': {'some': 'data'}
}
run_stuff = ParseBody(body=var)
output = run_stuff.run()
class ParseBody():
def __init__(self, body):
self.template = str(body['template'])
self.contents = body['data']
def _decode_template(self):
b64Doc = base64.b64decode(self.template)
bytesIODoc = BytesIO(b64Doc)
document = Document(bytesIODoc)
def run(self):
self.document = self._decode_template()
I've also tried the following change to _decode_template and am getting the same exception. This is running base64.decodebytes() on the b64Doc object and passing that to BytesIO instead of directly passing b64Doc.
def _decode_template(self):
b64Doc = base64.b64decode(self.template)
bytesDoc = base64.decodebytes(b64Doc)
bytesIODoc = BytesIO(bytesDoc)
I have successfully tried the following on the same exact .docx file to be sure that this is possible. I can open the document in Python, base64 encode it, decode into bytes, pass that to a BytesIO instance, and pass that to docx.Document successfully.
file = r'WordTemplate.docx'
doc = open(file, 'rb').read()
b64Doc = base64.b64encode(doc)
bytesDoc = base64.decodebytes(b64Doc)
bytesIODoc= BytesIO(bytesDoc)
newDoc = Document(bytesIODoc)
I've tried countless other solutions to no avail that have lead me further away from a resolution. This is the closest I've gotten. Any help is greatly appreciated!
The answer to the question linked below actually helped me resolve my own issue. How to generate a DOCX in Python and save it in memory?
All I had to do was change document = Document(bytesIODoc) to the following:
document = Document()
document.save(bytesIODoc)

flickrAPI upload photos

I'm trying to use python Flickr API to upload photos to my Flickr account. I already got the API key and secret and user them to get information about my albums and photos, but I got some sort of errors trying to upload new photos. Here is my code:
import flickrapi
api_key = u'xxxxxxxxxxxxxxxxxxxxxxxx'
api_secret = u'xxxxxxxxxxxxxxxxxxxx'
flickr = flickrapi.FlickrAPI(api_key, api_secret)
filename = 'd:/downloads/_D4_6263-Enhanced.png'
title = 'Fach Halcones'
description = 'Posting image using API'
tags = 'fidae'+','+'aviation'+','+'extra'+','+'air shows'
flickr.upload(filename, title, description, tags)
When I run the script, I got the following error:
File "uploadPhotos.py", line 15, in module
flickr.upload(filename, title, description, tags)
TypeError: upload() takes from 2 to 4 positional arguments but 5 were given
looking at the Flickr API documentation, it seems to accept up to five arguments (filename,
fileobj, title, description, tags), and I'm passing only four, since fileobj is optional.
I have googled for some examples, but I was unable to find something that does the trick. So, any help would be awesome.
Regards,
Marcio
I found the solution, and I'm sharing it here. There were two issues with my code.
First: We must use kwargs; Second: tags must be separated by space, not commas
Here the final version:
import flickrapi
api_key = u'xxxxxxxxxxxxxxxxxxxxxxxx'
api_secret = u'xxxxxxxxxxxxxxxxxxxx'
flickr = flickrapi.FlickrAPI(api_key, api_secret)
params = {}
params['filename'] = 'd:/downloads/_D4_6263-Enhanced.png'
params['title'] = 'Fach Halcones'
params['description'] = 'Posting image using API'
params['tags'] = '''fidae aviation extra "air shows" '''
flickr.upload(**params)
That's it...

How to define the same field for load_only and dump_only params at the Marshmallow scheme?

I am trying to build a marshmallow scheme to both load and dump data. And I get everything OK except one field.
Problem description
(If you understand the problem, you don't have to read this).
For load data its type is Decimal. And I used it like this before. Now I want to use this schema for dumping and for that my flask API responses with: TypeError: Object of type Decimal is not JSON serializable. OK, I understand. I changed the type to Float. Then my legacy code started to get an exception while trying to save that field to database (it takes Decimal only). I don't want to change the legacy code so I looked for any solution at the marshmallow docs and found load_only and dump_only params. It seems like those are what I wanted, but here is my problem - I want to set them to the same field. So I just wondered if I can define both fields and tried this:
class PaymentSchema(Schema):
money = fields.Decimal(load_only=True)
money = fields.Float(dump_only=True)
I have been expected for a miracle, of course. Actually I was thinking that it will skip first definition (correctly, re-define it). What I got is an absence of the field at all.
Workaround solution
So I tried another solution. I created another schema for dump and inherit it from the former schema:
class PaymentSchema(Schema):
money = fields.Decimal(load_only=True)
class PaymentDumpSchema(PaymentSchema):
money = fields.Float(dump_only=True)
It works. But I wonder if there's some another, native, "marshmallow-way" solution for this. I have been looking through the docs but I can't find anything.
You can use the marshmallow decorator #pre_load in this decorator you can do whatever you want and return with your type
from marshmallow import pre_load
import like this and in this you will get your payload and change the type as per your requirement.
UPD: I found a good solution finally.
NEW SOLUTION
The trick is to define your field in load_fields and dump_fields inside __init__ method.
from marshmallow.fields import Integer, String, Raw
from marshmallow import Schema
class ItemDumpLoadSchema(Schema):
item = Raw()
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
if not (self.only and 'item' not in self.only) and \
not (self.exclude and 'item' in self.exclude):
self.load_fields['item'] = Integer(missing=0)
self.dump_fields['item'] = String()
Usage:
>>> ItemDumpLoadSchema().load({})
{'item': 0}
>>> ItemDumpLoadSchema().dump({'item': 0})
{'item': '0'}
Don't forget to define field in a schema with some field (Raw in my example) - otherwise it may raise an exception in some cases (e.g. using of only and exclude keywords).
OLD SOLUTION
A little perverted one. It based on #prashant-suthar answer. I named load field with suffix _load and implemented #pre_load, #post_load and error handling.
class ArticleSchema(Schema):
id = fields.String()
title = fields.String()
text = fields.String()
class FlowSchema(Schema):
article = fields.Nested(ArticleSchema, dump_only=True)
article_load = fields.Int(load_only=True)
#pre_load
def pre_load(self, data, *args, **kwargs):
if data.get('article'):
data['article_load'] = data.pop('article')
return data
#post_load
def post_load(self, data, *args, **kwargs):
if data.get('article_load'):
data['article'] = data.pop('article_load')
return data
def handle_error(self, exc, data, **kwargs):
if 'article_load' in exc.messages:
exc.messages['article'] = exc.messages.pop('article_load')
raise exc
Why the old solution is not a good solution?
It doesn't allow to inheritate schemas with different handle_error methods defined. And you have to name pre_load and post_load methods with different names.
pass data_key argument to the field definition
Documentation mentions, data_key parameter can be used along with dump_only or load_only to be able to have same field with different functionality.
So you can write your schema as...
class PaymentSchema(Schema):
decimal_money = fields.Decimal(data_key="money", load_only=True)
money = fields.Float(dump_only=True)
This should solve your problem. I am using data_key for similar problem in marshmallow with SQLAlchemyAutoSchema and this fixed my issue.
Edit
Note: The key in ValidationError.messages (error messages) will be decimal_money by default. You may tweak the handle_error method of Schema class to replace decimal_money with money but it is not recommended as you yourself may not be able to differentiate between the error messages fields.
Thanks.

Indexing HTML in Elasticsearch via python3

I am new to Elasticsearch. I have to index many HTML files via python3. I've seen many examples of adding info into Elasticsearch, but couldn't actually find anything appropriate for me. Can I index HTML files without extracting all their information in JSON format? I've seen some examples of indexing PDF to Elasticsearch via PHP using pipeline, but could not find something like this for python.
What do you mean by index HTML files to Elasticsearch? What kind of information do you want to send to Elasticsearch?
Yes its definitely possible, but give a bit more details of what you want to be sending to Elasticsearch. (full HTML pages, only the name, certain information from HTML files, etc)
Here a sample of a class that might be handy for you..
#ELK credentials
ELK_HOST = "[hostname]"
ELK_USER = "[elastic_user]"
ELK_PASSWORD= "[elastic_password]"
HEADERS = {
'host' : '[put hostname again if using redirects ;)]',
'Content-Type' : 'application/json',
}
class ElasticSearch():
def __init__(self,host,user,password):
self._host = host
self._user = user
self._password = password
self._auth = (self._user, self._password)
def update_index(self, index, data):
endpoint = str(index)+"/doc/"
uri = self._host +"/"+ endpoint
_data = data
_data = python_to_json(_data)
response = requests.post(uri, headers=HEADERS, auth=self._auth,data=_data)
es = ElasticSeach(ELK_HOST,ELK_USER,ELK_PASSWORD);
#some random data
data = {"test1": 1, "test2" : 2}
#update index (if doesnt exist, it will create a new one)
es.update_index("testindex",data)
hope this will help you!

Resources