I'm writing an application based on GCP services and I need to access to an external project. I stored on my Firestore database the authentication file's informations of the other project I need to access to. I read this documentation and I tried to apply it but my code does not work. As the documentaion says, what I pass to the authentication method is a dictionary[str, str].
This is my code:
from googleapiclient import discovery
from google.oauth2 import service_account
from google.cloud import firestore
project_id = body['project_id']
user = body['user']
snap_id = body['snapshot_id']
debuggee_id = body['debuggee_id']
db = firestore.Client()
ref = db.collection(u'users').document(user).collection(u'projects').document(project_id)
if ref.get().exists:
service_account_info = ref.get().to_dict()
else:
return None, 411
credentials = service_account.Credentials.from_service_account_info(
service_account_info,
scopes=['https://www.googleapis.com/auth/cloud-platform'])
service = discovery.build('clouddebugger', 'v2', credentials=credentials)
body is just a dictionary containing all the informations of the other project. What I can't understand is why this doesn't work and instead using the method from_service_account_file it works.
The following code will give to that method the same informations of the previous code, but inside a json file instead of a dictionary. Maybe the order of the elements is different, but I think that it doesn't matter at all.
credentials = service_account.Credentials.from_service_account_file(
[PATH_TO_PROJECT_KEY_FILE],
scopes=['https://www.googleapis.com/auth/cloud-platform'])
Can you tell me what I'm doing wrong with the method from_service_account_info?
Problem solved. When I posted the question I manually inserted from the GCP Firestore Console all the info about the other project. Then I wrote the code to make it authomatically and it worked. Honestly I don't know why it didn't worked before, the informations put inside Firestore were the same and the format as well.
Related
I used to work with IBM's Natural Language Understanding API for analyzing URLs. I am using Python's IBM Watson SDK 5.1 on Python 3.8.
I successfully used the code below [all approprioate options have been imported] to extract metadata, in addition to entities, concepts, etc:
def NLU_analysis(url):
try:
response = natural_language_understanding.analyze(
url=url, return_analyzed_text=True, clean=True, language=True,
features=Features(keywords=KeywordsOptions(limit=10),
entities=EntitiesOptions(limit=10),
concepts=ConceptsOptions(limit=5),
metadata=MetadataOptions(),
categories=CategoriesOptions(limit=5))).get_result()
return response
except:
pass
The code above used to return the metadata. Now, in Python SDK 5.1.0, IBM recently changed to way to retrieve the URL's metadata. The "MetadataOptions" feature has been replaced by "FeatureMetadataResults". If I use the code above and replace the MetadataOptions by FeatureMetadataResults as shown below:
def NLU_analysis(url):
try:
response = natural_language_understanding.analyze(
url=url, return_analyzed_text=True, clean=True, language=True,
features=Features(keywords=KeywordsOptions(limit=10),
entities=EntitiesOptions(limit=10),
concepts=ConceptsOptions(limit=5),
metadata=FeaturesResultsMetadata(),
categories=CategoriesOptions(limit=5))).get_result()
return response
except:
pass
Now, if I run the modified code, I get the following error message:
"TypeError: Object of type FeaturesResultsMetadata is not JSON serializable"
If I read IBM's documentation, I am getting somewhat confused (Link to the API documentation. Here's the IBM's code example:
import json
from ibm_watson import NaturalLanguageUnderstandingV1
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
from ibm_watson.natural_language_understanding_v1
import Features, MetadataOptions
authenticator = IAMAuthenticator('{apikey}')
natural_language_understanding = NaturalLanguageUnderstandingV1(
version='2020-08-01',
authenticator=authenticator
)
natural_language_understanding.set_service_url('{url}')
response = natural_language_understanding.analyze(
url='www.ibm.com',
features=Features(metadata=MetadataOptions())).get_result()
Does anyone know whether it is still possible to retrieve an URL's metadata using IBM Watson's Natural Language Understanding API?
Have a nice day!
It appears that the sample in IBM's API-documentation is incorrect.
The code below has been pasted as plain text, to be able to strike through obsolete elements in IBM's sample code.
import json
from ibm_watson import NaturalLanguageUnderstandingV1
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
from ibm_watson.natural_language_understanding_v1
import Features, MetadataOptions
authenticator = IAMAuthenticator('{apikey}')
natural_language_understanding = NaturalLanguageUnderstandingV1(
version='2020-08-01',
authenticator=authenticator
)
natural_language_understanding.set_service_url('{url}')
response = natural_language_understanding.analyze(
url='www.ibm.com',
features=Features(metadata=MetadataOptions() {} )).get_result()
print(json.dumps(response, indent=2))
So for requesting the metadata object, just provide an empty dictionary (metadata={})
Enjoy your day!
Joost
I saw a few similar posts, but unfortunately none helped me.
I have an s3 bucket (on scaleway), and I'm trying to simply list all objects contained in that bucket, using boto3 s3 client as follow:
s3 = boto3.client('s3',
region_name=AWS_S3_REGION_NAME,
endpoint_url=AWS_S3_ENDPOINT_URL,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)
all_objects = s3.list_objects_v2(Bucket=AWS_STORAGE_BUCKET_NAME)
This simple piece of code responds with an error:
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the ListObjects operation: The specified key does not exist.
First, the error seems inapropriate to me since I'm not specifying any key to search. I also tried to pass a Prefix argument to this method to narrow down the search to a specific subdirectory, same error.
Second, I tried to achieve the same thing using boto3 Resource rather than Client, as follow:
session = boto3.Session(
region_name=AWS_S3_REGION_NAME,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)
resource = session.resource(
's3',
endpoint_url=AWS_S3_ENDPOINT_URL,
)
for bucket in resource.buckets.all():
print(bucket.name)
That code produces absolutely nothing. One weird thing that strikes me is that I don't pass the bucket_name anywhere here, which seems to be normal according to aws documentation
There's no chance that I misconfigured the client, since I'm able to use the put_object method perfectly with that same client. One strange though: when I want to put a file, I pass the whole path to put_object as Key (as I found it to be the way to go), but the object is inserted with the bucket name prepend to it. So let's say I call put_object(Key='/path/to/myfile.ext'), the object will end up to be /bucket-name/path/to/myfile.ext.
Is this strange behavior the key to my problem ? How can I investigate what's happening, or is there another way I could try to list bucket files ?
Thank you
EDIT: So, after logging the request that boto3 client is sending, I noticed that the bucket name is append to the url, so instead of requesting https://<bucket_name>.s3.<region>.<provider>/, it requests https://<bucket_name>.s3.<region>.<provider>/<bucket-name>/, which is leading to the NoSuchKey error.
I took a look into the botocore library, and I found this:
url = _urljoin(endpoint_url, r['url_path'], host_prefix)
in botocore.awsrequest line 252, where r['url_path'] contains /skichic-bucket?list-type=2. So from here, I should be able to easily patch the library core to make it work for me.
Plus, the Prefix argument is not working, whatever I pass into it I always receive the whole bucket content, but I guess I can easily patch this too.
Now it's not satisfying, since there's no issue related to this on github, I can't believe that the library contains such a bug that I'm the first one to encounter.
Does anyone can explain this whole mess ? >.<
For those who are facing the same issue, try changing your endpoint_url parameter in your boto3 client or resource instantiation from https://<bucket_name>.s3.<region>.<provider> to https://s3.<region>.<provider> ; i.e for Scaleway : https://s3.<region>.scw.cloud.
You can then set the Bucket parameter to select the bucket you want.
list_objects_v2(Bucket=<bucket_name>)
you can try this. you'll have to use your resource instead of my s3sr.
s3sr = resource('s3')
bucket = 'your-bucket'
prefix = 'your-prefix/' # if no prefix, pass ''
def get_keys_from_prefix(bucket, prefix):
'''gets list of keys for given bucket and prefix'''
keys_list = []
paginator = s3sr.meta.client.get_paginator('list_objects_v2')
# use Delimiter to limit search to that level of hierarchy
for page in paginator.paginate(Bucket=bucket, Prefix=prefix, Delimiter='/'):
keys = [content['Key'] for content in page.get('Contents')]
print('keys in page: ', len(keys))
keys_list.extend(keys)
return keys_list
keys_list = get_keys_from_prefix(bucket, prefix)
After looking more closely into things, I've found out that (a lot) of botocore services endpoints patterns starts with the bucket name. For example, here's the definition of the list_objects_v2 service:
"ListObjectsV2":{
"name":"ListObjectsV2",
"http":{
"method":"GET",
"requestUri":"/{Bucket}?list-type=2"
},
My guess is that in the standard implementation of AWS S3, there's a genericendpoint_url (which explains #jordanm comment) and the targeted bucket is reached through the endpoint.
Now, in the case of Scaleway, there's an endpoint_url for each bucket, with the bucket name contained in that url (e.g https://<bucket_name>.s3.<region>.<provider>), and any endpoint should directly starts with a bucket Key.
I made a fork of botocore where I rewrote every endpoint to remove the bucket name, if that can help someone in the future.
Thank's again to all contributors !
I'm trying to set up some simple web service with Python and Flask and SQlite3. It doesn't work.
The DB connection without web service works; the web service without DB connections works. Together they don't.
if I run this, it works:
import sqlite3
conn = sqlite3.connect('scuola.db')
sql = "SELECT matricola,cognome,nome FROM studenti"
cur = conn.cursor()
cur.execute(sql)
risultato = cur.fetchall()
conn.close()
print(risultato)
(so query is correct)
and if I run this, it works
import flask
app = flask.Flask(__name__)
def funzione():
return 'Applicazione Flask'
app.add_url_rule('/', 'funzione', funzione)
but if I run this...
from flask import Flask
import sqlite3
app = Flask(__name__)
#app.route('/',methods=['GET'])
def getStudenti():
conn = sqlite3.connect('scuola.db')
sql = "SELECT matricola,cognome,nome FROM studenti"
cur = conn.cursor()
cur.execute(sql)
risultato = cur.fetchall()
conn.close()
return risultato
It returns Internal Server Error in the browser, and
sqlite3.OperationalError: no such table: studenti
on the DOS prompt.
Thank you for your help!
You haven't provided the internal server error output - but my first guess is that you're trying to return the raw list object returned from fetchall.
When returning from a view function you need to send the results either by returning a template, or by jsonifying the output to make it a proper HTTP response that the browser can receive.
You need to add
from flask import jsonify
in your imports, then when returning;
return jsonify(risultato)
If you get errors like something is not JSON serializable if means you're trying to send an instance of a class or similar. You'll need to make sure you're returning only plain python data structures (e.g. list/dict/str etc).
For the command line problem, you need to make sure you've ran a CREATE TABLE command to first generate the table in the database, before you select from it. Also check you're accessing the correct sqlite database file with the table in it.
I'm not sure, but from the look of things I don't think you've configured the flask app to support the db you created There should be some sort of app.config() that integrates the db.
Is it possible to append data to a subcollection in the cloud firestore database using the python firebase admin sdk? If so, what am i missing?
I am trying to append data into a subcollection of a specific document located in googles cloud firestore - Not the real time database. I have done a fair amount of research with this being the most prevalent resource so far:
add data to firestore which does not explicitly say it isn't possible
i am able to create documents, read documents, etc. i just cannot seem to append to or access subcollections without getting the whole document
the flow of my code goes as follows:
import time
import json
import firebase_admin
from firebase_admin import credentials
from firebase_admin import db
from firebase_admin import firestore
#authentication
cred = credentials.Certificate('key.json')
app = firebase_admin.initialize_app(cred)
cloudDb = firestore.client() #open connection. uses 'app' implicitly
doc = cloudDb.collection(collection).document(document)
data['date'] = firestore.SERVER_TIMESTAMP
data['payload'] = [0,1,2,3,4,5]
doc.collection("data").set(data) #*
#--testing (collection)
# |
#----docId (document)
# |
#------company (field)
#------accountinfo(field)
#------data (subcollection)
# |
#--------date (field)
#--------payload (array)
this however fails on the last line (line with asterisk) with the error:
('Cannot convert to a Firestore Value', <object object at 0xb6b07e60>, 'Invalid type', <class 'object'>)
ok so solved my own question thanks to Doug getting me to write a script which could be run on another machine. (ie a smaller script with less going on)
my issue was trying to set the collection as my data object instead of creating a document within the collection and setting that. ie:
doc.collection("data").set(data) #error
doc.collection("data").document().set(data) #success
According to this answer one can retrieve immediate "subdirectories" by querying by prefix and then obtaining CommonPrefix of the result of Client.list_objects() method.
Unfortunately, Client is a part of so-called "low level" API.
I am using different API:
session = Session(aws_access_key_id=access_key,
aws_secret_access_key=secret_key)
s3 = session.resource('s3')
my_bucket = s3.Bucket(bucket_name)
result = my_bucket.objects.filter(Prefix=prefix)
and this method does not return dictionary.
Is it possible to obtain common prefixes with higher level API in boto3?
As noted in this answer, it seems that the Resource doesn't handle Delimiter well. It is often annoying, when your entire stack relies on Resource, to be told that, ah, you should have instantiated a Client instead...
Fortunately, a Resource object, such as your Bucket above, contains a client as well.
So, instead of the last line in your code sample, do:
paginator = my_bucket.meta.client.get_paginator('list_objects')
for resp in paginator.paginate(Bucket=my_bucket.name, Prefix=prefix, Delimiter='/', ...):
for x in resp.get('CommonPrefixes', []):
print(x['Prefix'])
You can access client from session.
session.client('s3').list_objects(Bucket=bucket_name, Prefix= prefix)