Using Prometheus with Connexion - ValueError: Duplicated timeseries in CollectorRegistry - python-3.x

I get the following error message when using prometheus with connexion using python3.6/3.7:
ValueError: Duplicated timeseries in CollectorRegistry: {'app_request_processing_seconds_sum', 'app_request_processing_seconds_count', 'app_request_processing_seconds_created', 'app_request_processing_seconds'}
#!/usr/bin/env python3
from gevent import monkey # noqa
# monkey.patch_all() # noqa
import json
import os
import connexion
import datetime
import logging
from connexion import NoContent
from prometheus_client import Summary, Counter
logger = logging.getLogger(__name__)
REQUEST_TIME = Summary('app_request_processing_seconds', 'time spent processing')
REQUEST_COUNTER = Counter('app_request_count', 'number of requests')
#REQUEST_TIME.time()
def get_health():
try:
'Hello'
except Exception:
return connexion.problem(503, "Service Unavailable", "Unhealthy")
else:
return "Healthy"
logging.basicConfig(level=logging.INFO)
app = connexion.App(__name__)
app.add_api("swagger.yaml")
if __name__ == "__main__":
# run our standalone gevent server
app.run(port=8080, server="gevent")
There is a swagger.yaml that is identical to:
https://github.com/hjacobs/connexion-example-redis-kubernetes/blob/master/swagger.yaml
Any help would be great

As a guess, you have named your file app.py. What happens it that when the swagger is loaded, the handling is specified as app.get_health:
paths:
/health:
get:
operationId: app.get_health
And it loads (a second time) app.py to import the get_health() function.
The reason it that the main file is loaded as __main__ module and thus get loaded a second time; see this other question for more information.
Therefore, you end-up with defining your Prometheus metrics twice which doesn't sit well with the collector.
The most simple solution is to rename your file and implement your application in another file named app.py.

Related

Backtrader in Python AttributeError: 'DataFrame' object has no attribute 'setenvironment'. What's the problem and how to solve it?

new to backtrader
when getting starting with the documentation and implemnting the code, i get the error
AttributeError: 'DataFrame' object has no attribute 'setenvironment' in the cerebro.adddata line
you can check in the image below:
the error when i run cerebro.adddata
i have already tried the solution mentionned in this Backtrader error: 'DataFrame' object has no attribute 'setenvironment' but nothing changed
if anyone has experienced this before and could help will be nice, this is the code i'm using and i don't know where the problem is:
from __future__ import (absolute_import, division, print_function,
unicode_literals)
import datetime # For datetime objects
import os.path # To manage paths
import sys # To find out the script name (in argv[0])
import pandas as pd
import pandas_datareader as pdr
#import backtrader platform
import backtrader as bt
if __name__ == '__main__':
# Create a cerebro entity
cerebro = bt.Cerebro()
#create a data feed
data= pd.read_csv('EURUSD20102020')
#add the Data Feed to Cerebro
cerebro.adddata(data)
# Set our desired cash start
cerebro.broker.setcash(100000.0)
# Print out the starting conditions
print('Starting Portfolio value: %.2f' % cerebro.broker.get_value())
# Run over everything
cerebro.run()
# Print out the final result
print('Final Portfolio Value: %.2f' % cerebro.broker.get_value())

Cloud Run connect to Cloud SQL module error Python

I keep getting an error trying to connect Cloud Run and I keep getting the following error. Any idea?
__import__("pg8000") ModuleNotFoundError: No module named 'pg8000'
import pandas as pd
import sqlalchemy
import datetime
import requests
from urllib.parse import urlencode
import warnings
from flask import Flask
import os
import google
db_user = os.environ.get("DB_USER")
db_pass = os.environ.get("DB_PASS")
db_name = os.environ.get("DB_NAME")
cloud_sql_connection_name = os.environ.get("CLOUD_SQL_CONNECTION_NAME")
db = sqlalchemy.create_engine(
# Equivalent URL:
# postgres+pg8000://<db_user>:<db_pass>#/<db_name>?unix_sock=/cloudsql/<cloud_sql_instance_name>/.s.PGSQL.5432
sqlalchemy.engine.url.URL(
drivername='postgres+psycopg2',
username=db_user,
password=db_pass,
database=db_name,
query={
'unix_sock': '/cloudsql/{}/.s.PGSQL.5432'.format(
cloud_sql_connection_name)
}
),
# ... Specify additional properties here.
# ...
)
You need to install one of the supported database drivers.
If you want to use postgres+pg8000, you need to install the pg8000 package, otherwise based on your example, you actually need to install psycopg2.

Why i get this error TypeError:'type' object is not iterable in Python Robot Framework

im getting this Error Then i run my Code
i have treid to make a class but nothing Works
import subprocess
import time
import pyautogui as pg
from robot.api import logger
import signal
from subprocess import Popen, PIPE, CalledProcessError, check_output
import weakref
def close_application(self):
"""
Closes a application prior opened by open_application.
"""
logger.info("Closing application...")
subprocess.Popen(self.sigmaProcess)
self.sigmaProcess.communicate(self.sigmaProcess, timeout=None)
self.sigmaProcess.kill()
self.sigmaProcess.communicate(self.sigmaProcess)
The Main Code
def open_application(self, path, workingDir):
"""
Opens a application.
:param path: The application path.
:param workingDir: The application's working dir.
"""
logger.info("Opening {} with working directory {}...".format(path, workingDir))
self.sigmaProcess = subprocess.Popen([path], cwd=workingDir)
here is subprocess.Popen defined
*** Settings ***
Resource ../sigma.robot
Suite Teardown click on processOverviewTab
*** Test Cases ***
Test title
close application
open application
wait for on tab processOverviewTab 600000
The Code for the Test
Test title :: Prüft den bug Using centered capture region (570.0, 390.0, 800, 480).
| FAIL |
TypeError: 'type' object is not iterable
and this is the Error Message
and when i Delete th subprocess.Popen and self.sigmaProcess.communicate i get this Error
AttributeError: type object 'Popen' has no attribute 'returncode'

Why does my Flask app work when executing using `python app.py` but not when using `heroku local web` or `flask run`?

I wrote a Flask-based web app that takes text from users and returns the probability that it is of a given classification (full script below). The app loads some of the trained models needed to make predictions before any requests are made. I am currently trying to deploy it on Heroku and experiencing some problems.
I am able to run it locally when I execute python ml_app.py. But when I use the Heroku CLI command heroku local web to try to run it locally to test before deployment, I get the following error
AttributeError: module '__main__' has no attribute 'tokenize'
This error is associated with the loading of a text vectorizer called TFIDF found in the line
tfidf_model = joblib.load('models/tfidf_vectorizer_train.pkl')
I have imported the required function at the top of the script to ensure that this is loaded properly (from utils import tokenize). This works given that I can run it when I use python ml_app.py. But for reasons I do not know, it doesn't load when I use heroku local web. It also doesn't work when I use the Flask CLI command flask run when trying to run it locally. Any idea why?
I admit that I do not have a good understanding of what is going on under the hood here (with respect to the web dev./deployment aspect of the code) so any explanation helps.
from flask import Flask, request, render_template
from sklearn.externals import joblib
from utils import tokenize # custom tokenizer required for tfidf model loaded in load_tfidf_model()
app = Flask(__name__)
models_directory = 'models'
#app.before_first_request
def nbsvm_models():
global tfidf_model
global logistic_identity_hate_model
global logistic_insult_model
global logistic_obscene_model
global logistic_severe_toxic_model
global logistic_threat_model
global logistic_toxic_model
tfidf_model = joblib.load('models/tfidf_vectorizer_train.pkl')
logistic_identity_hate_model = joblib.load('models/logistic_identity_hate.pkl')
logistic_insult_model = joblib.load('models/logistic_insult.pkl')
logistic_obscene_model = joblib.load('models/logistic_obscene.pkl')
logistic_severe_toxic_model = joblib.load('models/logistic_severe_toxic.pkl')
logistic_threat_model = joblib.load('models/logistic_threat.pkl')
logistic_toxic_model = joblib.load('models/logistic_toxic.pkl')
#app.route('/')
def my_form():
return render_template('main.html')
#app.route('/', methods=['POST'])
def my_form_post():
"""
Takes the comment submitted by the user, apply TFIDF trained vectorizer to it, predict using trained models
"""
text = request.form['text']
comment_term_doc = tfidf_model.transform([text])
dict_preds = {}
dict_preds['pred_identity_hate'] = logistic_identity_hate_model.predict_proba(comment_term_doc)[:, 1][0]
dict_preds['pred_insult'] = logistic_insult_model.predict_proba(comment_term_doc)[:, 1][0]
dict_preds['pred_obscene'] = logistic_obscene_model.predict_proba(comment_term_doc)[:, 1][0]
dict_preds['pred_severe_toxic'] = logistic_severe_toxic_model.predict_proba(comment_term_doc)[:, 1][0]
dict_preds['pred_threat'] = logistic_threat_model.predict_proba(comment_term_doc)[:, 1][0]
dict_preds['pred_toxic'] = logistic_toxic_model.predict_proba(comment_term_doc)[:, 1][0]
for k in dict_preds:
perc = dict_preds[k] * 100
dict_preds[k] = "{0:.2f}%".format(perc)
return render_template('main.html', text=text,
pred_identity_hate=dict_preds['pred_identity_hate'],
pred_insult=dict_preds['pred_insult'],
pred_obscene=dict_preds['pred_obscene'],
pred_severe_toxic=dict_preds['pred_severe_toxic'],
pred_threat=dict_preds['pred_threat'],
pred_toxic=dict_preds['pred_toxic'])
if __name__ == '__main__':
app.run(debug=True)
Fixed it. It was due to the way I picked the class instance stored in tfidf_vectorizer_train.pkl. The model was created in an ipython notebook where one of its attributes depended on a tokenizer function that I defined interactively in the notebook. I soon learned that pickling does not save the exact instance of a class, which means tfidf_vectorizer_train.pkl does not contain the function I defined in the notebook.
To fix this, I moved the tokenizer function to a separate utilities python file and imported the function in both the file where I trained and subsequently pickled the model and in the file where I unpickled it.
In code, I did
from utils import tokenize
...
tfidfvectorizer = TfidfVectorizer(ngram_range=(1, 2), tokenizer=tokenize,
min_df=3, max_df=0.9, strip_accents='unicode',
use_idf=1, smooth_idf=True, sublinear_tf=1)
train_term_doc = tfidfvectorizer.fit_transform(train[COMMENT])
joblib.dump(tfidfvectorizer, 'models/tfidf_vectorizer_train.pkl')
...
in the file where I trained the model and
from utils import tokenize
...
#app.before_first_request
def load_models():
# from utils import tokenize
global tfidf_model
tfidf_model =
joblib.load('{}/tfidf_vectorizer_train.pkl'.format(models_directory))
...
in the file containing the web app code.

Google speech long_running_recognizer does not work when fired through celery task

The code below works when fired as a script using python. But the same, when i encapsulate in a celery task and try to execute, it does not work. The celery task prints the line before the long_running_recognize, but does not print the one after the operation - seems like it gets stuck at the long_running_recognize call when executing as a celery task.
#!/usr/bin/env python3
import speech_recognition as sr
import json
import sqlalchemy
import io
import os
# Imports the Google Cloud client library
from google.cloud import speech
from google.cloud.speech import enums
from google.cloud.speech import types
from sqlalchemy import create_engine
# Instantiates a client
client = speech.SpeechClient()
audio=speech.types.RecognitionAudio(uri='gs://<bucket_name>/<audio_file>')
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
language_code='en-US')
print('FIRING GOOGLE SPEECH RECOGNITION')
# Detects speech in the audio file
operation = client.long_running_recognize(config, audio)
print('Waiting for operation to complete...')
response = operation.result(timeout=600)
outfile = open('test.txt', 'w')
for result in response.results:
for alternative in result.alternatives:
print('=' * 20)
outfile.write('Transcript: {}'.format(alternative.transcript))
outfile.write('=' * 20)
outfile.write("Confidence: {}".format(alternative.confidence))
print('Transcript: {}'.format(alternative.transcript))
print(alternative.confidence)
outfile.close()
I had the same issue today, but it worked recently. I rolled back to these requirements in pip and it solved the problem.
google-api-python-client==1.6.4
google-auth==1.1.1
google-cloud-core==0.27.1
google-cloud-speech==0.29.0
google-gax==0.15.15
googleapis-common-protos==1.5.2

Resources