I'm trying to build a project using an Apache Flink Stateful Function in Python, but I can't seem to get it to work. What I've narrowed the issue down to is that it seems when I send the request to my stateful function through my protobuf schema, the serializer is unable to serialize my message into the class I'm expecting. Here's what I'm trying to do:
import json
from statefun import StatefulFunctions, RequestReplyHandler
from jobs.session_event_pb2 import Event
functions = StatefulFunctions()
#functions.bind("namespace/funcname")
def funcname(context, session: Event):
print("hello world")
handler = RequestReplyHandler(functions)
if __name__ == '__main__':
inputFile = open("my_file.json", "r")
for line in inputFile:
data = json.loads(line).get('properties')
if data is not None and data.get('prop1') is not None and data.get('prop2') is not None:
request = Event()
request.prop1 = data["prop1"]
request.prop2 = data["prop2"]
request = request.SerializeToString()
handler(request)
Here's my Protobuf schema:
syntax = "proto3";
package mypackage;
message Event {
string prop1 = 1;
string prop2 = 2;
}
What am I doing wrong here?
That's because the RequestReply handler does not take direct protobuf messages. The Flink runtime sends a type called ToFunction and receives a response of type FromFunction. This payload contains your caller messages along with persisted values and other meta information.
If you can't to invoke the functions directly, such as in a test, I would encourage you to do that and not use the handler at all.
Related
I'm new to flask and in order to refactor an existing route method on a Flask API, i'm looking for the best practice to reduce it and call method inside the route method.
Acutally the route is designed like that :
#qman.route('/add_report/', methods=['POST'])
def create_report():
"""
Check if data send throught http POST request, is correct based on the report
schema and not already recorded in the table report of the DB.
:param: data from POST request
:return: Ok, valide and imported -> 201, Correct but AlreadyKnown -> 208,
InvalideScheme -> 422
"""
jsonData = request.get_json()
reportSchema = ReportSchema()
try:
data = reportSchema.load(jsonData)
except ValidationError as validation_err:
return(validation_err.messages), 422
nameReportCheck = data["report_name"]
report = Report.query.filter_by(report_name=nameReportCheck).first()
if report is None:
# Create new report
report = Report(
report_name=nameReportCheck,
hostname=data["hostname"],
status=data["status"],
date=data["date"],
nb_analysis_erreur=data["nb_analysis_erreur"]
)
db.session.add(report)
db.session.commit()
NewResult = reportSchema.dump(Report.query.get(report.reportID))
return{"message" : "Created new report" , "report" : NewResult}, 201
else :
reportAlreadyKnown = reportSchema.dump(Report.query.get(report.reportID))
return{"message" : "This report is already in the DB", "report" : reportAlreadyKnown}, 208
In the facts i would like to call a function named valid_schema(_schema, _jsondata) to check if the data send throught POST request match with my schema of model Report().
This function return a Response() object with serialized data and a 200 code if it's serialization is possible or an error that i cath inside try/except with 400 error code.
def valid_schema(_schema, _jsondata):
schema = _schema()
try:
data = schema.load(_jsondata)
except ValidationError as validation_err:
response = Response(validation_err.messages, 422)
return response
response = Response(data, 200, mimetype="application/json")
return response
Then the route method call an other function named create_report(report_data) if valid_schema(_schema, _jsondata) return report_data and 200 code in response object.
With his args, this method check if the records is not already in the DB and if is not, he create a Report() object from report_data arg and insert this one as a new record into the DB.
In fact I guess I can easily call this method inside the route function but it seem weird and there is probably an other way that I can't find, maybe decorator ?
One possibility for refactoring is the use of webargs, Flask-Marshmallow and marshmallow-sqlalchemy.
With Flask-Marshmallow you can check the input by specifying fields and validators. Webargs offers you the option of validating the defined scheme in a decorator and passing it on to the route as an argument. Using marshmallow-sqlalchemy in combination, this is immediately converted into a database model.
The following example is based on your information and gives you a brief overview of the usage. By defining your own error handler, the error messages can also be sent as JSON. Use in blueprints, views or the like is possible.
from flask import Flask
from flask import jsonify
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow
from marshmallow.validate import Length, OneOf
from webargs.flaskparser import use_args
app = Flask(__name__)
db = SQLAlchemy(app)
ma = Marshmallow(app)
class Report(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String, unique=True)
hostname = db.Column(db.String)
status = db.Column(db.String)
date = db.Column(db.DateTime)
nb_analysis_error = db.Column(db.String)
class ReportSchema(ma.SQLAlchemyAutoSchema):
class Meta:
model = Report
load_instance = True
sqla_session = db.session
name = ma.Str(required=True, validate=Length(min=3))
hostname = ma.Str(required=True)
date = ma.DateTime(required=True)
status = ma.Str(required=True, validate=OneOf(['online', 'offline']))
nb_analysis_error = ma.Str(missing='Unknown Error')
#app.route('/add_report', methods=['POST'])
#use_args(ReportSchema(), location='json')
def add_report(report):
report_schema = ReportSchema()
_report = Report.query.filter_by(name=report.name).first()
if _report:
report_data = report_schema.dump(_report)
return jsonify(message='Already Reported', report=report_data), 208
else:
db.session.add(report)
db.session.commit()
report_data = report_schema.dump(report)
return jsonify(message='Created', report=report_data), 201
with app.app_context():
db.drop_all()
db.create_all()
Trying to figure out the right mechanism to use here.
I want to modify the flask request coming in every time.
I think the request is immutable, so I am trying to figure out if this mechanism exists.
Basically, I want to append a string onto the end of the request coming in.
I can hook into the request and the right time in a before_request handler like this:
#app.before_app_request
def before_request_custom():
# Get the request
req = flask.request
method = str(req.method)
if method == "GET":
# Do stuff here
pass
But I am not sure what to actually do to add this in, and don't see a way to accomplish it...I guess i could redirect, but that seems silly in this case. Any ideas?
The request object is immutable (https://werkzeug.palletsprojects.com/en/1.0.x/wrappers/#base-wrappers), but request.args or request.form can be set from ImmutableOrderedMultiDict to just OrderedMultiDict using Subclassing on Flask (https://flask.palletsprojects.com/en/1.1.x/patterns/subclassing/). Here's an example of how you could add that filter[is_deleted]=False URL param:
from flask import Flask, request, Request
from werkzeug.datastructures import OrderedMultiDict
class MyRequest(Request):
parameter_storage_class = OrderedMultiDict
class MyApp(Flask):
def __init__(self, import_name):
super(MyApp, self).__init__(import_name)
self.before_request(self.my_before_method)
def my_before_method(self):
if "endpoint" in request.base_url:
request.args["filter[is_deleted]"] = "False"
app = MyApp(__name__)
app.request_class = MyRequest
#app.route('/endpoint/')
def endpoint():
filter = request.args.get('filter[is_deleted]')
return filter
This way you can modify request.args before you actually send the request.
How about this?
from flask import g
#app.before_request
def before_request():
# Get the request
req = flask.request
method = str(req.method)
if method == "GET":
g.my_addon = "secret sauce"
return None
Then, g.my_addon is available in every view function:
from flask import g
#app.route('/my_view')
def my_view():
if g.my_addon == "secret sauce":
print('it worked!')
Using test_request_context() you can make the trick.
Related: https://flask.palletsprojects.com/en/1.1.x/quickstart/#accessing-request-data
How do I create a object that I can invoke to mimic the following api call and response. I am aware of the mock library but the use case prohibits me from using it.
response = client.users.create(email='test#gmail.com', phone=123)
outcome = response.ok
My current solution below works however I feel like there is a more pythonic and generic way to do this so I can mimic other calls without having to rewrite different inner classes
class Client:
ok = True
class users:
class create():
ok = True
def __init__(self, email, phone):
pass
Input
client = Client()
response = client.users.create(email='test#gmail.com', phone=123)
response.ok
Output
True
I am attempting to write my own response handlers for Groovy's RESTClient (which wraps around HttpBuilder). I want to always print the response body if one is returned. However, I cannot find a consistent way to do it.
Typically a custom response handler would look like this:
def client = new RESTClient(url)
client.handler.success = { resp, reader ->
//do stuff
}
client.handler.failure = { resp, reader ->
//do stuff
throw new Exception("HTTP call failed. Status code: ${resp.getStatus()}")
}
However, what I noticed is that the variable "reader" can have a different class depending on the response. I've seen the reader be of type groovy.util.slurpersupport.NodeChild or org.apache.http.conn.EofSensorInputStream. I want it to be a predictable class so I can actually call the methods on this object. What's going on here?
Setting the content type to ANY and changing the HttpBuilder content parsers to the text parser fixed the issue. The type of reader in the response handler is now always java.io.InputStreamReader.
Before:
def headerMap = [:]
//populate headers
def response = client.get("headers":headerMap)
After:
client.parser.'application/xml' = client.parser.'text/plain'
client.parser.'application/xhtml+xml' = client.parser.'text/plain'
client.parser.'application/atom+xml' = client.parser.'text/plain'
client.parser.'application/json' = client.parser.'text/plain'
client.parser.'text/html' = client.parser.'text/plain'
client.parser.'application/x-www-form-urlencoded' = client.parser.'text/plain'
def headerMap = [:]
//populate headers
def response = client.get("headers":headerMap, contentType:groovyx.net.http.ContentType.ANY)
When running a CherryPy app it will send server name tag something like CherryPy/version.
Is it possible to rename/overwrite that from the app without modifying CherryPy so it will show something else?
Maybe something like MyAppName/version (CherryPy/version)
This can now be set on a per application basis in the config file/dict
[/]
response.headers.server = "CherryPy Dev01"
Actually asking on IRC on their official channel fumanchu gived me a more clean way to do this (using latest svn):
import cherrypy
from cherrypy import _cpwsgi_server
class HelloWorld(object):
def index(self):
return "Hello World!"
index.exposed = True
serverTag = "MyApp/%s (CherryPy/%s)" % ("1.2.3", cherrypy.__version__)
_cpwsgi_server.CPWSGIServer.environ['SERVER_SOFTWARE'] = serverTag
cherrypy.config.update({'tools.response_headers.on': True,
'tools.response_headers.headers': [('Server', serverTag)]})
cherrypy.quickstart(HelloWorld())
This string appears to be being set in the CherrPy Response class:
def __init__(self):
self.status = None
self.header_list = None
self._body = []
self.time = time.time()
self.headers = http.HeaderMap()
# Since we know all our keys are titled strings, we can
# bypass HeaderMap.update and get a big speed boost.
dict.update(self.headers, {
"Content-Type": 'text/html',
"Server": "CherryPy/" + cherrypy.__version__,
"Date": http.HTTPDate(self.time),
})
So when you're creating your Response object, you can update the "Server" header to display your desired string. From the CherrPy Response Object documentation:
headers
A dictionary containing the headers of the response. You may set values in
this dict anytime before the finalize phase, after which CherryPy switches
to using header_list ...
EDIT: To avoid needing to make this change with every response object you create, one simple way to get around this is to wrap the Response object. For example, you can create your own Response object that inherits from CherryPy's Response and updates the headers key after initializing:
class MyResponse(Response):
def __init__(self):
Response.__init__(self)
dict.update(self.headers, {
"Server": "MyServer/1.0",
})
RespObject = MyResponse()
print RespObject.headers["Server"]
Then you can can call your object for uses where you need to create a Response object, and it will always have the Server header set to your desired string.