I have just started working on Flask and Python. I am building an API server in which I am trying to read a csv file and display the output of csv file on /osdata at port 8080.
There might be some issue in returning the data from function. Any help would be appreciated.
from flask import Flask, request
from flask_restful import Resource, Api
import json, csv
app = Flask(__name__)
app.config["DEBUG"] = True
api = Api(app)
#app.route('/', methods=['GET'])
def home():
return "1 2 3 Start"
#app.route('/osdata', methods=['GET'])
def osdata():
with open("Output_data.csv", "r") as f:
data = csv.reader(f, delimiter = "~")
line_count = 0
for row in data:
print(row)
line_count += 1
return(row)
if __name__ == '__main__':
app.run(port=8080)
Related
I'm facing an issue in creating Realtime status update for merging new datasets with old one and machine learning model creation results via Web framework. The tasks are simple in following steps.
An user/ client will send a new datasets in .CSV file to the server,
On server side my windows machine will receive a file then send an acknowledge,
Merge the new dataset with the old one for new machine learning model creation and
Run another python script(that is to create a new sequential deep-learning model). After the successful completion of another python script my code have to return the response to the client!
I have deployed my python-flask application on IIS-10. To run an another python script, this main flask-api script should have to wait for completing that model creation script. On model creation python script it contains several process like loading datasets, tokenizing, oneHot Encoding, padding techniques, model training for 100 epochs and finally prediction results.
My exact goal is this Flask-API should have to wait for until completing the entire process. I'm sure definitely it will take 8-9 minutes to complete the whole script mentioned in subprocess.run(). While testing this code on development mode it's working excellently without any issues! But while testing it on production mode on IIS no it's not waiting for the whole process and within 6-7 seconds it returning response to the client.
For debugging purpose I included logging to record all events in both Flask script and machine learning model creation script! Through that I came to understand that model creation script only ran 10%!. First I tried simple methods with async def and await to run the subprocess.run() it didn't make any sense! Then I included threading and get_event_loop() and then run_until_complete() to make my parent code wait until finishing the whole process. But finally I'm helpless!! I couldn't able to find a rightful solution. Please let me know what I did wrong.. Thank you.
Configurations:
Python 3.7.9
Windows server 2019 and
IIS 10.0 Express
My code:
import os
import time
import glob
import subprocess
import pandas as pd
from flask import Flask, request, jsonify
from werkzeug.utils import secure_filename
from datetime import datetime
import logging
import asyncio
from concurrent.futures import ThreadPoolExecutor
ALLOWED_EXTENSIONS = {'csv', 'xlsx'}
_executor = ThreadPoolExecutor(1)
app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = "C:\\inetpub\\wwwroot\\iAssist_IT_support\\New_IT_support_datasets"
currentDateTime = datetime.now()
filenames = None
logger = logging.getLogger(__name__)
app.logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s:%(name)s:%(message)s')
file_handler = logging.FileHandler('model-creation-status.log')
file_handler.setFormatter(formatter)
# stream_handler = logging.StreamHandler()
# stream_handler.setFormatter(formatter)
app.logger.addHandler(file_handler)
# app.logger.addHandler(stream_handler)
def allowed_file(filename):
return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS
#app.route('/file_upload')
def home():
return jsonify("Hello, This is a file-upload API, To send the file, use http://13.213.81.139/file_upload/send_file")
#app.route('/file_upload/status1', methods=['POST'])
def upload_file():
app.logger.debug("/file_upload/status1 is execution")
# check if the post request has the file part
if 'file' not in request.files:
app.logger.debug("No file part in the request")
response = jsonify({'message': 'No file part in the request'})
response.status_code = 400
return response
file = request.files['file']
if file.filename == '':
app.logger.debug("No file selected for uploading")
response = jsonify({'message': 'No file selected for uploading'})
response.status_code = 400
return response
if file and allowed_file(file.filename):
filename = secure_filename(file.filename)
file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
print(filename)
print(file)
app.logger.debug("Spreadsheet received successfully")
response = jsonify({'message': 'Spreadsheet uploaded successfully'})
response.status_code = 201
return response
else:
app.logger.debug("Allowed file types are csv or xlsx")
response = jsonify({'message': 'Allowed file types are csv or xlsx'})
response.status_code = 400
return response
#app.route('/file_upload/status2', methods=['POST'])
def status1():
global filenames
app.logger.debug("file_upload/status2 route is executed")
if request.method == 'POST':
# Get data in json format
if request.get_json():
filenames = request.get_json()
app.logger.debug(filenames)
filenames = filenames['data']
# print(filenames)
folderpath = glob.glob('C:\\inetpub\\wwwroot\\iAssist_IT_support\\New_IT_support_datasets\\*.csv')
latest_file = max(folderpath, key=os.path.getctime)
# print(latest_file)
time.sleep(3)
if filenames in latest_file:
df1 = pd.read_csv("C:\\inetpub\\wwwroot\\iAssist_IT_support\\New_IT_support_datasets\\" +
filenames, names=["errors", "solutions"])
df1 = df1.drop(0)
# print(df1.head())
df2 = pd.read_csv("C:\\inetpub\\wwwroot\\iAssist_IT_support\\existing_tickets.csv",
names=["errors", "solutions"])
combined_csv = pd.concat([df2, df1])
combined_csv.to_csv("C:\\inetpub\\wwwroot\\iAssist_IT_support\\new_tickets-chatdataset.csv",
index=False, encoding='utf-8-sig')
time.sleep(2)
# return redirect('/file_upload/status2')
return jsonify('New data merged with existing datasets')
#app.route('/file_upload/status3', methods=['POST'])
def status2():
app.logger.debug("file_upload/status3 route is executed")
if request.method == 'POST':
# Get data in json format
if request.get_json():
message = request.get_json()
message = message['data']
app.logger.debug(message)
return jsonify("New model training is in progress don't upload new file")
#app.route('/file_upload/status4', methods=['POST'])
def model_creation():
app.logger.debug("file_upload/status4 route is executed")
if request.method == 'POST':
# Get data in json format
if request.get_json():
message = request.get_json()
message = message['data']
app.logger.debug(message)
app.logger.debug(currentDateTime)
def model_run():
app.logger.debug("model script starts to run")
subprocess.run("python C:\\.....\\IT_support_chatbot-master\\"
"Python_files\\main.py", shell=True)
# time.sleep(20)
app.logger.debug("script ran successfully")
async def subprocess_call():
# run blocking function in another thread,
# and wait for it's result:
app.logger.debug("sub function execution starts")
await loop.run_in_executor(_executor, model_run)
asyncio.set_event_loop(asyncio.SelectorEventLoop())
loop = asyncio.get_event_loop()
loop.run_until_complete(subprocess_call())
loop.close()
return jsonify("Model created successfully for sent file %s" % filenames)
if __name__ == "__main__":
app.run()
API beginner here,
I'm creating an API that calls both a character name and their description with one ENDPOINT. So far I have the data appearing at separate endpoints.
Goal: Getting output format ('http://127.0.0.1:5000/v1/pokemon/string:name') to look similar to this:
{ "name": "pichu", "description": "It is not yet skilled at storing electricity.It may send out a jolt if amused or startled."}
from flask import Flask, jsonify, Response
from flask_restful import Resource, Api, fields, marshal_with
import requests, json
app = Flask(__name__)
api = Api(app)
# Prints all Pokemon via JSON
#app.route('/v1/pokemon/all', methods=['GET'])
def poke_names():
data = []
name_url = "https://pokeapi.co/api/v2/pokemon?limit=151"
while True:
resp = requests.get(name_url)
json = resp.json()
data.extend(json.get('results', []))
name_url = json.get('next')
if not name_url: break
return jsonify(data)
#app.route('/v1/pokemon/<string:name>/title', methods=['GET'])
def get_poke(name):
return jsonify({'name': name})
#flavor Text ie pokemon description
#app.route('/v1/pokemon/<int:pokemon_id>', methods=['GET'])
def get_description(pokemon_id):
descrip_url = f"https://pokeapi.co/api/v2/pokemon-species/{pokemon_id}"
r = requests.get(descrip_url)
json_blob = r.json()
flav_text = extract_descriptive_text(json_blob)
return jsonify({'description': flav_text})
def extract_descriptive_text(json_blob, language='en'):
text = []
for f in json_blob['flavor_text_entries']:
if f['language']['name'] == language:
text.append(f['flavor_text'])
return text
#error occurs below
#app.route('/v1/pokemon/<string:name>')
def all_poke_data(name, flav_text):
return jsonify({'name':name, 'description':flav_text})
if __name__ == "__main__":
app.run(debug=True)
Error: When I try to combine them I get
TypeError: all_poke_data() missing 1 required positional argument: 'flav_text'
Thank you for any pointers provided!
Figured it out!
Just add this function, which is a combination of both route functions
#app.route('/v1/pokemon/<string:name>')
def get_poke(name):
descrip_url = f"https://pokeapi.co/api/v2/pokemon-species/{name}"
r = requests.get(descrip_url)
json_blob = r.json()
flav_text = extract_descriptive_text(json_blob)
return jsonify({'name': name},{'description': flav_text})
i want to pass my latitude and langitude values to my flask route but everytime i am getting this error: ValueError: Must be a coordinate pair or Point
however i have tried this and its working fine:
from flask import Flask, render_template
from geopy.geocoders import Nominatim
app = Flask(__name__)
geolocator = Nominatim()
#app.route('/location')
def lang_and_lat():
location = geolocator.reverse("21.0943, 81.0337")
address = location.address
return render_template('ip.html', address=address)
if __name__ == '__main__':
app.run(debug=True)
from flask import Flask, render_template
from geopy.geocoders import Nominatim
app = Flask(__name__)
geolocator = Nominatim()
#app.route('/location/<lat>/<lang>')
def lang_and_lat(lat, lang):
location = geolocator.reverse(lat, lang)
address = location.address
return render_template('ip.html', address=address)
if __name__ == '__main__':
app.run(debug=True)
you need to do
location = geolocator.reverse(f'{lat}, {lang}')
or
location = geolocator.reverse(Point(lat, lang))
in second case you need to from geopy.point import Point
After I implemented caching on my flask server, everything works perfectly on local host. First execution is 8000ms, second is 26ms, therefore its working.
When I deployed the application on the AWS ec2 box, the first execution is 21000ms and whenever I try to run it again, it comes with server not responsive.
This is the code:
#!flask/bin/python
from flask_cache import Cache
from flask import Flask, jsonify
from flask import request
from flask_caching import Cache
import json
import nltk, string
import operator
from sklearn.feature_extraction.text import TfidfVectorizer
import re
import time
import access_json
app = Flask(__name__)
cache = Cache(app, config={'CACHE_TYPE': 'simple'})
with open('JSON files/thesaurus.json', 'r') as fp:
thesaurus_dict = json.load(fp)
with open('JSON files/JOBS.json', 'r') as f:
json_list = json.load(f)
def output(word_list):
return filter_toplist
#app.route('/postjson', methods=['POST'])
#cache.cached(timeout=20)
def json_handler():
content = request.get_json(force=True)
word_list = access_json.read_parsed_JSON(content)
return jsonify ({'jobs': output(word_list)})
if __name__ == '__main__':
app.run(debug=True, host='0.0.0.0')
This is not all of the code but the one relevant to the flask server.
Python is throwing "Syntax Error" when I compile the code below.
File "app.py", line 11
#app.route('/')
^
SyntaxError: invalid syntax
I'm not sure what it means.
from flask import Flask, render_template
import urllib.request
import json
import time
app = Flask(__name__ ,template_folder='template')
namep = "PewDiePie"
namet = "TSeries"
key = "MY_API_KEY"
#app.route("/")
for x in range(5):
time.sleep(2)
datat = urllib.request.urlopen("https://www.googleapis.com/youtube/v3/channels?part=statistics&forUsername="+namep+"&key="+key).read()
datap = urllib.request.urlopen("https://www.googleapis.com/youtube/v3/channels?part=statistics&forUsername="+namet+"&key="+key).read()
subt = json.loads(datat)["items"][0]["statistics"]["subscriberCount"]
subsp = json.loads(datap)["items"][0]["statistics"]["subscriberCount"]
def main():
return render_template('index.html', pewds_sub = subsp, tseries_sub = subt)
if __name__ == "__main__":
app.run(debug=True, host="0.0.0.0", port=80)
Any help regarding this is appreciated.
Thanks!
You must define the function after the route decorator, i.e. after #app.route
Updated code
#app.route("/")
def function_main():
#all logics here
return render_template('index.html', pewds_sub = subsp, tseries_sub = subt)
Make sure to process your calculations inside function else try to pass those argument in defined function.
from flask import Flask, render_template
import urllib.request
import json
import time
app = Flask(__name__ ,template_folder='template')
namep = "PewDiePie"
namet = "TSeries"
key = "MY_API_KEY"
#app.route("/")
def main():
for x in range(5):
time.sleep(2)
datat = urllib.request.urlopen("https://www.googleapis.com/youtube/v3/channels?part=statistics&forUsername="+namep+"&key="+key).read()
datap = urllib.request.urlopen("https://www.googleapis.com/youtube/v3/channels?part=statistics&forUsername="+namet+"&key="+key).read()
subt = json.loads(datat)["items"][0]["statistics"]["subscriberCount"]
subsp = json.loads(datap)["items"][0]["statistics"]["subscriberCount"]
return render_template('index.html', pewds_sub = subsp, tseries_sub = subt)
if __name__ == "__main__":
app.run(debug=True, host="0.0.0.0", port=80)
in my case,
I initiated a try block just above for the database connection and forget to put catch block, that's why I have encountered this error.
so I suggest anyone facing the same error,
should check the code above #app.route('/') because if you have import flask
properly this should work pretty fine syntax error in this statement usually indicates that you might have a problem above this line and not at that line.