I am trying to make this web app to work but I am getting an error. these are the steps that web app is supposed to handle:
import a file
run the python script
export the results
when I run python script independently( without interfering with flask), it works fine( I use Jupyter notebook) on the other hand, when I run it with flask (from prompt) I get an error:
File "app.py", line 88, in <module>
for name, df in transformed_dict.items():
NameError: name 'transformed_dict' is not defined
Any idea of how can I make this web app to work?
This is my first time using flask and I will appreciate any suggestions or guidance.
python file & html file
from flask import Flask,render_template,request,send_file
from flask_sqlalchemy import SQLAlchemy
import os
import pandas as pd
from openpyxl import load_workbook
import sqlalchemy as db
def transform(df):
# Some data processing here
return df
app=Flask(__name__)
#app.route('/')
def index():
return render_template('firstpage.html')
#app.route('/upload',methods=['Get','POST'])
def upload():
file=request.files['inputfile']
xls=pd.ExcelFile(file)
name_dict = {}
snames = xls.sheet_names
for sn in snames:
name_dict[sn] = xls.parse(sn)
for key, value in name_dict.items():
transform(value)
transformed_dict={}
for key, value in name_dict.items():
transformed_dict[key]=transform(value)
#### wirte to excel example:
writer = pd.ExcelWriter("MyData.xlsx", engine='xlsxwriter')
for name, df in transformed_dict.items():
df.to_excel(writer, sheet_name=name)
writer.save()
if __name__=='__main__':
app.run(port=5000)
Your block:
#### wirte to excel example:
writer = pd.ExcelWriter("MyData.xlsx", engine='xlsxwriter')
for name, df in transformed_dict.items():
df.to_excel(writer, sheet_name=name)
writer.save()
should be part of your upload() function since that's where you define and fill transformed_dict. You just need to match the indentation there to the block above it.
The current error is coming up because it's trying to run that code as soon as you start your script, and transformed_dict doesn't exist at that point.
Related
I created a code (using Tkinter, Python3 and matplotlid) that could read data from different serial ports, save them to csv, then create graphs and finally preview data in GUI. The code was splited in two different scripts. The main script contained reading data, save data to csv an priview of data and the other script contained the graph creation.
Today I rewrote the code using the answer of #user2464430 here. The code is working, but I can't update the GUI. Opens once and then no refresh with new data.
The following code is a part of total code.
My code is:
from PIL import ImageTk, Image
import tkinter as Tk
import multiprocessing
from queue import Empty, Full
from time import strftime
import serial
import numpy as np
import matplotlib.pyplot as plt
from drawnow import *
from pylab import *
import pandas as pd
from datetime import timedelta
from datetime import datetime
import plotly.graph_objs as go
from plotly.subplots import make_subplots
import locale
import os
class GuiApp(object):
def __init__(self, image):
self.root = Tk.Tk()
self.root.resizable(width=False, height=False)
self.root.geometry("1600x800+0+0")
C = Canvas(self.root, bg="black", width=1600, height=800)
def BasicLabels():
....... # in this stage create multiple axis labels
ΥAxisLabels()
BasicLabels()
def ValueLabels():
....... # Read and munipulate datas from CSV file and print in in labels
ValueLabels()
C.pack()
def GenerateData(q): #Read Serial Ports and store data to CSV file
file_exists = os.path.isfile("BigData.csv")
header = [["Daytime,T1"]]
if not file_exists:
with open("BigData.csv", "a+") as csvfile:
np.savetxt(csvfile, header, delimiter=",", fmt="%s", comments="")
while True:
try:
ser1 = serial.Serial(port="COM4", baudrate=9600)
read_ser1 = ser1.readline()
if read_ser1 == "":
read_ser1 = "Missing Value"
else:
read_ser1 = ser1.readline()
read_ser1 = str(read_ser1[0 : len(read_ser1)].decode("utf-8"))
# print("COM4:", read_ser1)
ser1.close()
except:
print("Failed 1")
read_ser1 = "9999,9999,9999,9999,9999"
daytime = strftime(" %d-%m-%Y %H:%M:%S")
rows = [
daytime
+ ","
+ read_ser1.strip()
]
with open("BigData.csv", "a+") as csvfile:
np.savetxt(csvfile, rows, delimiter=",", fmt="%s", comments="")
CreateGraphs()
def CreateGraphs():
#Code to generate graph. Called every time i have new line in CSV.
if __name__ == "__main__":
# Queue which will be used for storing Data
q = multiprocessing.Queue()
q.cancel_join_thread() # or else thread that puts data will not term
gui = GuiApp(q)
t1 = multiprocessing.Process(target=GenerateData, args=(q,))
t1.start()
gui.root.mainloop()
t1.join()
The graphs are generating after while True in GenerateData.
All datas for labels and graphs are coming from CSV file and not directly from serial port.
Is it possible to update GUI with latest datas from CSV and created graphs?
Thank for your time.
Use Case
Within a Logic App, I create some data using an Azure Function with a Pandas DataFrame. After employing the Azure Function, I want to further process the data in .xlsx format within the Logic App. Therefore I need the Azure Function to return an .xlsx file.
Problem
I am unable to format the HTTPResponse of my Azure Function so that I can further process the .xlsx file within the Logic App. Basically I require the correct conversion from my Pandas DataFrame to the HTTPResponse.
What to do in convert_to_xlsx() (see below) to achieve the desired output?
Toy Example
import azure.functions as func
import logging
import pandas as pd
from openpyxl import Workbook
from openpyxl.utils.dataframe import dataframe_to_rows
def main(req: func.HttpRequest) -> func.HttpResponse:
df = pd.DataFrame(np.random.randint(0, 100, size=(2, 4)), columns=list('ABCD'))
excel = convert_to_xlsx(df)
return func.HttpResponse(excel, status_code=200)
def convert_to_xlsx(df):
# Create excel representation
wb = Workbook()
sheet = wb.active
for row in dataframe_to_rows(df, index=False, header=True):
sheet.append(row)
logging.info('sheet: ' + str(list(sheet.values))) # So far, so good.
# Convert for HTTPResponse
res = ''
res = do_something(sheet) # <---- What to do here?
return res
What I tried
I tried converting the data to openpyxl's Workbook, which worked fine. But then I did not know how to proceed from here to convert from a Workbook.
Also, there is this answer using xlrd, which I could not get to work for my use case. Additionally, xlrd does not support .xlsx anymore. Based on that post, I tried the following, which did not work as intended:
def convert_to_xlsx(df):
# ... see above
# Only returns column names without values.
# Also, apparently not the right format?
return f'{[row for row in sheet]}'
One option might be to return some kind of JSON response and then convert it back to an excel file within the logic app. But I hoped that I might be able to skip that and immediately return a .xlsx file from the function as HTTP payload.
In order to obtain an Excel file you also have to manipulate the header, see https://stackoverflow.com/a/67276395/7641854.
Without the changed header you will obtain a zip object I assume.
Thus a working example to return an Excelfile via Azure function looks like this:
import pandas as pd
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
d = {'col1': [1, 2], 'col2': [3, 4]}
df = pd.DataFrame(data=d)
buffer = io.BytesIO()
excel_buf = df.to_excel(buffer, index=False)
return func.HttpResponse(
buffer.getvalue(),
headers={"Content-Disposition": 'attachment; filename="test.xlsx"'},
mimetype='application/vnd.ms-excel',
status_code=200,
)
The approach could be to write the output to a buffer and return the buffer's content within the HTTPResponse
...
from io import BytesIO
def main(req: func.HttpRequest) -> func.HttpResponse:
df = pd.DataFrame(np.random.randint(0, 100, size=(2, 4)), columns=list('ABCD'))
buffer = BytesIO()
excel_buf = df.to_excel(buffer)
return func.HttpResponse(buffer.getvalue(), status_code=200)
However, due to concerns regarding file size and execution times for returning large files via http, I opted for uploading the resulting excel to an Azure BLOB storage, by using something like (snippet):
...
out_blob = BlobClient.from_connection_string(...)
excel_buf = df.to_excel(buffer)
out_blob.upload_blob(buffer.getvalue())
...
So, the goal is to create a webpage to load a .shp file into and get a summary of some calculations as a JsonResponse. I have prepared the calculations and everything and it works nicely when I add a manual path to the file in question. However, the goal is for someone else to be able to upload the data and get back the response so I can't hardcode my path.
The overall approach:
Read in a through forms.FileField() and request.FILES['file_name']. After this, I need to transfer this request.FILES object to DataSource in order to read it in. I would rather not upload the file on pc if possible but work directly from the memory.
forms.py
from django import forms
from django.core.files.storage import FileSystemStorage
class UploadFileForm(forms.Form):
# title = forms.CharField(max_length=50)
file = forms.FileField()
views.py
import json
import os
from django.http import Http404, HttpResponse, HttpResponseRedirect
from django.shortcuts import render
from django.template import loader
from django.contrib import messages
from django.views.generic import TemplateView
from django.http import JsonResponse
from django.conf import settings
from .forms import UploadFileForm
from . import models
from django.shortcuts import redirect
from gisapp.functions.functions import handle_uploaded_file, handle_uploaded_file_two
from django.contrib.gis.gdal import DataSource
from django.core.files.uploadedfile import UploadedFile, TemporaryUploadedFile
import geopandas as gpd
import fiona
def upload_file(request):
if request.method == 'POST':
form = UploadFileForm(request.POST, request.FILES)
if form.is_valid():
f = request.FILES['file']
# geo2 = gpd.read_file(f)
# print(geo2)
# f_path = os.path.abspath(os.path.join(os.path.dirname(f), f))
# f_path = TemporaryUploadedFile.temporary_file_path(UploadedFile(f))
# print(f_path)
# f_path = f.temporary_file_path()
# new_path = request.FILES['file'].temporary_file_path
# print(f'This is file path: {f_path}')
# print(f'This is file path: {new_path}')
# data = DataSource(f'gisapp/data/{f}') -- given an absolute path it works great
data = DataSource(f) -- constantly failing
# data = DataSource(new_path)
# print(f'This is file path: {f_path}')
layer = data[0]
if layer.geom_type.name == "Polygon" or layer.geom_type.name == "LineString":
handle_uploaded_file(request.FILES['file'])
elif layer.geom_type.name == "Point":
handle_uploaded_file_two(request.FILES['file'])
return JsonResponse({"Count": f"{handle_uploaded_file_two(request.FILES['file'])[0]}", "Bounding Box": f"{handle_uploaded_file_two(request.FILES['file'])[1]}"})
# return JsonResponse({"Count": f"{handle_uploaded_file(request.FILES['file'])[0]}", "Minimum": f"{handle_uploaded_file(request.FILES['file'])[1]}", "Maximum": f"{handle_uploaded_file(request.FILES['file'])[1]}"})
# instance = models.GeometryUpload(file_field=request.FILES['file'])
# instance.save()
# # return HttpResponseRedirect('/success/')
else:
form = UploadFileForm()
return render(request, 'upload.html', {'form': form})
Error I get:
django.contrib.gis.gdal.error.GDALException: Invalid data source input type: <class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
Now as you can see from the upload_file() in views.py, I tried a multitude of operations and when I add an absolute path, it works, but besides that I can't seem to upload the file to DataSource so that I can use it in my later analysis.
Looking at how Django handles this, it doesn't appear possible to work off of an in memory file. The path to the file is passed to the C API for OGR which then handles opening the file and reading it in.
A possible solution that I am trying myself is to have the user zip their shape files (.shp,.shx.,dbf etc.) beforehand. The zip file is then uploaded and unzipped. The shp files can then be read. Hope this helps
I face the same problem and my workaround was to save the file upload by the user in a temporary folder, then pass the absolute path of the temporary file to my DataSource. After finish all my process with the temporary file, I deleted.
The downside of this method is the execution time, is slow.
I want to create a web service using Python.
The service contents are as follows.
Receive music files from users
Analyze the received file and create a new music file. (The function implementation has already been completed using the python library.)
Output the newly created music file to the user.
The main goal is to analyze the music file and create a new one, and input and output the music file for this purpose is a secondary function. So I have no knowledge of creating web services and need help to implement this feature. I went to the internet and implemented it to some extent, but I get an error that I don't know why.
I Upload the code for app.py and __init__.py. I also uploaded the structure of the directory. It looks like something needs to be fixed to use the functions inside the controller. I've been searching the internet and YouTube for more than twelve hours, but I don't really know. I created a class named NoteConvertor and MarcovMatrix in the controller directory.
It runs until I go to http://127.0.0.1:5000/ and receive a file. Then I get the following error:
File "C:\Users\K\git\flask\gp\app.py", line 24, in wav_transform
sr, data = scipy.io.wavfile('static/file1.wav')
TypeError: 'module' object is not callable
-gp
|----controller
| MarcovMatrix.py
| MatrixBuilder.py
| NoteConvertor.py
|----static
| style.css
|----templates
| index.html
| upload.html
| wavplay.html
|----__init__.py
|----app.py
app.py
import flask
from flask import Flask, request, render_template
from werkzeug.utils import secure_filename
import scipy
import numpy as np
from scipy import misc
import pysynth as ps
from scipy.io import wavfile
app = Flask(__name__)
#app.route('/')
#app.route("/upload")
def index():
return render_template('index.html')
#app.route('/uploader', methods=['GET', 'POST'])
def wav_transform():
if request.method == 'POST':
f = request.files['file']
f.save(f'static/file1.wav')
sr, data = scipy.io.wavfile('static/file1.wav')
empty_notes = controller.NoteConvertor(data)
notes = empty_notes.convertor()
song = sum(notes, [])
matrix = controller.MarcovMatrix(song)
start_note = ['e4', 4]
random_song = []
for i in range(0, 100):
start_note = matrix.next_note(start_note)
random_song.append(start_note)
ps.make_wav(random_song, fn='static/random.wav')
return render_template('wavplay.html')
if __name__ == '__main__':
app.run(debug = True)
__init__.py
from gp.controller import MarcovMatrix
from gp.controller import MatrixBuilder
from gp.controller import NoteConvertor
I would like to tell you how to use the functions written in python (inside the controller) inside the flask.
I would be very grateful if you would like to know how to do what I want to implement. Please understand this question because learning the basics of the web is not my main goal.
This is an import issue. You have the choice of either
import scipy
...
sr, data = scipy.io.wavfile('static/file1.wav')
or
from scipy.io import wavfile
...
sr, data = wavfile('static/file1.wav')
My function was working perfectly fine few minutes ago. Did not modify the code, just installed PyAudio. I get the error as per subject. It doesn't matter if run it from command line or IDE, same error. Any ideas?
def DataFinder():
#imports and initialize
import pandas as pd
import tkinter as tk
finder = tk.Tk()
finder.withdraw()
__dataFlag = False
#ask user to select a file
__path = tk.filedialog.askopenfilename()
#check the extension to handle reader
#for csv files
if __path.endswith('.csv')==True:
df = pd.read_csv(__path,header=0)
return df
__dataFlag = True
#and for excel files
elif __path.endswith('.xls')==True:
df = pd.read_excel(__path,header=0)
return df
__dataFlag = True
#if a file is not a supported type display message and quit
else:
__dataFlag = False
#check if there is a data to be returned
if __dataFlag==True:
return df
else:
print('The file format is not supported in this version.')
Explicitly import of filedialog can solve the problem.
So, you just need to add this line to your codes:
import tkinter.filedialog
You can find more information at Why tkinter module raises attribute error when run via command line but not when run via IDLE?
the following code didn't work for me:
import tkinter as tk
import tkinter.filedialog
But the following did work:
import tkinter
import tkinter.filedialog
and also this:
import tkinter.filedialog
import tkinter as tk
Hope this helps
Note
As mentioned by Vaidøtas I., you can't import filedialog from tkinter. Because you did not import the original tkinter but an aliased version tk.