I am running a python3/flask app on gunicorn in heroku. This app presents the user with a list of items, pulled in from an API call, for the user to accept or reject. Depending on whether the user hits the accept or reject links associated with each item, the app appends the item to either an internal list of accepted items are one of rejected items.
Currently I'm storing each list (suggestions, acceptances and rejections) as a pandas dataframe object within the app.
i.e. I initialise my app with an empty data frame:
app = Flask(__name__)
app.accepted = pd.DataFrame()
app.suggested = pd.DataFrame()
populate the suggestions data frame with an API call:
#app.route("/get_suggestions")
def get_suggestions():
app.suggested = <some data returned from an API>
then append a suggested item to the accepted dataframe once an 'accept' link is hit:
#app.route("/accept/<suggest_id>")
def accept_item(suggest_id):
app.accepted(len(app.accepted)) = app.suggested.loc[int(suggest_id)]
This all works fine running on gunicorn on my local miniconda virtual environment (running "heroku local web") but when deployed on heroku I keep getting "Internal Server Error". When i look at the logs, it looks like the app's internal variables (e.g. app.suggested) are not being preserved, such that when accept_item is run, app.suggested is always empty. Why would they be preserved on the local version but not on the heroku deployment?
What's the simplest way of preserving this state? I'd like a small number of multiple users to be able to use the app and each build their own temporary lists. Do i need to use SQLite to preserve state? Do i need to drop a cookie into the user's browser so i can tell different users apart? I'd prefer not to require users to create an account on my site.
You can do one of two things:
Either save this state into cookies (which I would not recommend), or
Store this data into a database like Heroku Postgres (for instance), and use SQLAlchemy or some other ORM to retrieve and store that data.
Heroku dynos (which run your web application) are stateless. They reboot at will, don't persist disk, etc.
Storing global state won't work because those variables you're defining are changing on every incoming request.
You could use the app context to store this state, but since you're running on Heroku this is asking for trouble since the dynos will restart at random.
Related
I recently just deployed a web app in Azure App Service. This web app uses 2 separate database (2 connection strings) to be able to function fully. Upon deploying and browsing the web app, the first database got automatically created but the 2nd one was not. I was expecting for both to be automatically created but I can't make it happen for the 2nd database.
I checked the connection strings and it is correct. What do I need to so the 2nd database gets created as well?
EDIT:
Code was requested but I am not really sure what relevant code I need to post. This is the only thing I can think of that might be involved in the database creation.
in ConfigureServices:
services.AddDbContext<FirstDbContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("FirstConnection")));
services.AddDbContext<SecondDbContext>(options =>
options.UseSqlServer(Configuration.GetConnectionString("SecondResourceConnection")));
Not sure if this is the correct code I need to post relating to this.
EDIT 2:
One thing to note is that this worked in my local, both DBs were created.
I'm trying to build a NodeJS REST API project based on the so called "micro architecture" (basically multiple smaller NodeJS projects that can run totally independent, but at the same time work together).
Currently users are able to upload images from the app, and my NodeJS backend then processes and saves them appropriately.
Now, what I want to do is the following:
User selects an image to upload from the app -> The app makes a request to the "Main API" endpoint -> The Main API endpoint then forwards this request to the "Image Service" -> Once the Image Service (which is a totally different server) has successfully finished, it should return the URL where the image is stored to the Main API server endpoint, which will then return the info back to the app.
My question is, how do I forward the image upload request from one server to another? Ideally, I don't want the Main API to store the image temporarily and then make a request to the Image Service.
What I'd like is try and forward the data the Main API receives straight to the Image Service server. I guess you could say I want to "stream" the data from one place to another without having to temporarily store on disk or memory. I literally just want it to "tunnel" from one server to another.
Is this possible and is this an efficient way? I just want 1 central point for the app to access, I don't want it to know about this Image Service server. I'd like the app to only ever make requests to the Main API, which will then call my other little services as required.
I'm using NodeJS, Express, Multer (for image uploads) and Digital Ocean hosting (if that should make any difference at all).
What you would basically be doing is setting up a proxy server that will pass requests straight through to another machine and back. There are a few libraries out there to help with this, and this article in particular http://blog.vanamco.com/proxy-requests-in-node-js/ will explain how to go about setting it up even though they are really just trying to get around HTTPS, the same concept applies here.
In short, you get the file upload POST, and then immediately just make that same request to another server and when the response is returned, immediately return it back to the front end. Your entry point can be set up as a hub, and you can proxy requests through to other servers or even just handle them on the same server if necessary.
I have built a node.js app with this structure:
In app.js:
var myList = ['0'];
app.get('/webpage',function(req,res){
console.log(myList);
res.render('webpage.ejs', {exps: myList});
});
On "webpage" I can display myList and there is also a form that allows me to add elements to myList. Let's say I append '1' to myList through this form.
I have the following problem I don't know how to debug:
locally on my computer, this app works fine: I can see ['0','1'] in my console each time I refresh "webpage".
online on Heroku, when I refresh "webpage" sometimes I see ['0','1'] sometimes I see just ['0'] and a couple of refresh later I see ['0','1'] again: it is like myList randomly oscillates between its default value when the app was first launched and the value that was specified later.
I use the same npm and node versions locally and on heroku and the same versions of dependencies. To my knowledge I have the same environment locally and heroku, so I have no idea where this problem could be coming from.
You may be running multiple instances of on Heroku, in which case each request might be assigned to a different instance, each with its own process and memory space.
I believe Heroku also shuts down instances after a period of inactivity, so that might be an issue too.
If you intend to persist something, how about storing it in a database?
I'm using Firebase and the SimpleLogin to allow users to login via Google, Twitter etc.
I'd like to use some of the thirdpartyuserdata object to create a user profile for my application which runs on Node.
Currently I'm posting this data to the server so that I can add to it and create the profile object, but I wondered if there's a better way of doing this - is there something I can call server side to get this thirdpartyuserdata without having to post it from the client?
Start by considering that your "server" is actually just another consumer of Firebase data. Since FirebaseSimpleLogin is simply a token generator with some fancy tools for doing OAuth, and because this happens completely client-side, there is nothing to consume about this.
If you want to consume the data at the server, you will either need to POST it, as you have done, or use Firebase to transfer the information. You'll find that a queue approach can save you a large amount of code, as this allows you to use Firebase as the API, and avoid creating RESTful services in Node, and all the baggage that comes with that.
The idea of a queue is simply that you push data into Firebase at one client and read it out (and probably delete it) at the intended recipient (in this case your node worker).
I am writing service using NodeJS + Restify. I have split each actual service into separate file (what, I assume, everyone is doing). They all are going to be using mysql database so I thought I could open a single connection to database which could be used by each service rather than opening connections every time a request is done.
The problem is that I don't seem to find a way to pass user data. By user data I mean any custom data that would be accessible by every service callbacked by the server.
I primarily use NodeJS + Express, but having looked through some of the documentation of Restify, I believe you could use the authorization parser (under Bundled Plugins on their site: click here to go there)
I think that would be the most basic way to pass user data.
I haven't tested it but, I believe you'd just add this to use it:
server.use(restify.authorizationParser());
You could then access the user data with:
//This is based on the structure of req.authorization in the documentation.
req.authorization.basic.user
I believe you could set new user data (when the user logs in or something) like:
req.authorization.id = 'id'