I'm fairly new at google cloud and node. Based on Google's recommendation (the requirement to watch Firebase at all times) I deployed managed VM node app, instead of just app engine. There are now 22-23 instances every time I deploy. Is this expected? I thought it would only scale when necessary.
This node app has a method which watches Firebase variables and change, in turn, the script fires off a notification.
What happens now is that multiple notifications are being fired and I only expect one. I suspect it's because there are multiple instances of this app.
What is the right way to do this so that only one is watching?
Thanks.
You can use the method suggested by google for flexible server environments and firebase. https://cloud.google.com/solutions/mobile/mobile-firebase-app-engine-flexible and https://cloudplatform.googleblog.com/2016/06/learn-to-build-a-mobile-backend-service-with-Firebase-and-App-Engine.html .
Have the instance "claim users" by transactioning it's instance ID at a location where the user can reach it and send updates to that instance by setting the instanceID in the path.
Related
Our existing system uses App Services with API controllers.
This is not a good setup because our scaling support is poor, its basically all or nothing
I am looking at changing over to use Azure Functions
So effectively each method in a controller would become a new function
Lets say that we have a taxi booking system
So we have the following
Taxis
GetTaxis
GetTaxiDrivers
Drivers
GetDrivers
GetDriversAvailableNow
In the app service approach we would simply have a TaxiController and DriverController with the the methods as routes
How can I achieve the same thing with Azure Functions?
Ideally, I would have 2 function apps - Taxis and Drivers with functions inside for each
The problem with that approach is that 2 function apps means 2 config settings, and if that is expanded throughout the system its far too big a change to make right now
Some of our routes are already quite long so I cant really add the "controller" name to my function name because I will exceed the 32 character limit
Has anyone had similar issues migrating from App Services to Azure Functions>
Paul
The problem with that approach is that 2 function apps means 2 config
settings, and if that is expanded throughout the system its far too
big a change to make right now
This is why application setting is part of the release process. You should compile once, deploy as many times you want and to different environments using the same binaries from the compiling process. If you're not there yet, I strongly recommend you start by automating the CI/CD pipeline.
Now answering your question, the proper way (IMHO) is to decouple taxis and drivers. When requested a taxi, your controller should add a message to a Queue, which will have an Azure Function listening to it, and it get triggered automatically to dequeue / process what needs to be processed.
Advantages:
Your controller response time will get faster as it will pass the processing to another process
The more messages in the queue / more instances of the function to consume, so it will scale only when needed.
Http Requests (from one controller to another) is not reliable (unless you implement properly a circuit breaker and a retry policy. With the proposed architecture, if something goes wrong, the message will remain in the queue or it won't get completed by the Azure function and will return to the queue.
Heroku does not support health checks on its own. It will restart services that crashed, but there is nothing like health checks.
It sometimes happen that service become unresponsive, but the process is still running. In most of modern cloud solution, you can provide health endpoint which is periodically called by the cloud hosting service and if that endpoints return either error or not at all, it will shut down such service and start new one.
That seems like industrial standard these days, but I am unable to find any solution to this for Heroku. I can even use external service with Heroku CLI, but just calling some endpoint is not sufficient - if there are multiple instances, they all share same URL and load balancer calls one of them randomly -> therefore it is possible to not hit failed instance at all. Even when I hit it, usually the health checks have something like "after 3 failed health checks in a row restart that instance", which is highly unprobable if there are 10 instances and one of it become unhealthy.
Do you have any solution to this?
You are right that this is industry standard and shame that it's not provided out of box.
I can think of 2 solutions (both involve running some extra code that does all of this:
a) use heroku API which allows you to get the IP of individual dynos, and then you can call each dyno how you want
b) in each dyno instance you can send a request to webserver like https://iamaalive.com/?dyno=${process.env.HEROKU_DYNO_ID}
I have an Azure function running on a timer every few minutes that after a varied amount of time of running will begin to fail every time it runs because of an external API and hitting the restart button manually in the azure portal fixes the problem and the job works again.
Is there a way to either get an azure function to restart itself or have something externally restart an azure function via a web hook or API request or running on a timer
I have tried using Azures API Management service which can be used to restart other kinds of app services in azure but it turns out there is no functionality in the API to request a restart of an azure function, Also looked into power shell and it seems to be the same problem you can restart different app services but not azure functions
i have tried working with the API
https://learn.microsoft.com/en-us/rest/api/azure/
Example API request where you can list functions within an azure function
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/sites/{name}/functions?api-version=2016-08-01
but there is no functionality to restart an azure function from what i have researched
Basically i want to Restart the Azure function as if i was to hit this button
Azure functions manual stop/start and restart buttons in azure portal
because there is a case where the job gets into a bad state every time it runs because of an external API i have no control over and hitting restart manually gets the job going again
Another way to restart your function is by using the "watchDirectories" setting in the host.json file. If your host.json looks like this:
{
"version": "2.0",
"watchDirectories": [ "Toggle" ]
}
You could toggle a restart by using following statement in a function:
System.IO.File.WriteAllText("D:/home/site/wwwroot/Toggle/restart.conf", DateTime.Now.ToString());
Looking at the logs, the function reloads as it has detected the file change in the directory:
Watched directory change of type 'Changed' detected for 'D:\home\site\wwwroot\Toggle\restart.conf'
Host configuration has changed. Signaling restart
Azure functions by their nature are called upon an event. That may be a timer, a trigger or invocation like a HTTP event. They cannot be restarted per se, i.e. if you a function throws and exception, you cannot find the specific instance and re-run it using the out of the box functionality.
However, you can engineer your way to a more reliable solution:
Replay the event that invoked the function (i.e. kick it off again)
For non-sensitive data, log the payload of the function and create a another function that can be called on demand to re-run it. I.e. you create a proxy to "re-invoke" the function.
Harden your code by implementing a retry policy. See Polly.
Add a service bus in to your architecture. Have a simple function to write the call payload to a message bus payload. Have another function to pick up the payload and process it more extensively where there may be unreliable integrations etc). That way if the call fails you can abandon and dead letter failures for later reprocessing.
Consider using Durable Function Extensions and leveraging the durable patterns, these can help make your functions code more robust and manage state.
Why don't you try below ARM API. Since Azure function also fall under App service category, sometimes this may be helpful,
https://learn.microsoft.com/en-us/rest/api/appservice/webapps/restart
I am using Firebase in my App in Nodejs. I have JS API which takes imprints of website users(kind of Google Analytics) and from there it updates Data in Firebase and event Listeners binded in Nodejs compute the analytics.
Problem:
Currently, App is hosted on GAE and I have to manage minimum 300k concurrent connections.
I am facing issue with scaling the App, as Firebase works on sockets, so if multiple instances are working, it makes incur duplicate processing of the same request depending on the number of instances working.
Sol 1 - Queues: I tried using firebase-queues but it could not take the heavy load and logs to Firebase Disconnect for all the queues, as many tasks got queued which result to timeout
Sol 2 - Cloud Functions: I suppose cloud functions in background uses GAE, which will incur the same duplicate request issue.
How actually functions scale? does it add CPU & memory or add servers?
Can anyone suggest me? how should I scale with Firebase? else I will be left with last option to remove Firebase and make it REST based.
I would like to have a Node.js app running in multiple instances in Cloud Foundry.
The app has - among others - a scheduler ("node-schedule": "^0.2.7") which writes a status to a mongo-db once every minute. Now with multiple instances of the app, I get multiple entries - for each instance one - in the database.
I'm looking for something to synchronize instances to ensure just one instance "gets the lock" to write to the database once every minute. I already found suggestions to use singleton frameworks or middleware but they are request-centric, not "from the inside" of the app - the scheduler.
Is there a solution for this?
Each application instance will have an environment variable set, CF-INSTANCE-INDEX, to expose the instance index. You could make only the instance with index '0' responsible for scheduling logs entries.
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#CF-INSTANCE-INDEX