How to define task in node-celery ? - node.js

I am using node-celery.
I just need to implement scheduling task so that Task can run on the background at the specific time.
I am confused how can I define my task, currently I am defining task at the same file where I am implementing node-celery.
const celery = require('node-celery');
let client = celery.createClient({
CELERY_BROKER_URL: 'amqp://guest:guest#localhost:5672//',
CELERY_RESULT_BACKEND: 'amqp://'
});
In the above code, I just require node-celery and then created a client for amqp.
Now I have to connect client and then call my task send_batch_email_using_mailgun with some parameter.
client.on('connect', function() {
client.call('send_batch_email_using_mailgun', {
campaign_data: campaign_data,
subject: subject,
template: template
}, {
eta: new Date(Date.now() + 120000)
});
});
Here _send_batch_email_using_mailgun_ is the task which is defined below the code in the same file with some parameter.
I want that my function _send_batch_email_using_mailgun_ should be called after a certain time. My code is not working I think I have to define my task function elsewhere but I don't know where to define them.
Do I need to create my task in python file? If yes then how can I import them in my js file?

You can start small and use the setInterval() api to make a scheduled task.
Let's say your task is to send email to all users.
Define it in a function like this:
function sendScheduledEmails() {
// Get all the emails
// Send emails with your provider
}
Test it if it works by calling the func, but just putting it and running your server once:
sendScheduledEmails()
When you see your function is working, use the api:
setInterval(sendScheduledEmails, 120000)
Check if your app is doing the task within the intervals. Once you see it's working, you can look into other modules and further tweak it.
If you want to take action after something happens. Let's say you want to send email after a user is registered.
Then when the user registers, you can emit an event like this:
const EventEmitter = require('events');
class MyEmitter extends EventEmitter {}
const myEmitter = new MyEmitter();
// User registered
// Redirect to success page
myEmitter.emit('userRegistered');
This will fire an event everytime that action happens, and you can listen to it.
Then in your code you can do:
myEmitter.on('userRegistered', function() {
setInterval(sendScheduledEmails, 120000)
})

TL;DR
its a python function.
for more info see the docs
details
the lingo here is a bit confusing so i'll try to clear it up:
your celery client is actually connecting to the broker (which is the task queue).
The broker will recieve a message with the function name and parameters you wish to run.
The celery worker is a python process which pulls the messages from the broker and then executes the function you requested.
The implementation of send_batch_email_using_mailgun only needs to be known to the celery worker.

Related

Firebase Functions: How to maintain 'app-global' API client?

How can I achieve an 'app-wide' global variable that is shared across Cloud Function instances and function invocations? I want to create a truly 'global' object that is initialized only once per the lifetime of all my functions.
Context:
My app's entire backend is Firestore + Firebase Cloud Functions. That is, I use a mix of background (Firestore) triggers and HTTP functions to implement backend logic. Additionally, I rely on a 3rd-party location service to continually listen to location updates from sensors. I want just a single instance of the client on which to subscribe to these updates.
The problem is that Firebase/Google Cloud Functions are stateless, meaning that function instances don't share memory/objects/state. If I call functionA, functionB, functionC, there's going to be at least 3 instances of locationService clients created, each listening separately to the 3rd party service so we end up with duplicate invocations of the location API callback.
Sample code:
// index.js
const functions = require("firebase-functions");
exports.locationService = require('./location_service');
this.locationService.initClient();
// define callable/HTTP functions & Firestore triggers
...
and
// location_service.js
var tracker = require("third-party-tracker-js");
const self = (module.exports = {
initClient: function () {
tracker.initialize('apiKey')
.then((client)=>{
client.setCallback(async function(payload) {
console.log("received location update: ", payload)
// process the payload ...
// with multiple function instances running at once, we receive as many callbacks for each location update
})
client.subscribeProject()
.then((subscription)=>{
subscription.subscribe()
.then((subscribeMsg)=>{
console.log("subscribed to project with message: ", subscribeMsg); // success
});
// subscription.unsubscribe(); // ??? at what point should we unsubscribe?
})
.catch((err)=>{
throw(err)
})
})
.catch((err)=>{
throw(err)
})
},
});
I realize what I'm trying to do is roughly equivalent to implementing a daemon in a single-process environment, and it appears that serverless environments like Firebase/Google Cloud Functions aren't designed to support this need because each instance runs as its own process. But I'd love to hear any contrary ideas and possible workarounds.
Another idea...
Inspired by this related SO post and the official GCF docs on stateless functions, I thought about using Firestore to persist a tracker value that allows us to conditionally initialize the API client. Roughly like this:
// read value from db; only initialize the client if there's no valid subscription
let locSubscriberActive = await getSubscribeStatusFromDb();
if (!locSubscriberActive) {
this.locationService.initClient();
}
// in `location_service.js`, do setSubscribeStatusToDb(); // set flag to true when we call subscribe(). reset when we get terminated
The problem faced: at what point do I unset/reset that value? Intuitively, I would do so the moment the function instance that initialized the client gets recycled/killed. However, it appears that it is not possible to know when a Firebase Cloud Function instance is terminated? I searched everywhere but couldn't find docs on how to detect such an event...
What you're trying to do is not at all supported in Cloud Functions. It's important to realize that there may be any number of server instances allocated for each deployed function. That's how Cloud Functions scales up and down to match the load on the function in a cost-effective way. These instances might be terminated at any time for any reason. You have no indication when an instance terminates.
Also, instances are not capable of performing any computation when they are idle. CPU resources are clamped down after a function terminates, and are spun up again when the next function is invoked on that instance. You can't have any "daemon" code running when a function is not actively being invoked. I don't know what your locationService does, but it is certainly doing nothing at all after a function terminates, regardless of how it terminated.
For any sort of long-running or daemon-like code, Cloud Functions is not a suitable product. You should instead consider also using another product that lets you run code 24/7 without disruptions. App Engine and Compute Engine are viable alternatives, and you will have to think carefully about if and how you want their server instances to scale with load.

Send request progress to client side via nodejs and express

I am using this (contentful-export) library in my express app like so
const app = require('express');
...
app.get('/export', (req, rex, next) => {
const contentfulExport = require('contentful-export');
const options = {
...
}
contentfulExport(options).then((result) => {
res.send(result);
});
})
now this does work, but the method takes a bit of time and sends status / progress messages to the node console, but I would like to keep the user updated also.. is there a way I can send the node console progress messages to the client??
This is my first time using node / express any help would be appreciated, I'm not sure if this already has an answer since im not entirely sure what to call it?
Looking of the documentation for contentful-export I don't think this is possible. The way this usually works in Node is that you have an object (contentfulExport in this case), you call a method on this object and the same object is also an EventEmitter. This way you'd get a hook to react to fired events.
// pseudo code
someLibrary.on('someEvent', (event) => { /* do something */ })
someLibrary.doLongRunningTask()
.then(/* ... */)
This is not documented for contentful-export so I assume that there is no way to hook into the log messages that are sent to the console.
Your question has another tricky angle though. In the code you shared you include a single endpoint (/export). If you would like to display updates or show some progress you'd probably need a second endpoint giving information about the progress of your long running task (which you can not access with contentful-export though).
The way this is usually handled is that you kick of a long running task via a certain HTTP endpoint and then use another endpoint that serves infos via polling or or a web socket connection.
Sorry that I can't give a proper solution but due to the limitation of contentful-export I don't think there is a clean/easy way to show progress of the exported data.
Hope that helps. :)

How to use newrelic.createBackgroundTransaction without a handle function

I am looking into using newrelic APM to monitor certain parts of our codebase.
I want to watch transactions that are not simple HTTP calls, but background processes. These transactions are completed by worker processes and we want to monitor them in the main part of the app.
Pseudo code:
var fork = childProcess.spawn('node', ['--harmony', 'path-to-worker.js', args]);
fork.stdout.on('data', function(data) {
// a finished transaction
// this fires most likely more than once
});
We basically need something like newrelic.createBackgroundTransaction() that can log a transaction immediately, without having to pass it a function to execute and time (I can do that myself).
Can I do something like this on the free tier of newrelic?

How to lock (Mutex) in NodeJS?

There are external resources (accessing available inventories through an API) that can only be accessed one thread at a time.
My problems are:
NodeJS server handles requests concurrently, we might have multiple requests at the same time trying to reserve inventories.
If I hit the inventory API concurrently, then it will return duplicate available inventories
Therefore, I need to make sure that I am hitting the inventory API one thread at a time
There is no way for me to change the inventory API (legacy), therefore I must find a way to synchronize my nodejs server.
Note:
There is only one nodejs server, running one process, so I only need to synchronize the requests within that server
Low traffic server running on express.js
I'd use something like the async module's queue and set its concurrency parameter to 1. That way, you can put as many tasks in the queue as you need to run, but they'll only run one at a time.
The queue would look something like:
var inventoryQueue = async.queue(function(task, callback) {
// use the values in "task" to call your inventory API here
// pass your results to "callback" when you're done
}, 1);
Then, to make an inventory API request, you'd do something like:
var inventoryRequestData = { /* data you need to make your request; product id, etc. */ };
inventoryQueue.push(inventoryRequestData, function(err, results) {
// this will be called with your results
});

Callback hook into AWS APIs

Does anyone know of a way or has anyone devised a clever workaround to place a callback/hook into the Amazon APIs (http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/query-apis.html) such that for actions like create instance, one can simply be notified via the callback when the instance is in the running state?
I'm thinking that I could write a loop in node.js that simply checks for the desired state and eventually timesout after a certain # of requests but I would like to hear better programmatic approaches :)
Unless the AWS APIs support some kind of notification endpoint (I'm not very familiar with the APIs) you're probably stuck with polling. However, you could use an EventEmitter to hide this behind a clever API that exposes a callback. Pseudo-ish code:
// aws_server.js
var EventEmitter = require('events').EventEmitter;
var util = require('util');
function AwsServer(some_data) {
this.data = some_data;
EventEmitter.call(this);
};
util.inherits(AwsServer, EventEmitter);
AwsServer.prototype.createInstance = function() {
// Do something with an API to create an EC2 instance
console.log("Creating instance, data:", this.data);
// Here, you would begin polling for state changes, etc. waiting for
// the server to change state. We will simulate this with a setTimeout call.
setTimeout(function() {
this.emit('running');
}.bind(this), 3000);
};
module.exports = AwsServer;
// somewhere_else.js
var AwsServer = require('./aws_server')
var newServer = new AwsServer('some_data');
newServer.on('running', function() {
console.log('New instance is running');
});
newServer.createInstance();
Your best bet would be to add a shell script on the servers init.d, which will run whenever the server is stopped or started.
Probably useless for this question but other ways of programmatically detecting whether an instance on is by using Amazon's EC2 shell tools:
ec2-describe-instance-status <ec2 instance id>
As described here. Which will return blank if the machine is not running, and data about it if it is.

Resources