Can I save instance-specific functions as environment variables (and should I)? - node.js

Each instance of my app has a different policy for paying commission to sales reps. Currently I'm using environment variables such as:
// Client 1
SALES_COMMISSION_BASIS = "PERCENTAGE"
SALES_COMMISSION_PERCENT = 15%
// Client 2
SALES_COMMISSION_BASIS = "FLAT_RATE"
SALES_COMMISSION_AMOUNT = £42
I then have a function that returns the commission calculated according to the basis
const commission = (totalFee) => {
switch (SALES_COMMISSION_BASIS) {
case "PERCENTAGE":
return (totalFee * SALES_COMMISSION_PERCENT)
break;
case "FLAT_RATE":
return (SALES_COMMISSION_AMOUNT)
break;
}
}
I could of course simplify this to always return (totalFee * SALES_COMMISSION_PERCENT + SALES_COMMISSION_AMOUNT) but this would still mean that I am including logic for percent-based commission in a flat-rate client and vice-versa, which is what I am seeking to avoid.
What I'd like to do instead is save the commission function into my environment variables.
// Client 1
SALES_COMMISSION_FUNCTION = "(totalFee) => totalFee * 0.15"
// Client 2
SALES_COMMISSION_FUNCTION = "() => 42"
While none of this is particularly language specific, it's worth noting that I'm working with an Express app in Node JS, and I'm currently using dotenv to store environment variables.
My questions are:
Is saving the function to an environment the correct approach?
How would I implement this in a NodeJS app?

Related

Does a NodeJS app has to be purely stateless in order for it to be replicated?

In my Node application, there are Values that the User can define. These Values, once created, can change, either from a user-triggered action or from something else, for example, a MQTT message received on the server. A Value can change very sporadically or a few times per second.
class Value {
constructor(valueStore, id, name) {
this.valueStore = valueStore;
this.id = id;
this.name = name;
}
// ...
}
Because some Values can change many times per second, I don't save the Values to MongoDB Atlas every time they change (data transfer costs are quite expensive). Instead, I have a "ValueStore", which is basically a global object where all my Values are stored with their current value. In case my app goes down, I save the contents of the ValueStore to Atlas every 5 seconds, which is much less expensive.
// This is a global object
class ValueStore {
constructor() {
this.values = [];
this.bufferUpdateInterval = setInterval(() => {
// Every 5s, save values in ValueStore collection
}, VALUES_UPDATE_DB_FREQUENCY);
}
setValue(value) {
this.value = value;
}
// ...
}
I haven't yet implemented zero-downtime deployment. When I update my application, I have to bring the app down. When I put it back up, I want all the Values to be initialized with the Values they had when I put the app down. So, before anything else, I have to query the collection in which I save my Values every 5 seconds in order to reinstantiate my ValueStore.
// When my app starts:
const initializeValueStore = async (streams, variables) => {
const values = await Values.getLastValues(); // call to get the last known Values
for (let i = 0; i < values.length; i++) {
const lastKnownValue = // ...
valueStore.addValue(valueStore, values[i].id, values[i].name, lastKnownValue);
}
}
Now, I want to scale my application and implement patterns such as zero-downtime deployment, which implies replicating my app across several nodes. The more I think about it and the more I am under the impression that I won't be able to do that until I make my app stateless (i.e.: that I get rid of the ValueStore).
Am I right to think that my app should be stateless in order for it to be replicated and if so, how could I do differently what I'm currently doing with my ValueStore? Could a Redis cache come into play?

How to Increment Value Atomically with Redis

I have below code in nodejs:
const decrease = async (userId, points) => {
const user = await redisClient.hgetall(userId);
if(user.points - points >= 0) {
await redisClient.hset(userId, userId, user.points - points);
}
}
since async/await is not blocking the execution, if there are multiple requests for the same userId, the code is not running as atomically. That means the user points may be decreased multiple times even there is not enough point left on users account. How can I make the method run atomically?
I have checked redis multi command and it works for multiple redis statements. But in my case, I need to calculate the user points which is not part of redis command. So how to make them run as an atomic function.
I also read the INCR pattern: https://redis.io/commands/incr
But it doesn't seem to fix my issue. The patterns listed there need to work with expire which I don't have such requirement to give a specific timeout value.

Use the power of (Redis) server-side Lua scripts by calling EVAL. It should probably look something like this:
const lua = `
local p = redis.call('HGET',KEYS[1],'points')
local d = p - ARGV[1]
if d >= 0 then
redis.call('HSET', KEYS[1], 'points', d)
end`
const decrease = async (userId, points) => {
await redisClient.eval(lua, 1, userId, points);
}

Automatic run function daily at specific time

I'm doing a school project about CMS System to help operate my school website. It use 3 database:
MongoDB (Head database, store all information)
Redis (Store the menu of website)
Elasticsearch (Store posts)
Currently when I insert/edit/delete data, I also insert/edit/delete to a related database. But my mentor want me to write a function that let's system auto sync data between those 3 database at a specific time (user can choose when).
My server uses Node JS to operate, this requirement is new for me, never heard about it before. My new approach is:
Add 1 flag to database field
Select all rows which contain flag == true.
Sync data
But I don't know how to auto run above function at a specific time. I hope you guys can help me to optimize my new flow and solve the sync problem.
Thank you all !!!
Edit 1: Change tittle.
Edit 2: I found a solution from this topic: Running a function everyday midnight, is there anyway to re-run this function if error occur when sync data. Something like this:
function syncDataAtMidNight(){
var now = new Date();
var night = new Date(
now.getFullYear(),
now.getMonth(),
now.getDate() + 1, // the next day, ...
0, 0, 0 // ...at 00:00:00 hours
);
var msToMidnight = night.getTime() - now.getTime();
setTimeout(function() {
// is this a correct way to loop, in case of error when sync ???
myFunctionSync(err, resp) {
if(err){
myFunctionSync();
} else {
changeFlagToFalse();
syncDataAtMidNight();
}
}
}, msToMidnight);}

You should consider to use cron, he have the significant advantage to be at system level, he can get the code error return and mostly launch/relaunch node if necessary.

How do you implement AWS Elasticache auto discovery for node.js

I'm a node noob and trying to understand how one would implement auto discovery in a node.js application. I'm going to use the cluster module and want each worker process to be kept up to date (and persistently connected to) the elasticache nodes.
Since there is no concept of shared memory (like PHP APC) would you have to have code that runs in each worker, that wakes up every X seconds and somehow updates the list of IP's and re-connects the memcache client?
How do people solve this today? Example code would be much appreciated.

Note that at this time, Auto Discovery is only available for cache clusters running the memcached engine.
For Cache Engine Version 1.4.14 or Higher you need to create a TCP/IP socket to the Cache Cluster Configuration Endpoint (or any Cache Node Endpoint) and send this command:
config get cluster
With Node.js you can use the net.Socket class to to that.
The reply consists of two lines:
The version number of the configuration information. Each time a node is added or removed from the cache cluster, the version number increases by one.
A list of cache nodes. Each node in the list is represented by a hostname|ip-address|port group, and each node is delimited by a space.
A carriage return and a linefeed character (CR + LF) appears at the end of each line.
Here you can find a more thorough description of how to add Auto Discovery to your client library.
Using the cluster module you need to store the same information in each process (i.e. child) and I would use "setInterval" per child to periodically check (e.g. every 60 seconds) the list of nodes and re-connect only if the list has changed (this should not happen very often).
You can optionally update the list on the master only and use "worker.send" to update the workers. This could keep all the processes running in a single server more in sync, but it would not help in a multi server architecture, so it is very important to use consistent hashing in order to be able to change the list of nodes and loose the "minimum" amount of keys stored in the memcached cluster.
I would use a global variable to store this kind of configuration.
Thinking twice you can use the AWS SDK for Node.js to get the list of ElastiCache Nodes (and that works for the Redis engine as well).
In that case the code would be something like:
var util = require('util'),
AWS = require('aws-sdk'),
Memcached = require('memcached');
global.AWS_REGION = 'eu-west-1'; // Just as a sample I'm using the EU West region
global.CACHE_CLUSTER_ID = 'test';
global.CACHE_ENDPOINTS = [];
global.MEMCACHED = null;
function init() {
AWS.config.update({
region: global.AWS_REGION
});
elasticache = new AWS.ElastiCache();
function getElastiCacheEndpoints() {
function sameEndpoints(list1, list2) {
if (list1.length != list2.length)
return false;
return list1.every(
function(e) {
return list2.indexOf(e) > -1;
});
}
function logElastiCacheEndpoints() {
global.CACHE_ENDPOINTS.forEach(
function(e) {
util.log('Memcached Endpoint: ' + e);
});
}
elasticache.describeCacheClusters({
CacheClusterId: global.CACHE_CLUSTER_ID,
ShowCacheNodeInfo: true
},
function(err, data) {
if (!err) {
util.log('Describe Cache Cluster Id:' + global.CACHE_CLUSTER_ID);
if (data.CacheClusters[0].CacheClusterStatus == 'available') {
var endpoints = [];
data.CacheClusters[0].CacheNodes.forEach(
function(n) {
var e = n.Endpoint.Address + ':' + n.Endpoint.Port;
endpoints.push(e);
});
if (!sameEndpoints(endpoints, global.CACHE_ENDPOINTS)) {
util.log('Memached Endpoints changed');
global.CACHE_ENDPOINTS = endpoints;
if (global.MEMCACHED)
global.MEMCACHED.end();
global.MEMCACHED = new Memcached(global.CACHE_ENDPOINTS);
process.nextTick(logElastiCacheEndpoints);
setInterval(getElastiCacheEndpoints, 60000); // From now on, update every 60 seconds
}
} else {
setTimeout(getElastiCacheEndpoints, 10000); // Try again after 10 seconds until 'available'
}
} else {
util.log('Error describing Cache Cluster:' + err);
}
});
}
getElastiCacheEndpoints();
}
init();

Redis Bitset operations in Node.js / Express.js

I'm very new to Node.js and Redis. I read this article, and want to use a bitset to store all the user information for my Express.js app, as mentioned in this article: http://blog.getspool.com/2011/11/29/fast-easy-realtime-metrics-using-redis-bitmaps/
I'm having a bit of a trouble. In my function, I get the current year, month, and date, and then use client.setbit() to set appropriate key and value. But how can I count all the keys? I'm on Redis 2.4*, and the BITCOUNT command is in 2.6. Is there any other way? The article uses a Java bitset, so that's a different thing. I don't quite understand it.
How could I use, for example, a for loop, to count all the bits set to 1? Is there any operation to count the size of the bitset, so I could do something like this:
for (var i = initial_offset; i < bitset_length; i++){
if (i == 1){
total_users++;
}
}
Or am I going about it in a totally wrong way?

You need to count the number of bits of a given string stored in Redis.
There are basically two ways to do this:
you can try to do it on server-side with Redis 2.6 and the new BITCOUNT/BITOP operations.
you can retrieve the whole string (containing all the bits) and process the data on client side. In the original article, the author retrieves the Redis string and converts it to a Java bitset on which bit-level algorithms can be applied. The same strategy can be applied with any client, any language: you just have to find a good library to deal with arrays of bits, or implement one by yourself (it is not that hard). It would work with Redis 2.2 or higher.
A strategy that would not work very well is to iterate on client-side and check each individual bits by executing the GETBIT command. It would be really inefficient.
With node.js, here are a few resources you may want to use to implement the second option:
https://gist.github.com/1455345
https://github.com/bramstein/bit-array
How do I create bit array in Javascript?
Node.js is not a very good environment to implement CPU consuming operations, but in the worst case, should you have very large bitsets, you can still rely on an efficient C++ implementation to be called from Node.js. You have a good one in boost::dynamic_bitset.
Here is a Node.js example with a very simple (and probably inefficient) counting algorithm:
var redis = require('redis')
var rc = redis.createClient(6379, 'localhost', {return_buffers:true} );
var bitcnt = [ 0,1,1,2,1,2,2,3,1,2,2,3,2,3,3,4,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,4,5,5,6,5,6,6,7,5,6,6,7,6,7,7,8]
function count(b)
{
var cnt = 0
for (i=0; i<b.length; ++i ) {
cnt += bitcnt[ b[i] ]
}
return cnt
}
function fetch( callback )
{
rc.get( 'mybitset', function(err,reply) {
callback(reply)
});
}
function fill( callback )
{
rc.setbit( 'mybitset', 0, 1 )
rc.setbit( 'mybitset', 10, 1 )
rc.setbit( 'mybitset', 20, 1 )
rc.setbit( 'mybitset', 60, 1, function(err,reply) {
callback()
});
}
rc.flushall( function(err,rr) {
fill( function() {
fetch( function(b) {
console.log( "Count = ",count(b) );
});
})
})
Please note the {return_buffers:true} option is used to be sure Redis output is processed as binary data (ignoring possible character conversion).

Resources