Node-RED listen PostgreSQL in real-time - node.js

I need to listen PostgreSQL on changes in real-time with Node-RED.
How can I do this?
I created trigger on new record in the table and notify this to 'changes' channel.
CREATE FUNCTION notify_trigger() RETURNS trigger AS $$
DECLARE
BEGIN
PERFORM pg_notify('changes', TG_TABLE_NAME || ',id,' || NEW.id );
RETURN new;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER watched_table_trigger AFTER INSERT ON users
FOR EACH ROW EXECUTE PROCEDURE notify_trigger();
But I don't know how to listen it from Node-RED. Could you help me please?
Maybe I can do it differently?

Have a look at this previous SO question:
MQTT Client subscribe to PostgreSQL DB Changes
It looks like Postgress supports Python based triggers which could be used to send a MQTT message which Node-RED could easily subscriber to.

I found a good solution for yourself with WebSocket. Look at example below:
var pg = global.get('pg'),
WebSocket = global.get('ws'),
config = {
user: 'user',
password: 'user',
host: 'somehost',
port: 1234,
database: 'somedb'
},
client = new pg.Client(config);
client.connect(function(err) {
if (err) node.error(err);
client.on('notification', function(msg) {
node.send(msg);
});
var query = client.query("LISTEN changes");
});
delete msg._session;
return msg;
Post your solution, I really want to know more ways to solve this.

Related

Creating sub connections with azure-mobile-apps and NodeJS

I'm trying to create an API using nodeJS, express and azure-mobile-apps to do some data synchronisation between an Ionic3 mobile app (which use an SQLite local database) and a Microsoft SQL Database.
My API has to create a synchronisation connection for each mobile application. Each application will be linked to a distant database. For example, if user_01 wants to synchronise his data, he's going to be linked to his client_01 database. So each time it'll have to, the API will create a new process running on a different port.
here is an example : https://zupimages.net/up/19/36/szhu.png
The problem is that i'm not able to create more than one connection with azure-mobile-apps. The first one always works, but the second, third etc are still using the first connection that i have instantiated. I've looked into the app stack and everything seems fine.
Is that an issue with azure-mobile-app, or did I misunderstand something with express ?
Thanks for your responses !
var azureMobileApps = require('azure-mobile-apps');
var express = require('express');
module.exports = {
async createConnection(client) {
try {
let app = express();
mobileApp = azureMobileApps({
homePage: true,
swagger: true,
data: {
server: '****',
user: client.User,
password: client.Password,
port: '1443',
database: client.Database,
provider: 'mssql',
dynamicSchema: false,
options: {
encrypt: false
}
}
});
await mobileApp.tables.import('./tables');
await mobileApp.tables.initialize();
app.listen(global.portCounter);
app.use(mobileApp);
console.log(app._router.stack);
console.log('Listening on port ',global.portCounter);
global.portCounter++;
} catch (error) {
console.log(error);
}
}
}
It's working now. The thing is, it's impossible to do multiple connection with the azure-mobile-apps SDK for nodeJS.
I had to use worker-thread which seems to isolate the memory in a sub-proccess.
Hope it can help somebody one day

how to get notifiction if any changes/transaction happen in redis using nodejs

I'am new to redis, i need help to get notify to my running server if any chages/transaction happen in redis(either from bakend also). The server which is in running state is created using node.js
Use the keyspace notification system. This uses the pub sub layer of redis
Steps :
Set this parameter -notify-keyspace-events in redis.conf file
By default its empty
This will enable capture of almost all DML events
I'm not sure about how to get notification about transaction, but here is a working code (using ioredis) to get notified about any change in redis:
const redisClient = const redisClient = new Redis({
host: 'localhost',
port: 20000,
lazyConnect: true, // because i will try to connect manually in the next line
connectTimeout: 1000,
})
await redisClient.connect()
// "AKE" === listen to all events of all redis-operations
await redisClient.config('SET', 'notify-keyspace-events', 'AKE')
await redisClient.psubscribe([`__key*__:*`])
redisClient.on(`pmessage`, (_pattern, redisKey, redisOpeation) => {
const redisKey = redisKey.replace('__keyspace#0__:', '')
if(redisKey === 'cool-redis-key' && redisOpeation === 'set'){
console.log(`there was a "set" operation on "cool-redis-key" key`)
}
})
Addtional info:
example: https://stackoverflow.com/a/58687129/806963
activate notifications: https://redis.io/topics/notifications

How does one correctly set up a server based deepstream RPC provider?

I am building a SOA with deepstream and I want to use a deepstream client server to perform API-KEY based look ups that the user should not know. How do I actually set up an RPC client provider? I have looked in the deepstream docs and on google, but there is not a full code example on how to do this. I have created a file like below and run it with node. The output I get is below it:
var deepstream = require('deepstream.io-client-js')
const client = deepstream('localhost:6020').login()
console.log('Starting up')
client.on('error', (error,event,topic) => {
console.log(error, event, topic);
})
client.on('connectionStateChanged', connectionState => {
console.log(connectionState);
})
client.login({username: 'USER', password: 'PASSWORD'}, (success, data) => {
if (success) {
client.rpc.provide('the-rpc', function( data, response ){
response.send(data);
});
} else {
console.log(data);
}
})
--
Starting up
AWAITING_CONNECTION
As you can see it runs the code, but does not actually connect to the deepstream server. I already have the deepstream server running, and a browser client that connects to it, so the config is correct. Please help!
I think your issue is based on the fact your trying to connect node via the webport. Try using port 6021 instead for tcp ( used by the node client ).
const client = deepstream('localhost:6021').login()
You should also only call .login() once, so the line would be:
const client = deepstream('localhost:6021')
We are working on a 2.0 release coming out very soon which will remove tcp entirely and only require a single port to make life easier in terms of deployment and performance.

Can't receive redis data from socket io

I'm building a realtime visualization using redis as pubsub messenger between python and node. There's a python script always running which sets a redis hash with hmset. That side of the app is working fine, if I enter the following example command: "HGETALL 'sellers-80183917'" in a redis client I end up getting the proper data.
The problem is in the js side. I'm using socketio and redis nodejs libraries to listen to the redis instance and publish the results online through a d3js viz.
I run the following code with node:
var express = require('express');
var app = express();
var redis = require('redis');
app.use(express.static(__dirname + '/public'));
var http = require('http').Server(app);
var io = require('socket.io')(http);
var sredis = require('socket.io-redis');
io.adapter(sredis({ host: 'localhost', port: 6379 }));
redisSubscriber = redis.createClient(6379, 'localhost', {});
redisSubscriber.on('message', function(channel, message) {
io.emit(channel, message);
});
app.get('/sellers/:seller_id', function(req, res){
var seller_id = req.params.seller_id;
redisSubscriber.subscribe('sellers-'.concat(seller_id));
res.render( 'seller.ejs', { seller:seller_id } );
});
http.listen(3000, '127.0.0.1', function(){
console.log('listening on *:3000');
});
And this is the relevant part of the seller.ejs file that's receiving the user requests and outputting the viz:
var socket = io('http://localhost:3000');
var stats;
var seller_key = 'sellers-'.concat(<%= seller %>);
socket.on(seller_key, function(msg){
stats = [];
console.log('Im in');
var seller = $.parseJSON(msg);
var items = seller['items'];
for(item in items) {
var item_data = items[item];
stats.push({'title': item_data['title'], 'today_visits': item_data['today_visits'], 'sold_today': item_data['sold_today'], 'conversion_rate': item_data['conversion_rate']});
}
setupData(stats);
});
The problem is that the socket_on() method never receives anything and I don't see where the problem is as everything seems to be working fine besides this.
I think that you might be confused as to what Pub/Sub in Redis actually is. It's not a way to listen to changes on hashes; you can have a Pub/Sub channel called sellers-1, and you can have a hash with the key sellers-1, but those are unrelated to each other.
As documented here:
Pub/Sub has no relation to the key space.
There is a thing called keyspace notifications that can be used to listen to changes in the key space (through Pub/Sub channels); however, this feature isn't enabled by default because it'll take up more resources.
Perhaps an easier method would be to publish a message after the HMSET, so any subscribers would know that the hash got changed (they would then retrieve the hash contents themselves, or the published message would contain the relevant data).
This brings us to the next possible issue: you only have one subscriber connection, redisSubscriber.
From what I understand from the Node.js Redis driver, calling .subscribe() on such a connection would remove any previous subscriptions in favor of the new one. So if you were previously subscribed to the sellers-1 channel and subscribe to sellers-2, you wouldn't be receiving messages from the sellers-1 channel anymore.
You can listen on multiple channels by either passing an array of channels, or by passing them as a arguments:
redisSubscriber.subscribe([ 'sellers-1', 'sellers-2', ... ])
// Or:
redisSubscriber.subscribe('sellers-1', 'sellers-2', ... )
You would obviously have to track each "active" seller subscription. Either that, or create a new connection for each subscription, which also isn't ideal.
It's probably a better idea to have a single Pub/Sub channel on which all changes would get published, instead of a separate channel for each seller.
Finally: if your seller id's aren't hard to guess (for instance, if it's based on an incremental integer value), it would be trivial for someone to write a client that would make it possible to listen in on any seller channel they'd like. It might not be a problem, but it is something to be aware of.

sails.io.js - nodejs - Resourceful PubSub not receiving model events

I am trying to subscribe a nodejs application to model events in sails. Here is my code:
var socketIOClient = require('socket.io-client'),
sailsIOClient = require('sails.io.js');
var io = sailsIOClient(socketIOClient);
io.sails.url = 'http://localhost:1337';
io.socket.on("agent", function(event) {
console.log(event);
})
io.socket.get("/agent", function(resData, jwres) {})
Here is a link to all of the output on the sails server when the client(nodejs) connects:
https://gist.github.com/CiscoKidxx/e5af93ebcc24702ba4f8
My understanding is that when I create a new agent it should trigger a console.log(event) which lists the changes. This is not happening. I do get a "now connected to sails" upon script start up. Any thoughts?
Here is my call to create a new agent in my UserController:
Agent.create({
syncToken: token,
owner: user.id
}).exec(function (err, newAgent) {
Agent.publishUpdate(newAgent.id, {syncToken: newAgent.syncToken});
The server side code snippet above doesn't show a call to Agent.publishCreate in the callback, but rather a publishUpdate.
From what I understand, the publishCreate is only automatically triggered when using the blueprints - but not for programatic calls like your Agent.create above.
So changing it from publishUpdate to publishCreate might fix it in your context.

Resources