I'm trying to use Redis in a nextJs API which has a custom express server. I created a util file for Redis in order to connect to the Redis labs, the same way I did with mongodb. The problem is that when I try to use the client.Hset the console keeps sending me errors that econ refused, but I also made sure that when Redis is connected it console logs a string. So the console, logs both the error of econ refused, and the log connected to redis. This is how my util file looks like:
/* ------ connecting to redis ------ */
const redis = require('redis');
const { promisify } = require('util');
const client = redis.createClient({
host: process.env.REDIS_HOSTNAME,
port: process.env.REDIS_PORT,
password: process.env.REDIS_PASSWORD
});
client.on('connect', ()=>{
console.log('connected to redis')
})
/* promisifying redis in order to use async functionality */
const redisHget = promisify(client.hget).bind(client)
const redisHset = promisify(client.hset).bind(client)
module.export = {redisHget, redisHset}
It seems like every time I call the API, it reconnects to Redis. may I know if there is a way I can use redis the same way I use MongoDB by just creating and importing collections so that it doesn't keep reconnecting to the redis labs server?
Lee Robinson (DevRel # Vercel), wrote a nice article about using Redis in Next.js. https://leerob.io/blog/serverless-redis-nextjs.
Complete Example here: https://github.com/vercel/next.js/blob/canary/examples/with-redis/README.md
Snippets:
# env.local
REDIS_URL=redis://:password#endpoint:port
We'll use ioredis, a Node.js Redis client with async/await support, to connect to Redis.
// lib/redis.js
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL);
export default redis;
Using it, either in pages/ or pages/api/,
import redis from '#/lib/redis';
const value = JSON.parse(await redis.hget('feedback', id));
Related
I have a question about redis connections.
I'm developing an app in react native which will use websockets for chat messages. My backend consists of a node.js-app with redis as pubsub mechanism for socket.io.
I'm planning on deploying on heruko. I'm currently on the free hobby plan, which has a limit of 20 connections to redis.
My question now is: how can I optimize my code so that a minimum of connections are used. I'm ofc planning to upgrade my heroku plan once I launch, but then still I want to optimize.
My node.js-code looks like this (simplified):
const Redis = require('ioredis');
const pubClient = new Redis(/* redis url */);
const subClient = new Redis(/* redis url */);
const socketClient = new Redis(/* redis url */);
const io = require('socket.io')(server);
io.on('connection', async (socket) => {
// store socket.id in redis so I can send messages to individual users
// based on the user ID
const userId = socket.handshake.query.userId;
await socketClient.hset('socketIds', userId, socket.id);
socket.on('message', async (data) => {
/**
* data {
* userId,
* message
* }
*/
const data2 = JSON.parse(data);
// get the socket.id based on the user ID
const socketId = await socketClient.hget('socketIds', data2.userId);
// send the message to the correct socket.id
io.to(socketId).emit('message', data.message);
};
});
So when I deploy this code to heroku, when started, it will create 3 connections to the same redis server. But what if 2-3-4-... people connect to this node.js-server? If 2 people connect, will there be 6 redis-connections, or only 3? Like: will the node.js-server initiate every time a users accesses the server 3 new redis connections, or will it always be 3 connections?
I'm trying to track all connections with CLIENT LIST in redis-cli, but I does not give me the correct thing I guess. I was just testing my code with only one user connection to the socket server and it gave me 1 client in redis (instead of 3 connections).
Thanks in advance.
It doesn't matter how many people are using the app, each client instance will have only 1 socket at any time, which means you'll see at most 3 clients per node process.
You see only 1 connection because by default ioredis initiates the connection when the first command is executed, and not when the client is created. You can call client.connect() in order to initiate the socket without executing a command.
I want to have more than one Redis client object as multiple DB instance like DB0 (O INDEX REDIS DB).
Currently, I'm using
let RedisClient = require("./redis");
RedisClient.select(1, function (err, res) {
// any operation here
});
so RedisClient is having DB1 instance. what is the best way to deal with multiple db if we want to use another DB2 ?
Use redis.createClient() twice
const redis = require("redis");
const client_1 = redis.createClient(REDIS_1_HOST, REDIS_PORT_1);
const client_2 = redis.createClient(REDIS_2_HOST, REDIS_PORT_2);
see: https://www.npmjs.com/package/redis#rediscreateclient
This is a little hard articulate so I hope my title isn't too terrible.
I have a frontend/backend React/Node.js(REST API) Web app that I want to add Redis support to for storing retrieving app global settings and per-user specific settings (like language preference, last login, etc... simple stuff) So I was considering adding a /settings branch to my backend REST API to push/pull this information from a redis instance.
This is where my Node.js inexperience comes through. I'm looking at using the ioredis client and it seems too easy. If I have a couple of helpers (more than one .js which will call upon redis) will constructing the client as a const in each be safe to do? Or is there another recommended approach to reusing a single instance of it be the way to go?
Here's a sample of what I'm thinking of doing. Imagine if I had 3 helper modules that require access to the redis client. Should I declare them as const in each? Or centralize them in a single helper module, and get the client from it? Is there a dis-advantage to doing either?
const config = require('config.json');
const redis_url = config.redis_url;
//redis setup
const Redis = require('ioredis');
const redis = new Redis(redis_url);
module.exports = {
test
}
async function test(id) {
redis.get(id, function (err, result) {
if (err) {
console.error(err);
throw(err);
} else {
return result;
}
});
Thank you.
If no redis conflicts...
If the different "helper" modules you are referring to have no conflicts when interacting with redis, such as overwriting / using the same redis keys, then I can't see any reason not to use the same redis instance (as outlined by garlicman) and export this to the different modules in which it is used.
Otherwise use separate redis databases...
If you do require separate redis database connections, redis ships with 16 databases so you can specify which to connect to when creating a new instance - see below:
const redis = new Redis({ // SET UP CONFIG FOR CONNECTION TO REDIS
port: 6379, // Redis port
host: 127.0.0.1, // Redis host
family: 4, // 4 (IPv4) or 6 (IPv6)
db: 10, // Redis database to connect to
});
Normally what I would do (in Java say) is implement any explicit class with singleton access the hold the connection and any connection error/reconnect handling.
All modules in Node.js are already singletons I believe, but what I will probably go with will be a client class to hold it and my own access related methods. Something like:
const config = require('config.json');
const Redis = require("ioredis");
class Redis {
constructor(){
client = new Redis(config.redis_url);
}
get(key) {
return this.client.get(key);
}
set(key, value, ttl) {
let rp;
if (ttl === 0) {
rp = this.client.set(key, value);
}
else {
rp = this.client.set(key, value)
.then(function(res) {
this.client.expire(key, ttl);
});
}
return rp;
}
}
module.exports = new Redis;
I'll probably include a data_init() method to check and preload an initial key/value structure on first connect.
When i try to connect my Nodsjs application to RedisCloud on Heroku I am getting the following error
Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (ECONNREFUSED)
I have even tried to directly set the redis URL and port in the code to test it out as well. But still, it tried to connect to the localhost on Heroku instead of the RedisCloud URL.
const {Queue} = require('bullmq');
const Redis = require('ioredis');
const conn = new Redis(
'redis://rediscloud:mueSEJFadzE9eVcjFei44444RIkNO#redis-15725.c9.us-east-1-4.ec2.cloud.redislabs.com:15725'
// Redis Server Connection Configuration
console.log('\n==================================================\n');
console.log(conn.options, process.env.REDISCLOUD_URL);
const defaultQueue = () => {
// Initialize queue instance, by passing the queue-name & redis connection
const queue = new Queue('default', {conn});
return queue;
};
module.exports = defaultQueue;
Complete Dump of the Logs https://pastebin.com/N9awJYL9
set REDISCLOUD_URL on .env file as follows
REDISCLOUD_URL =redis://rediscloud:password#hostname:port
import * as Redis from 'ioredis';
export const redis = new Redis(process.env.REDISCLOUD_URL);
I just had a hard time trying to find out how to connect the solution below worked for me.
Edit----
Although I had been passed the parameters to connect to the Redis cloud, it connected to the local Redis installed in my machine. Sorry for that!
I will leave my answer here, just in case anyone need to connect to local Redis.
let express = require('express');
var redis = require('ioredis');
pwd = 'your_pwd'
url = 'rediss://host'
port = '1234'
redisConfig = `${url}${pwd}${port}`
client = redis.createClient({ url: redisConfig })
client.on('connect', function() {
console.log('-->> CONNECTED');
});
client.on("error", function(error) {
console.error('ERRO DO REDIS', error);
});
Just wanted to post my case in case someone has the same problem like me.
In my situation I was trying to use Redis with Bull, so i need it the url/port,host data to make this happened.
Here is the info:
https://devcenter.heroku.com/articles/node-redis-workers
but basically you can start your worker like this:
let REDIS_URL = process.env.REDISCLOUD_URL || 'redis://127.0.0.1:6379';
//Once you got Redis info ready, create your task queue
const queue = new Queue('new-queue', REDIS_URL);
In the case you are using local, meaning 'redis://127.0.0.1:6379' remember to run redis-server:
https://redis.io/docs/getting-started/
There are a lot of examples of graceful stop for expressjs, how can I achieve the same for koajs?
I would like to disconnect database connections as well
I have a mongoose database connection, and 2 oracle db connection (https://github.com/oracle/node-oracledb)
I created an npm package http-graceful-shutdown (https://github.com/sebhildebrandt/http-graceful-shutdown) some time ago. This works perfectly with http, express and koa. As you want to add also your own cleanup stuff, I modified the package, so that you now can add your own cleanup function, that will be called on shutdown. So basically this package handles all http shutdown things plus calling your cleanup function (if provided in the options):
const koa = require('koa');
const gracefulShutdown = require('http-graceful-shutdown');
const app = new koa();
...
server = app.listen(...); // app can be an express OR koa app
...
// your personal cleanup function - this one takes one second to complete
function cleanup() {
return new Promise((resolve) => {
console.log('... in cleanup')
setTimeout(function() {
console.log('... cleanup finished');
resolve();
}, 1000)
});
}
// this enables the graceful shutdown with advanced options
gracefulShutdown(server,
{
signals: 'SIGINT SIGTERM',
timeout: 30000,
development: false,
onShutdown: cleanup,
finally: function() {
console.log('Server gracefulls shutted down.....')
}
}
);
I have answered a variation of "how to terminate a HTTP server" many times on different node.js support channels. Unfortunately, I couldn't recommend any of the existing libraries because they are lacking in one or another way. I have since put together a package that (I believe) is handling all the cases expected of graceful HTTP server termination.
https://github.com/gajus/http-terminator
The main benefit of http-terminator is that:
it does not monkey-patch Node.js API
it immediately destroys all sockets without an attached HTTP request
it allows graceful timeout to sockets with ongoing HTTP requests
it properly handles HTTPS connections
it informs connections using keep-alive that server is shutting down by setting a connection: close header
it does not terminate the Node.js process
Usage with Koa:
import Koa from 'koa';
import {
createHttpTerminator,
} from 'http-terminator';
const app = new Koa();
const server = app.listen();
const httpTerminator = createHttpTerminator({
server,
});
await httpTerminator.terminate();
To make sure the Oracle DB connections are closed nicely, you can use a connection pool and call pool.close() with a drainTime of 0 or greater. This will let the app relatively cleanly interrupt any operation that is currently using a connection. It allows freeing the DB end of the connections without the DB waiting for whatever timeout period to expire before it cleans itself up. Even with two connections this is a solution I'd look at, since it doesn't matter that the pool is small. You may need to set the Oracle Net out-of-band break detection as well, see Connections and High Availability.
Modern versions of node have support for AbortController, so no need for external libraries. A Simple example:
const app = new Koa();
const server = http.createServer(app.callback());
const controller = new AbortController();
server.listen({
host: 'localhost',
port: 80,
signal: controller.signal
});
// middleware... etc.
app.use(async (ctx) => {
ctx.body = 'Hello World';
});
// Later, when you want to close the server.
controller.abort();