Can't get redis cache to work with apollo server express - node.js

I'm trying to setup a caching system for my apollo express server, but it seems like the guides I've followed from the official docs: https://www.apollographql.com/docs/apollo-server/performance/caching/#memcachedredis-setup do not appear to work for me.
I can verify that in my resolver for a query that it keeps getting hit by console logging that it is being hit each time instead of only being hit once and then the cache being set
// my resolver definition
const getUserProfile = async (_, args, context, info) => {
info.cacheControl.setCacheHint({ maxAge: 300 });
console.log('MISS, set cache hint') // this console logs every time instead of just once
return context;
};
Here is my code:
server.js
...
const redisClient = require('./cacheClient');
let httpServer;
if (process.env.ENV === 'prod') {
httpServer = httpCreateServer(app);
} else {
httpServer = httpsCreateServer(
{
key: fs.readFileSync('./localhost-key.pem'),
cert: fs.readFileSync('./localhost.pem'),
},
app
);
}
const apolloServer = new ApolloServer({
typeDefs,
resolvers,
cache: new BaseRedisCache({
client: redisClient
}),
cacheControl: true,
});
apolloServer.context = ({ req }) => ({
...
});
await apolloServer.start();
apolloServer.applyMiddleware({ app });
httpServer.listen(PORT, () =>
console.log(`Server is now running on port ${PORT}`)
);
cacheClient.js
require('dotenv').config();
const { REDIS_SERVER, REDIS_PORT } = process.env;
const redis = require('redis');
const client = redis.createClient({
host: REDIS_SERVER,
port: REDIS_PORT,
});
client.on('connect', () => {
console.log('Redis client connected');
});
client.on('message', (channel, message) => {
console.log(`Received ${message} from ${channel}`);
});
module.exports = client;
I'm also making sure my web app is sending a Cache-Control: max-age=300 header on the graphql query as well. I also checked my Redis server and verified that no new keys are added after making the graphql query.
I have also tried setting the cacheControl directive onto the type like:
type UserProfile #cacheControl(maxAge: 300) {
...
}
but no luck with that either. I appreciate any help, thank you!

Whether you use koa or express, you will need to add these plugins: https://www.apollographql.com/docs/apollo-server/api/plugin/cache-control/
https://www.apollographql.com/docs/apollo-server/performance/caching#identifying-users-for-private-responses
...
const responseCachePlugin = require('apollo-server-plugin-response-cache').default;
const { ApolloServerPluginCacheControl } = require("apollo-server-core");
const server = new ApolloServer({
typeDefs,
resolvers,
plugins: [responseCachePlugin({
sessionId: ({ context }) => (context.ctx.header ? context.ctx.header.authorization : null),
}),
ApolloServerPluginCacheControl({
// Cache everything for 1000 seconds.
defaultMaxAge: 1000,
// Don't send the `cache-control` response header.
calculateHttpHeaders: false,
}),
],
});
Important Notes:
Change this code context.ctx.header ? context.ctx.header.authorization to get session Id for specific user.
Make sure that the version of apollo-server-express you use will fire events willResolveField for getting values from the cache and willSendResponse for setting the value in cache when needed (because I had a problem with v2.25.x in Koa).
Using const redis = require('redis'); will not set ttl (maxAge) correctly. So, please make sure to use ioredis instead to be able to set ttl.

Related

How to setup a WebSocket server inside SvelteKit using the ws package?

I'm trying to create a simple WebSocket server that will run in a SvelteKit application. I found this tutorial online which shows how to do it using Socket.io, however I would like to use the ws module instead.
This is the vite.config.ts file that I've come up with so far:
import type { UserConfig } from 'vite';
import { sveltekit } from '#sveltejs/kit/vite';
import { WebSocketServer } from "ws";
const webSocketServer = {
name: "webSocketServer",
configureServer: () => {
const webSocketServer = new WebSocketServer({
port: 8080
});
webSocketServer.on("connection", (socket, request) => {
socket.on("message", (data, isBinary) => {
console.log(`Recieved ${data}`);
});
socket.send("test from server");
});
}
}
const config: UserConfig = {
plugins: [sveltekit(), webSocketServer]
};
export default config;
I can connect to this WebSocket server from the frontend (in +page.svelte, for example) and send messages between them.
But every time I make a change to my server's code and save the file, I get an error saying that "there's already something listening on port 8080" and my Vite dev server terminates. If I then start back up my Vite dev server by running npm run dev then it will all work fine again, until I make any change to the WebSocket server code and save the file, and then the same thing will repeat.
I would rather not have to re-start my Vite dev server every time I make a change to my WebSocket server's code. If possible, I would rather Vite automatically shut down my old webSocket server to make room for the new one before booting it up each time so that they won't conflict.
The tutorial I linked above shows this code snippet:
import adapter from '#sveltejs/adapter-node'
import preprocess from 'svelte-preprocess'
import { Server } from 'socket.io'
const webSocketServer = {
name: 'webSocketServer',
configureServer(server) {
const io = new Server(server.httpServer)
io.on('connection', (socket) => {
socket.emit('eventFromServer', 'Hello, World 👋')
})
},
}
/** #type {import('#sveltejs/kit').Config} */
const config = {
preprocess: preprocess(),
kit: {
adapter: adapter(),
vite: {
plugins: [webSocketServer],
},
},
}
export default config
This snippet is using an old SvelteKit version where Vite configuration was done inside the svelte.config.js file, so the layout is a little different, but it seems they're simply spinning up their Socket.io server directly inside the configureServer method just like I am, but they're tapping into the existing Vite http server instead of creating a new one like I am. I tried doing this and I still get the same problem. Every time I try httpServer.listen(8080); on Vite's http server I'm told that the server was already listening on a port and the dev server terminates. I also tried manually creating an http server using require("http").createServer() and using that, but (unsurprisingly) that also did not work and acted identically to my initial attempt shown at the beginning of this question.
The tutorial seems to be booting up a Socket.io server the same way I'm trying to boot up mine, but they don't seem to be running into any conflicts like I am. I checked Socket.io's documentation to see if perhaps the Server constructor has a built-in failsafe to make sure it doesn't listen on a port if it's already listening on that port (and avoid creating the error), but the docs didn't give any information in that regard, so I'm still unsure as to what's going on there.
Is there any way to do what I'm trying to do, or am I going about this entirely the wrong way? I can't find hardly any information about this anywhere on the internet.
And also, what's going on in the Socket.io example that allows it to work where mine won't? Is Socket.io doing something special?
Here's my attempt at using Vite's built-in http server. This behaves the same as my other attempts.
vite.config.ts
import type { UserConfig } from 'vite';
import { sveltekit } from '#sveltejs/kit/vite';
import { WebSocketServer } from "ws";
const webSocketServer = {
name: "webSocketServer",
configureServer: (server: any) => {
const httpServer = server.httpServer;
const webSocketServer = new WebSocketServer({
noServer: true
});
webSocketServer.on("connection", (socket, request) => {
socket.on("message", (data, isBinary) => {
console.log(`Recieved ${data}`);
});
socket.send("hi c:");
});
httpServer.on("upgrade", (request: any, socket: any, head: any) => {
webSocketServer.handleUpgrade(request, socket, head, socket => {
webSocketServer.emit("connection", socket, request);
});
});
httpServer.listen(8080);
}
}
const config: UserConfig = {
plugins: [sveltekit(), webSocketServer]
};
export default config;
ws supports initialising with an existing server like socket.io
Don't store the application logic in the vite/svelte config, as it will be needed for the production build when vite is not available (I'm not familiar with sveltekit though so maybe it does some magic?). Create a standalone file for the ws setup:
import { WebSocketServer } from "ws";
export const configureServer = (server) => {
const webSocketServer = new WebSocketServer({
server: server.httpServer,
});
webSocketServer.on("connection", (socket, request) => {
socket.on("message", (data, isBinary) => {
console.log(`Recieved ${data}`);
});
socket.send("test from server");
});
}
export const webSocketServer = {
name: "webSocketServer",
configureServer,
}
Then you should be able to use your version of webSocketServer like the socket.io example as they both attach to the '#sveltejs/adapter-node' server.
import adapter from '#sveltejs/adapter-node'
import { webSocketServer } from './sockets.js';
/** #type {import('#sveltejs/kit').Config} */
const config = {
kit: {
adapter: adapter(),
vite: {
plugins: [webSocketServer],
},
},
}
export default config
configureServer can now be reused when you setup the custom server for production.

NodeJS to SQL Server Connection not working: socket hang up issue

Here is my complete code for sql connection, all code I have got from stackoverflow issues.
Everywhere, I found the same code is being suggested, hence I also tried with the same.
I have some other application which uses same connection with NextJs and it works fine, however, If I try only with NodeJS code, it gives some socket hang up error (code:'ESOCKET' name:'ConnectionError').
Please make a note that TCP is already configured on remote server and its working fine with other applications.
Any help is appreciated, thank you.
const express = require('express');
const fs = require('fs');
const path = require('path');
const cheerio = require("cheerio");
const sql = require('mssql');
require('dotenv').config(); //to use the env variables
// config for your database
var config = {
user: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
server: process.env.DATABASE_HOST,
database: process.env.SOMEDB,
port: 14345, // process.env.DATABASE_PORT,
options: {
encrypt: true, // for azure
trustServerCertificate: false // change to true for local dev / self-signed certs
}
};
// make sure that any items are correctly URL encoded in the connection string
let appPool = new sql.ConnectionPool(config);
//I got error on below connect
sql.connect(config).then(function(pool) {
//It never reaches here, it directly goes to the catch block
app.locals.db = pool;
const server = app.listen(3000, function () {
const host = server.address().address
const port = server.address().port
console.log('Example app listening at http://%s:%s', host, port)
})
}).catch(function(err) {
console.error('Error creating connection pool', err)
});
I have the same issue.
Try to use mssql version 6.0.1, it works on my code, but for sure we need to figure out the problem, since we can't think to mantain forever an old version of a package.
I kept trying to find the solution with different different configuration changes.
Finally, I have made a proper config, which worked and now its connecting properly as well as returning the data from the table.
require('dotenv').config(); //to access the process.env params
const sql = require("mssql"); //mssql object
var dbConfig = {
user: "ajay",
password: "abcd123",
server: "your_remote_sql_server_path",
port: 1433,
database: "your_database_name",
options: {
database: 'your_database_name',
trustServerCertificate: true
}
};
try {
//connection config will be used here to connect to the local/remote db
sql.connect(dbConfig)
.then(async function () {
// Function to retrieve the data from table
const result = await sql.query`select top 1 * from table_name`
console.dir(result)
}).catch(function (error) {
console.dir(error);
});
} catch (error) {
console.dir(error);
}
I am not sure what was the exact issue, but as per the previous config and this one, it seems like adding database name to the options has solved the issue.
Please make sure to save all the sensitive data to the .env file. (which you can access as PROCESS.env.parametername)
For me in driver mssql#9.1.1 making encrypt=false worked
const config = {
user: process.env.DATABASE_USER,
password: process.env.DATABASE_PASSWORD,
server: process.env.DATABASE_HOST,
database: process.env.SOMEDB,
port: 14345, // process.env.DATABASE_PORT,
options: {
encrypt: false
}
};

Difficulty setting Cookie when using an ngrok tunnel (Express server on Node.js, React app frontend)

As outlined in the title, I am having difficulty setting a http cookie to be used for auth purposes when tunnelling using ngrok.
The following code works fine (obviously with the relevant endpoints specified) when i am running a query from from localhost to a localhost endpoint in my dev environment but breaks down as soon as i start to query the ngrok tunnel endpoint.
Frontend api query (simplified as part of larger application)
function fetchRequest (path, options) {
const endpoint = 'http://xxx.ngrok.io'; // the ngrok tunnel endpoint
return fetch(endpoint + path, options)
.then(res => {
return res.json();
})
.catch((err) => {
console.log('Error:', err);
});
}
function postRequest (url, body, credentials='include') {
return fetchRequest(`${url}`, {
method: 'POST',
withCredentials: true,
credentials: credentials,
headers: {'Content-Type': 'application/json', Accept: 'application.json'},
body: JSON.stringify(body)
});
}
// data to be passed to backend for authentication
let data = {pin: pin, username : username};
postRequest('/',data)
Express server on Node.js with ngrok tunnel (app.js)
const express = require('express')
const session = require('express-session')
const cors = require('cors')
const router = require('./router');
const tunnel = require('./ngrok')
const app = express()
const port = process.env.PORT || 4001;
app.use(cors({
origin: 'http://localhost:3000'
credentials: true,
}))
app.use(express.json());
const expiryDate = new Date(Date.now() + 60 * 60 * 1000) // 1 hour
app.use(session({
secret: 'keyboard cat',
resave: false,
saveUninitialized: true,
cookie: {
httpOnly: true,
expires: expiryDate
// sameSite: 'none'
// secure: true
}
}))
app.use(router)
let useNGROK = true;
if (useNGROK) {
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
})
tunnel.createHTTPtunnel().then((url) => {
console.log(`New tunnel created with endpoint: ${url}`)
});
} else {
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
})
}
Ngrok configuration (ngrok.js)
const ngrok = require('ngrok');
const find = require('find-process');
const port = process.env.PORT || '3000';
const tunnel = {
createHTTPtunnel: async function () {
const list = await find('name', 'ngrok');
if (list.length > 0) {
let api = ngrok.getApi();
if (api == null) {
this.kill_existing_tunnel();
} else {
let open_tunnels = await ngrok.getApi().listTunnels();
return open_tunnels.tunnels[0].public_url;
}
}
let ngrok_config = {
proto: 'http',
bind_tls: false,
name: process.env.NGROK_NAME,
hostname: process.env.NGROK_CUSTOM_DOMAIN,
// host_header: 'rewrite',
authtoken: '',
region: 'eu',
};
return ngrok.connect({ ...ngrok_config, addr: port });
},
kill_existing_tunnel: async () => {
const list = await find('name', 'ngrok');
list.forEach((p) => {
try {
process.kill(p.pid);
console.log(`Killed process: ${p.name} before creating ngrok tunnel`);
} catch (e) {
console.log(e);
}
});
}
}
module.exports = tunnel;
** router & controller (router.js & controller.js respectively) **
*router.js*
const router = require('express').Router();
const example = require('./controller')
router.post('/', example.authenticate);
module.exports = router;
*controller.js*
async function authenticate (req, res) {
try {
res.send(JSON.stringify('trying to send cookie'))
} catch (e) {
console.log('Error', e)
res.sendStatus(500)
}
}
module.exports = {
authenticate
};
The following information is provided when inspecting the Set-Cookie response header in the network requests:
This Set-Cookie header didn’t specify a “SameSite” attribute and was defaulted to “SameSite=Lax” and was blocked because it came from a cross-site response which was not the response to a top-level navigation. The Set-Cookie had to have been set with “SameSite=None” to enable cross site usage.
Attempted fix 1//
If I add the following options to the cookie {sameSite: ‘none’, secure:true}, amend the ngrok config to set {bind_tls: true} and run https on my front end (using a custom SSL certificate as per the create react app documentation), and query the https tunnel, then no cookie is received in the response from the server at all (request is sent and response 200 is received but with no cookie).
Attempted fix 2//
I also tried to change the host_header option to rewrite in the ngrok config (to mirror a response from localhost rather than from ngrok) and this did not work.
Any help would be much appreciated as I have little experience and I am stuck!

can't connect to Socket.IO server on Heroku

I want to host my frontend on Vercel (I'm using Nextjs) and since it doesn't support socket connections in it's API routes I decided to move this part of my app to Heroku. My problem is that when I use the server from my frontend in dev environment it works just fine, but when I deploy it to Heroku I get this error:
Access to XMLHttpRequest at 'https://my-socket-server.herokuapp.com/socket.io/?EIO=3&transport=polling&t=NMPkkyL' from origin 'http://localhost:3000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
my server code looks like this:
import express from 'express';
import io, { Namespace } from 'socket.io';
import { PORT, menus, socketServerConfig, adminRoom } from './util/config';
const server = express().listen(PORT, () =>
console.log(`server running on port ${PORT}`)
);
const socketServer = io(server, socketServerConfig);
const attachSocketHandlers = (server: Namespace) => {
server.on('connection', socket => {
const handlers = {
'admin-log-in': () => {
socket.join(adminRoom);
},
'need-waiter': (table: Table) => {
server.to(adminRoom).emit('need-waiter', table);
},
'need-receipt': (table: Table) => {
server.to(adminRoom).emit('need-receipt', table);
},
order: (order: Order) => {
server.to(adminRoom).emit('order', order);
},
disconnect: () => {
socket.leaveAll();
},
};
Object.entries(handlers).map(([event, handler]) => {
socket.on(event, handler);
});
});
};
menus.forEach(menu => {
const namespacedServer = socketServer.of(`/${menu}`);
attachSocketHandlers(namespacedServer);
});
What I understood from the socket.io docs is that if you list no origins in the config it allows all origins to access the socket server.This is my socketServerConfig:
import { ServerOptions } from 'socket.io';
const defaultPort = 4000;
export const PORT = process.env?.PORT ?? defaultPort;
export const menus = ['more'];
export const adminRoom = 'admin';
export const socketServerConfig: ServerOptions = {
serveClient: false, // i don't serve any static files
};
this is how I connect from my frontend:
const url = 'https://my-socket-server.herokuapp.com/';
const io = connect(`${url}${menu}`);
I tried various solutions from SOF but I just can't get it to work, any help would be appreciated. Thanks in advance.
Try doing npm install cors and adding this to your server:
const cors = require('cors');
const whitelist = [
'http://localhost:3000',
YOUR FRONTEND URL
];
const corsOptions = {
origin: function (origin, callback) {
console.log('** Origin of request ' + origin);
if (whitelist.indexOf(origin) !== -1 || !origin) {
console.log('Origin acceptable');
callback(null, true);
} else {
console.log('Origin rejected');
callback(new Error('Not allowed by CORS'));
}
},
};
express().use(cors(corsOptions));
Hope this helped!

MongoDB Change Stream watcher has been fired on second execution of aws-lambda function or has not been fired at all

I am building Angular PWA app with nodejs server. After few investigations I decided to put my front-end on S3 and back-end on aws-lambda using serverless-framework and connect it with mongo atlas.
In my back-end I use mongoose.watch() to listen for appropriate changes in my DB. On change happenes I use FCM to send data to device according to token.
Once app is launched I send 3-4 requests (different pathes) to get data from my DB.
So here is my issue:
If I use sls offline start --skipCacheInvalidation to work locally everything works almost fine - I received 2 or more dublications, so it seems like each my starting requests begin aws-lambda function.
Once I sls deploy I get all my starting data, but once I update DB (by another request) in aws-lambda logs streams I don't see that my code reach .on('change') listener. And aws-lambda request execution ends. Sometimes my code reaches this listener, and print change stream data of my previous request.
Here is part of my serverless.yml
provider:
name: aws
runtime: nodejs8.10
stage: dev
region: us-east-1
memorySize: 128
timeout: 30
versionFunctions: false
functions:
app:
handler: server.handler
events:
- http:
method: any
path: /{proxy+}
cors:
origin: '*'
headers:
- Content-Type
- X-Amz-Date
- Authorization
- X-Api-Key
- X-Amz-Security-Token
- X-Amz-User-Agent
- x-customauthheader
allowCredentials: false
server.js
require('dotenv').config({ path: './variables.env' });
const sls = require('serverless-http');
const app = require('./lib/app');
module.exports.handler = sls(app, { callbackWaitsForEmptyEventLoop: false });
app.js
const app = require('express')();
const bodyParser = require('body-parser');
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
// // Allow CORS support and remote requests to the service
app.use((req, res, next) => {
res.setHeader('Access-Control-Allow-Origin', '*');
next();
});
const { checkHeaders } = require('../bin/routes-handler');
app.use(checkHeaders);
require('./db');
const routes = require('./routes');
app.use('/api', routes);
module.exports = app;
db.js
const mongoose = require('mongoose');
mongoose.connect(
process.env.DB,
{ useNewUrlParser: true, useCreateIndex: true, useFindAndModify: false },
);
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'Connection Error:'));
db.once('open', () => {
console.log('mongoose Connection established');
require('../services/quote-watcher')(db);
});
and my quote-watcher.js
const { sendDataByTokens } = require('../bin/notify-devices');
module.exports = db => {
'use strict';
const taskCollection = db.collection('quotes');
const changeStream = taskCollection.watch([], { fullDocument: 'updateLookup' });
// const changeStream = taskCollection.watch();
changeStream.on('error', err => {
console.log(err);
});
changeStream.on('change', change => {
console.log(change);
if (change.operationType === 'insert') {
return sendDataByTokens(
change.fullDocument.participants,
change.operationType,
change.fullDocument,
change.ns.coll,
null,
{ stage: change.fullDocument.currStage },
);
}
if (change.operationType === 'update') {
const { updatedFields } = change.updateDescription;
return sendDataByTokens(
change.fullDocument.participants,
change.operationType,
change.fullDocument,
change.ns.coll,
change.documentKey._id,
{ stage: change.fullDocument.currStage, updatedFields },
);
}
});
};

Resources