Node.js/Vuetify- Is there a way to get data from server based on time? - node.js

I have a node.js server set up for a vuetify project. In my server, I am parsing a csv file that has information in it about scheduling and time. In my vuetify project, is there a way to get data from the csv based on the time that the client is being used?

OK, let's go with an example.
From what I understand, you have the following information in your CSV file:
Time,Activity
07:00,Breakfast
08:00,Go to work
12:00,Lunch break
Since you didn't specify, I will use an example parser, which will push all rows, as objects, into an array:
[
{ Time: '07:00', Activity: 'Breakfast' },
{ Time: '08:00', Activity: 'Go to work' },
{ Time: '12:00', Activity: 'Lunch break' }
]
You need to send that information to your clients, so assuming you are using Express, you could go with something in the lines of:
const csv = require('csv-parser');
const fs = require('fs');
const express = require('express');
const app = express();
const timeSchedule = [];
function parseCsv() {
return new Promise((resolve, reject) => {
fs.createReadStream('data.csv')
.pipe(csv())
.on('data', (data) => timeSchedule.push(data))
.on('error', (err) => reject(err))
.on('end', () => {
csvParsed = true;
resolve();
});
}
}
app.get('/scheduled-for/:hour', function (req, res) {
// You need to come up with the logic for your case
// As an example, I will assume that if I make a request at any time
// between 7 and 8, I will get "Breakfast"
// between 8 and 9 - "Go to work"
// between 12 and 13 - "Lunch break"
parseCsv().then(() => {
res.json(timeSchedule.find(row => row.Time.startsWith(req.params.hour)))
})
})
Please note, all of the above is happening on the Nodejs server.
From the client, you will have to call the scheduled-for GET handle with the hour param. Another option is to allow the back-end to determine the hour of the request, by using the Date object, but the above is more flexible for the client. It will also avoid issues with different timezones, given that your client requests are coming from different timezones than the one your server is on.
Assuming you are using axios in your Vue application, the simplest way to get the schedule is to call your API in your component:
new Vue({
el: '#app',
data () {
return {
activity: null
}
},
mounted () {
axios
.get('https://urlToYourApi/v1/scheduled-for/' + new Date().getHours())
.then(schedule => {
if (schedule) {
this.activity = schedule.Activity;
}
else {
console.log("No activity found for this hour!");
}
}
}
})
This code is NOT for production! You need to handle many cases, as with new Date().getHours() returning single-digit hours, parsing of CSV, not to mention the domain logic itself, which depends on your specific case. This is just a simple example. I hope it helps to guide you in the right direction!

Related

Keeping track of socket.io sessions for logging

I am working on making a generic logging module for my application and am trying to add session information to each log (requestId/socketId, userId, etc.) But I am running into some issues with logging websockets.
Basically my application has 2 parts: a restAPI (express) and websockets (socket.io)
Both the restAPI and websockets use some of the same functions (database edits etc.), now these functions should log errors or other useful data.
But passing the session information to the logger module will create a lot of overhead and makes the code quite unreadable, so I am looking for a way to save the session information so that the logger can get the information from there.
For the restAPI this was fairly simple using asyncLocalStorage and I was hoping to utilize the same principle for the websockets but I guess its not that simple.
My (partially) working code setup is as follows:
Global context creator (logAsyncContext.ts):
import { AsyncLocalStorage } from "async_hooks";
export const context = new AsyncLocalStorage();
export const createContext = (data: any, callBack: () => any) => {
const store = data;
return context.run(store, () => callBack());
};
This is then used by the middleware of the restAPI and websockets
RestAPI middleware (apiLogContext.ts):
// Import the required modules
import { v4 } from "uuid";
import { Request, Response, NextFunction } from "express";
// Import custom utilities
import { createContext } from "../../utils/logAsyncContext";
import { logger } from "../../utils/logger";
// Generate a unique ID for incoming requests and store in context so logger can access it
export const apiLogContext = (
req: Request,
_res: Response,
next: NextFunction
) => {
const logData = {
api: {
requestId: v4(),
originalUrl: req.originalUrl,
},
};
return createContext(logData, () => debugLog(next));
};
const debugLog = (next: NextFunction) => {
logger. Debug("API log context created");
return next();
};
websocket middleware (wsLogContext.ts):
// Import the required modules
import { v4 } from "uuid";
import { Socket } from "socket.io";
// Import custom utilities
import { createContext } from "../../utils/logAsyncContext";
import { logger } from "../../utils/logger";
// Generate a unique ID for incoming requests and store in context so logger can access it
export const wsLogContext = (socket: Socket, next: () => void) => {
const logData = {
ws: {
socketId: v4(),
nameSpace: socket.nsp.name,
},
};
return createContext(logData, () => debugLog(next));
};
const debugLog = (next: () => void) => {
logger.debug(`WS log context created`);
return next();
};
Now the logger can get the context from logAsyncContext.ts:
import { context } from "./logAsyncContext";
const getStore = () => {
// Get the store from the AsyncLocalStorage
const store = context.getStore();
// If the store is not defined, log an error
if (!store) {
console.log("Store is not defined");
return undefined;
}
return store;
};
export function debug(message: string) {
// Get the context
const store = getStore();
if (!store) {
return;
}
if (isAPILog(store)) {
console.debug(
`DEBUG LOG: ${store.api.requestId} | ${store.api.originalUrl} - ${message}`
);
} else {
console.debug(
`DEBUG LOG: ${store.ws.socketId} | ${store.ws.nameSpace} - ${message}`
);
}
};
This works perfectly for the restAPI but for the websockets its a different story, it does log the initial debug message ("WS log context created") but everything logged after cannot access the store ("Store is not defined")
Now I am sure this is very logical but I don't fully understand the structure of data for websocket connections, so I am asking, am I just making a simple mistake or is this whole setup of logging for websockets incorrect? If so what would be the better way (without needing to pass the session info with every log)?
I faced with same issue.
After shallow investigation, I can suppose following moments:
socket.io middlewares are not the same as in express.(not 100% sure)
There known issue https://github.com/nodejs/node/issues/32330 (closed but with tricky code)
To go forward with AsyncLocalStorage in sockets.io I do next steps:
// context.js
const uuid = require('uuid').v4;
const { AsyncLocalStorage } = require('async_hooks');
const context = new AsyncLocalStorage();
const enterWith = (data) => context.enterWith({traceId: uuid(), ...data });
module.exports = { context, enterWith };
// sockets.js
// I have legacy socket.io v2, your code may be different
io.use(contextMiddleware);
io.use(authSocket);
io.on('connection', (socket) => {
socket.on('USER_CONNECT', async () => {
socket.emit('Exo', `USER_CONNECT`);
try {
// The main solution is here, enter a valid context before actual controller execution
await enterWith({ userId: socket.chatuser });
await userService.createOrUpdateChatUser({ userId: socket.chatuser, customerId });
socket.emit('Exo', `User created`);
} catch (e) {
logger.error(`Create user failure ${e.message}`, { error: e });
socket.emit('Error', e.message);
}
});
Thanks for #bohdan for reminding me that this issue was still unanswered. While his solution works, I will also explain what I did for anyone wondering how to do this using middleware.
What I learned is that WebSockets can be very confusing but quite logical, for me the most important thing to realize was that you cannot use the "same" asyncLocalStorage for a single socket as long as that socket is connected. So I use a different asyncLocalStorage for each event (I will call them stores)
For me there are 4 different "stores" for a websocket connection Which cannot share the same store.
When a connection is made
When an event is received (frontend --> backend)
When an event is sent (backend --> frontend)
When a connection is closed
For all of these types I (mostly) use the same middleware:
import { AsyncLocalStorage } from "async_hooks";
import { Socket } from "socket.io"
import { v4 } from "uuid";
const context = mew AsyncLocalStorage();
const wsLogStore = (socket: Socket, next: () => void) => {
const newData: any = {
// Any data you want to save in the store
// For example socket Id
socketId: socket.id
// I also add an eventId which I can later use in my logging to combine all logs belonging to a single event
eventId: v4()
}
return context.run(newData, () => callBack())
}
#1 For the first type (when a connection is made)
You can use the middleware like this:
// Import the middleware we just created
import wsLogStore from "./wsLogStore"
// io = socketIO server instance (io = new Server)
io.use(wsLogStore)
Now a store will be available everywhere as long as it happens directly after the connection
#2 When a event is received (frontend --> backend)
io.use((socket, next) => {
socket.use((event, next) => {
wsLogStore(socket, () => {
next()
});
});
})
Now everywhere you use socket.on("<any event>") A store will have been created and usable
#3 When an event is sent (backend --> frontend)
Now this one is a little bit different since depending on your implementation this will not be easy, for example when you sent something to a specific room, is it enough to create a single store for the whole room? Or do you want to have a separate one for each socket that is receiving a event? And how do we create a store since we don't have a specific socket available?
For my use case it was absolutely necessary to have a separate store for each socket that is receiving an event.
const sentEventsToSockets = () => {
// Get the sockets you want to send a event to,
// For example, you could get the sockets from a room
const sockets = (Array.from(io.sockets.values()) as Socket[]).filter((socket) => socket.rooms.has("your room"))
for (const socket of sockets) {
wsLogStore(socket, () => {
//Here a separate store for each socket will be available
socket.emit("your event")
})
}
}
#4 When a connection is closed
Sadly, the store we created in step 1 is not available in this case so we would need to create a new one.
io.use((socket, next) => {
socket.on("disconnect", () => {
wsLogStore(socket, () => {
// A separate store will be available here if a connection is closed
});
});
})
Conclusion
While it would be easier if we could create a single store for each socket and use it the whole time, it seems like that is simply not possible.
By saving the socketId in our store we can however combine all data that we need afterwards. For example, in logging.
Note: If you use namespaces the socketId will be different for each namespace, you could use the connection id socket.conn.id which is a unique ID for each socket (no matter which namespace). Why this value is marked as private (if using TS) I have no clue
All of this will of course be slightly different depending on your use case and implementation. For example, if you use namespaces then you need to make sure the middleware is applied in each namespace.
I hope someone finds this helpful and if there are any question about how I do things or how to improve my setup, I would love to hear from you!

How to make a function wait for data to appear in the DB? NodeJS

I am facing a peculiar situation.
I have a backend system (nodejs) which is being called by FE (pretty standard :) ). This endpoint (nodejs) needs to call another system (external) and get the data it produces and return them to the FE. Until now it all might seem pretty usual but here comes the catch.
The external system has async processing and therefore responds to my request immediately but is still processing data (saves them in a DB) and I have to get those data from DB and return them to the FE.
And here goes the question: what is the best (efficient) way of doing it? It usually takes a couple of seconds only and I am very hesitant of making a loop inside the function and for the data to appear in the DB.
Another way would be to have the external system call an endpoint at the end of the processing (if possible - would need to check that with the partner) and wait in the original function until that endpoint is called (not sure exactly how to implement that - so if there is any documentation, article, tutorial, ... would appreciate it very much if you could share guys)
thx for the ideas!
I can give you an example that checks the Database and waits for a while if it can't find a record. And I made a fake database connection for example to work.
// Mocking starts
ObjectID = () => {};
const db = {
collection: {
find: () => {
return new Promise((resolve, reject) => {
// Mock like no record found
setTimeout(() => { console.log('No record found!'); resolve(false) }, 1500);
});
}
}
}
// Mocking ends
const STANDBY_TIME = 1000; // 1 sec
const RETRY = 5; // Retry 5 times
const test = async () => {
let haveFound = false;
let i = 0;
while (i < RETRY && !haveFound) {
// Check the database
haveFound = await checkDb();
// If no record found, increment the loop count
i++
}
}
const checkDb = () => {
return new Promise((resolve) => {
setTimeout(async () => {
record = await db.collection.find({ _id: ObjectID("12345") });
// Check whether you've found or not the record
if (record) return resolve(true);
resolve(false);
}, STANDBY_TIME);
});
}
test();

How do conditionally thread in a nodejs application

My website's api works fine and runs fast, except for one route. Since nodejs is single threaded, we wanted to make the api call DATA be separately threaded so that it doesnt block the rest of the incoming calls, and because it took too long to fork. Basically the code I want is like this:
router.get(req, res){
if(MasterThread){
create a thread to run the DATA functions
}
else{
res.json(getDATA())
}
}
is this possible at all? All the tutorials I found implied that I either had to use cluster, or my threading had to occur in my main.js, neither of which I want to do.
When I tried to set up the threading across 2 files, the imports for nodejs threading were always null, or my imports never worked.
So again, if this a possible thing?
You can use worker_threads for this case I use it for one of my sites and it works perfectly here is minimal example
const { Worker } = require('worker_threads')
const worker_script = path.join(__dirname, "./worker.js")
router.get('/', function(req, res) {
const worker = new Worker(worker_script, {
workerData: JSON.stringify(obj)
})
worker.on("error", (err) => console.log(err))
worker.on("exit", () => console.log("exit"))
worker.on("message", (data) => {
console.log(data)
res.send(data)
})
})
and here is the worker
const { parentPort, workerData, isMainThread } = require('worker_threads')
if (!isMainThread) {
console.log("workerData: ", workerData)
//do some heavy work with the data
var parsed = JSON.parse(workerData)
parentPort.postMessage(parsed)
}

Testing a callback nested in a promise

tl;dr I need to test that my method adds a row to a spreadsheet on successful load of a Google spreadsheet.
saveDataToGoogleSpreadSheet(conversationData){
return new Promise((resolve, reject) => {
Spreadsheet.load(this.getGoogleAPISettings(), (err, spreadsheet) => {
if (err) {
return reject(err);
}
return spreadsheet.receive((receivedError, rows, info) => {
if (receivedError) {
return reject(receivedError);
}
const rowData = this.getSpreadsheetRowData(conversationData, info.totalRows);
spreadsheet.add(rowData);
return spreadsheet.send(sendError => (sendError ? reject(sendError) : resolve()));
});
});
});
}
I tested the case one and case two of the function (the two first errors) but I couldn't do it for the last one, the case of success where we an add a row to a spreadsheet.
I need some help with the structure of the test, or a hint on how could my test be.
Edit: how the previous tests were made
it('should add a row to a Google Spreadsheet', (done) => {
nock('https://spreadsheets.google.com')
.post('/feeds/cells/1ZOd7Sysc-JNa-D5AHb7ZJkwBRMBGaeKpzIwEl7B8RbQ/1/private/full/batch')
.replyWithError({ message: 'abcde' });
api.saveDataToGoogleSpreadSheet({ data: 'some data' })
.then(() => done(new Error('should not have made the call')))
.catch((err) => {
expect(err).to.equal('Error Reading Spreadsheet');
done();
});
}).timeout(4000);
It is hard to tell what is the problem with the test from the little code you have, and no background on what the various objects are and how they come to be, so I will just assume that Spreadsheet is an object that is created through a library you require, and other than that, all other objects are created by the module. i.e. I assume you somewhere have a line resembling something like this:
const Spreadsheet = require('some-google-docs-spreadsheet-lib');
That means one problem is finding out how to control the Spreadsheet object so we can stub out its behavior.
Just to start you out, you might get some good pointers on general code and test structure for easy testing from these two answers, as they cover the two most relevant techniques: dependency injection and exploiting link seams.
Mocking Redis Constructor with Sinon
How to test an ES6 class that needs jquery?
For all I know, you might already utilize one of these techniques, as you say you have been able to test the two error situations. But maybe you have not really been unit testing and done the actual network calls to the service instead (which is more of an integration test)? Anyway, I'll assume no more than what I wrote above and show you how to do the testing using proxyquire:
const assert = require('assert');
const dummy = ()=> {};
const SpreadSheetStubLibrary = { load: dummy };
const MyClassToTest = proxyquire('../src/my-module', {
'some-google-docs-spreadsheet-lib': SpreadSheetStubLibrary
})
const config = {};
const conversationData = {};
let stubSetup;
let spreadsheet;
let myObj;
function setupStubs() {
stubSetup = stubSpreadsheetLoadFunction();
spreadsheet = stubSetup.spreadsheet;
SpreadSheetStubLibrary.load = stubSetup.load;
myObj = new MyClassToTest(config);
conversationData = {};
};
function createSpreadsheetStubObj(){
return {
receive: sinon.stub(),
add: sinon.stub(),
send: sinon.stub()
}
}
function stubSpreadsheetLoadFunction(){
const spreadsheet = createSpreadsheetStubObj();
return {
load: (settings, cb) => cb(null, spreadsheet),
spreadSheetStubObj: spreadsheet
};
}
it('should add a row to the spreadsheet on successful load', () => {
// Arrange
setupStubs();
const rowData = { foo: 1, bar: 2};
spreadsheet.receive.yields(); // calls any callback given
myObj.getSpreadsheetRowData = () => rowData; // see lines below
// if you want to use the real getSpreadsheetRowData, uncomment these lines
//const rows = []; // not used in the method?
//const info = { totalRows : 100 };
//spreadsheet.receive.yields(null, rows, info);
// Act
return myObj.saveDataToGoogleSpreadSheet(conversationData).then(()=>{
// Assert
assert(spreadsheet.add.calledOnce);
assert(spreadsheet.add.calledWith(rowData));
});
});
it('should add a row to the spreadsheet on successful load', () => {
// reuse the above
});
See the Sinon docs for the stub API.
Disclosure: I am part of the Sinon maintainer team.

Throttling Axios Requests

I'm using axios to make requests to the Deezer API. Unfortunately, with Deezer's API when you request an artist's albums it does not include album tracks. So, I am working around this by requesting the artist's albums and then performing a subsequent axios request for each album. The problem I'm running into is that the API limits requests to 50 per 5 seconds. If an artist has more than 50 albums I usually get a "quota exceeded" error. Is there a way to throttle axios requests to 50 per 5 seconds, specifically when using axios.all?
var axios = require('axios');
function getAlbums(artistID) {
axios.get(`https://api.deezer.com/artist/${artistID}/albums`)
.then((albums) => {
const urls = albums.data.data.map((album) => {
return axios.get(`https://api.deezer.com/album/${album.id}`)
.then(albumInfo => albumInfo.data);
});
axios.all(urls)
.then((allAlbums) => {
console.log(allAlbums);
});
}).catch((err) => {
console.log(err);
});
}
getAlbums(413);
First of all, let's see what you really need. Your goal here is to make request at most each 100 milliseconds, if you have a large number of albums. (Using axios.all for this matter is no different from using Promise.all, you just want to wait for all of the requests to complete.)
Now, with axios you have the interception API, allowing to plug your logic before requests. So you can use an interceptor like this:
function scheduleRequests(axiosInstance, intervalMs) {
let lastInvocationTime = undefined;
const scheduler = (config) => {
const now = Date.now();
if (lastInvocationTime) {
lastInvocationTime += intervalMs;
const waitPeriodForThisRequest = lastInvocationTime - now;
if (waitPeriodForThisRequest > 0) {
return new Promise((resolve) => {
setTimeout(
() => resolve(config),
waitPeriodForThisRequest);
});
}
}
lastInvocationTime = now;
return config;
}
axiosInstance.interceptors.request.use(scheduler);
}
What it does is timing requests so they are performed at intervalMs milliseconds intervals.
In your code:
function getAlbums(artistID) {
const deezerService = axios.create({ baseURL: 'https://api.deezer.com' });
scheduleRequests(deezerService, 100);
deezerService.get(`/artist/${artistID}/albums`)
.then((albums) => {
const urlRequests = albums.data.data.map(
(album) => deezerService
.get(`/album/${album.id}`)
.then(albumInfo => albumInfo.data));
//you need to 'return' here, otherwise any error in album
// requests will not propagate to the final 'catch':
return axios.all(urls).then(console.log);
})
.catch(console.log);
}
This is, however, a simplistic approach, in your case you probably would like to receive the results as fast as possible for number of requests less than 50. For this, you have to add some kind of a counter inside the scheduler which will count the number of requests and delay their execution based both on the interval and the counter.
Here is my solution using simple async setTimeout / Es6 code :
you can setup the delay in the sleep param func
const sleep = (delay) => {
return new Promise(function(resolve) {
setTimeout(resolve, delay);
});
}
axios.interceptors.response.use(async function (response) {
await sleep(3000)
return response;
}, function (error) {
// Do something with response error
console.error(error)
return Promise.reject(error);
});
There is also this npm package you can use :
https://www.npmjs.com/package/axios-request-throttle
There is a module that worked for me outside the box with a NestJS application
Throttle axios request-per-second rate with 3 lines of code. The main
difference with this module and others like axios-throttled is that
you don't have to create a new axios instance, and by extension don't
have to fix imports project-wide. Apply once and every axios.get,
post, put, delete etc is throttled.
axios-request-throttle

Resources