I'm running a NodeJS script that will generate several PDF reports.
Thing is I need to generate several graph for each PDFs, so after several problems, I decided to generate graphs in PNG format, then, make the html page including images. From the HTML, I generate a PDF.
Thing is I don't really need routes, but I need EJS, and I need req / res to generate my graphs:
app.get("/operations/:operation/meters/:meter/weekly_report", async (req, res) => { // Used to generate PNG from graph
const meterId = req.params.meter;
const week = req.query.week;
// get meters from meter
const meter = meters.find(it => it.prm === meterId);
const weeklyData = await generateWeeklyGraphForPRM(meter, week);
ejs.renderFile(path.join(__dirname, './views/partials/', "weekly_graph.ejs"), {
days: weeklyData.days,
conso: weeklyData.consoByHour,
meterLabel: meter.label,
}, (err) => {
if (err) {
res.send(err);
} else {
res.render('partials/weekly_graph.ejs', {
days: weeklyData.days,
conso: weeklyData.consoByHour,
meterLabel: meter.label,
});
}
});
And Then:
async function makePngScreenshot(url, meterId, filename) {
axios.get(url, null); // Make the request to generate html page
const destination = "public/images/" + operation.data.name + "/" + DATE_INI + "_" + DATE_END + "/" + meterId
return new Pageres({delay: 2, filename: filename})
.src(url, ['1300x650'], {crop: true})
.dest(destination)
.run()
}
});
Thing is working, but right now, everything is in index.js
I am trying to break the code into several files.
As I extract each routes into a routes.js, I have the problem that I can't share any longer global vars with all my endpoints.
So, here I find 3 solutions:
Use functions instead of endpoints: I don't need endpoints, but I don't know how to render an EJS file without routes, req / res.
In each routes, get each object again ( inefficient )
Use a redis, or any cache ( ok, but I would like to avoid any extra component for now )
The easiest one should be converting routes into functions, but how can I generate EJS files, without routes, is it possible ?
I hope I understand your task correctly. I made an example of a program that starts using the command line, receives the command line arguments meterId and week, generates a .html file from the .ejs template. I also used the yargs package to easily parse command line arguments.
const path = require('path');
const fs = require('fs');
const argv = require('yargs').argv;
const ejs = require('ejs');
const fsp = fs.promises;
// It would be a good idea to store these parameters in an .env file
const INPUT_FILENAME = 'test.ejs';
const OUTPUT_FILENAME = 'result.html';
const TEMPLATE_FILE = path.resolve(__dirname, './templates', INPUT_FILENAME);
const STORAGE_PATH = path.resolve(__dirname, './storage', OUTPUT_FILENAME);
(async function main({ meterId, week }) {
if (!meterId) {
return console.warn('Specify the command line parameter "meterId"!');
}
if (!week) {
return console.warn('Specify the command line parameter "week"!');
}
try {
const html = await ejs.renderFile(TEMPLATE_FILE, { meterId, week }, { async: true });
await fsp.writeFile(STORAGE_PATH, html);
console.log('Done.');
} catch (error) {
console.error(error);
process.exit(1);
}
})(argv);
And an example command to run the script:
node script.js --meterId=141 --week=44
Please let me know if I understood your task correctly and if my example helps somehow.
Related
As far as I know (correct me if i'm wrong please) the flow of downloading a file should be that the frontend make a call to an api route and everything else is going on on the server.
My task was to read from firestore and write it to the CSV file, I populated the CSV file with the data and now when I try to send it to the frontend only thing that is in the file after the download it the first line containing headers name and email (the file that was written on my computer is correctly willed with the data). This is my route
import { NextApiHandler } from "next";
import fs from "fs";
import { stringify } from "csv-stringify";
import { firestore } from "../../firestore";
import { unstable_getServerSession } from "next-auth/next";
import { authOptions } from "./auth/[...nextauth]";
const exportFromFirestoreHandler: NextApiHandler = async (req, res) => {
const session = await unstable_getServerSession(req, res, authOptions);
if (!session) {
return res.status(401).json({ message: "You must be authorized here" });
}
const filename = "guestlist.csv";
const writableStream = fs.createWriteStream(filename);
const columns = ["name", "email"];
const stringifier = stringify({ header: true, columns });
const querySnapshot = await firestore.collection("paprockibrzozowski").get();
await querySnapshot.docs.forEach((entry) => {
stringifier.write([entry.data().name, entry.data().email], "utf-8");
});
stringifier.pipe(writableStream);
const csvFile = await fs.promises.readFile(
`${process.cwd()}/${filename}`,
"utf-8"
);
res.status(200).setHeader("Content-Type", "text/csv").send(csvFile);
};
export default exportFromFirestoreHandler;
since I await querySnapshot and await readFile I would expect that the entire content of the file would be sent to the frontend. Can you please tell me what am I doing wrong?
Thanks
If anyone will struggle with this same stuff here is the answer base on the # Nelloverflowc thank you for getting me this far, hoverver files not always were populated with data, at first I tried like so
stringifier.on("close", async () => {
const csvFile = fs.readFileSync(`${process.cwd()}/${filename}`, "utf-8");
res
.status(200)
.setHeader("Content-Type", "text/csv")
.setHeader("Content-Disposition", `attachment; filename=${filename}`)
.send(csvFile);
});
stringifier.end();
the api of https://csv.js.org/ must have changed becuase instead of on.('finish') it is on close now, so reading file sync did the job regarding always getting the file populated with the correct data, but along with it there was an error
API resolved without sending a response for /api/export-from-db, this may result in stalled requests.
the solution to that is to convert file into readable stream like so
try {
const csvFile = fs.createReadStream(`${process.cwd()}/${filename}`);
res
.status(200)
.setHeader("Content-Type", "text/csv")
.setHeader("Content-Disposition", `attachment; filename=${filename}`)
.send(csvFile);
} catch (error) {
res.status(400).json({ error });
}
Here is the tread and the discussion that helped me
Node.js send file in response
The await on that forEach is most definitely not doing what you expect it to do, also you probably shouldn't use await and forEach together
Either switch to using the Sync API for the csv-stringify library or do something along these lines (assuming the first .get() actually contains the actual values from a promise):
[...]
stringifier.pipe(writableStream);
stringifier.on('finish', () => {
const csvFile = await fs.promises.readFile(
`${process.cwd()}/${filename}`,
"utf-8"
);
res.status(200).setHeader("Content-Type", "text/csv").send(csvFile);
});
for (const entry of querySnapshot.docs) {
stringifier.write([entry.data().name, entry.data().email], "utf-8");
);
stringifier.end();
[...]
I am writing a Firebase function that exposes an API endpoint using express. When the endpoint is called, it needs to download an image from an external API and use that image to make a second API call. The second API call needs the image to be passed as a readableStream. Specifically, I am calling the pinFileToIPFS endpoint of the Pinata API.
My Firebase function is using axios to download the image and fs to write the image to /tmp. Then I am using fs to read the image, convert it to a readableStream and send it to Pinata.
A stripped-down version of my code looks like this:
const functions = require("firebase-functions");
const express = require("express");
const axios = require("axios");
const fs = require('fs-extra')
require("dotenv").config();
const key = process.env.REACT_APP_PINATA_KEY;
const secret = process.env.REACT_APP_PINATA_SECRET;
const pinataSDK = require('#pinata/sdk');
const pinata = pinataSDK(key, secret);
const app = express();
const downloadFile = async (fileUrl, downloadFilePath) => {
try {
const response = await axios({
method: 'GET',
url: fileUrl,
responseType: 'stream',
});
// pipe the result stream into a file on disc
response.data.pipe(fs.createWriteStream(downloadFilePath, {flags:'w'}))
// return a promise and resolve when download finishes
return new Promise((resolve, reject) => {
response.data.on('end', () => {
resolve()
})
response.data.on('error', () => {
reject()
})
})
} catch (err) {
console.log('Failed to download image')
console.log(err);
throw new Error(err);
}
};
app.post('/pinata/pinFileToIPFS', cors(), async (req, res) => {
const id = req.query.id;
var url = '<URL of API endpoint to download the image>';
await fs.ensureDir('/tmp');
if (fs.existsSync('/tmp')) {
console.log('Folder: /tmp exists!')
} else {
console.log('Folder: /tmp does not exist!')
}
var filename = '/tmp/image-'+id+'.png';
downloadFile(url, filename);
if (fs.existsSync(filename)) {
console.log('File: ' + filename + ' exists!')
} else {
console.log('File: ' + filename + ' does not exist!')
}
var image = fs.createReadStream(filename);
const options = {
pinataOptions: {cidVersion: 1}
};
pinata.pinFileToIPFS(image, options).then((result) => {
console.log(JSON.stringify(result));
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "Authorization, Origin, X-Requested-With, Accept");
res.status(200).json(JSON.stringify(result));
res.send();
}).catch((err) => {
console.log('Failed to pin file');
console.log(err);
res.status(500).json(JSON.stringify(err));
res.send();
});
});
exports.api = functions.https.onRequest(app);
Interestingly, my debug messages tell me that the /tmp folder exists, but the file of my downloaded file does not exist in the file system.
[Error: ENOENT: no such file or directory, open '/tmp/image-314502.png']. Note that the image can be accessed correctly when I manually access the URL of the image.
I've tried to download and save the file using many ways but none of them work. Also, based on what I've read, Firebase Functions allow to write and read temp files from /tmp.
Any advice will be appreciated. Note that I am very new to NodeJS and to Firebase, so please excuse my basic code.
Many thanks!
I was not able to see you are initializing the directory as suggested in this post:
const bucket = gcs.bucket(object.bucket);
const filePath = object.name;
const fileName = filePath.split('/').pop();
const thumbFileName = 'thumb_' + fileName;
const workingDir = join(tmpdir(), `${object.name.split('/')[0]}/`);//new
const tmpFilePath = join(workingDir, fileName);
const tmpThumbPath = join(workingDir, thumbFileName);
await fs.ensureDir(workingDir);
Also, please consider that if you are using two functions, the /tmp directory would not be shared as each one has its own. Here is an explanation from Doug Stevenson. In the same answer, there is a very well explained video about local and global scopes and how to use the tmp directory:
Cloud Functions only allows one function to run at a time in a particular server instance. Functions running in parallel run on different server instances, which have different /tmp spaces. Each function invocation runs in complete isolation from each other. You should always clean up files you write in /tmp so that they don't accumulate and cause a server instance to run out of memory over time.
I would suggest using Google Cloud Storage extended with Cloud Functions to achieve your goal.
Making a basic blog with an admin section to learn the basics of node and express. I just implemented multer middleware to save images for a blog post to a folder ("images") on the server - not to mongo or an s3 bucket - keeping it simple for now to learn.
I am using EJS and using res.render to send and render the frontend. However, I want to put the image in the EJS file as well. I've tried simply passing in the filename like so:
res.render(path.resolve(__dirname, "../", "views", "posts.ejs"), {postData, file});
postData being the data on the post from the mongodb collection. All this does is send the filename itself which is not helpful.
I've looked around, but don't seem to find an answer to this, or I'm over thinking this?
Here is the rest of the code for the controller:
const path = require("path");
const fs = require('fs');
const Post = require('../models/modelPosts');
exports.getPost = (req, res, next) => {
const postPath = req.params.post;
Post.findOne({ postPath: postPath }, (error, postData) => {
if (error) { return next(error) };
if (postData.postReadyToView == true) {
// find the correct image
fs.readdirSync('./images').forEach(file => {
const stringOfFile = JSON.stringify(file);
const stringOfPathJPEG = JSON.stringify(postPath + ".jpeg");
const stringOfPathJPG = JSON.stringify(postPath + ".jpg");
const stringOfPathPNG = JSON.stringify(postPath + ".png")
// send the ejs file and image
if (stringOfFile == stringOfPathJPEG ||
stringOfFile == stringOfPathJPG ||
stringOfFile == stringOfPathPNG) {
res.render(path.resolve(__dirname, "../", "views", "posts.ejs"), {
postData, file
});
}
})
} else {
res.redirect(404, "/404");
}
})
};
Send the file path of the page to be rendered as data, register the image garden folder (ex: public/images) as a static folder using express.static in nodejs, and load the image when the file path is loaded in the rendering page. I think you can.
I'm trying to consolidate a bunch of route usage throughout my Express API, and I'm hoping there's a way I can do something like this:
const app = express()
const get = {
fetchByHostname({
name
}) {
return `hey ${name}`
}
}
const map = {
'/public/hostname/:hostname': get.fetchByHostname
}
app.use((req, res, next) => {
const url = req.originalUrl
const args = { ...req.body, ...req.query }
const method = map[url] // this won't work
const result = method(args)
return res.json({
data: result
})
})
I'm trying to avoid passing round the req and res objects and just handle the response to the client in one place. Is there an Express/Node/.js module or way to match the URL, like my map object above?
I really don't understand what you are trying to achieve, but from what i can see, your fectchByHostname({name})should be fetchByHostname(name) and you might be able to return hey $name. You should be sure you are using ES6 because with you args. Else you have to define the as in es5 args = {body: req.body, query: req.query};. Hope it helps.
I want to export my data into csv file so for that purpose i used fast-csv in node js. its working fine my code is
var csv = require("fast-csv");
app.get('/file/exportToExcel', function(req, res) {
var whereCondi = {
'Id': req.query.id,
}
var response = {};
table.find(whereCondi, function(err, tableData) {
if (err) {
response.status = 400;
response.message = err.message;
res.send(response);
} else {
var csvStream = csv.createWriteStream({headers: true}),
writableStream = fs.createWriteStream("code.csv");
writableStream.on("finish", function() {
});
csvStream.pipe(writableStream);
for (var i = 0; i < tableData.length; i++) {
csvStream.write({allcodes: tableData[i].code});
}
csvStream.end();
}
});
});
but the problem is its saving that csv file in my root folder i want to download that csv file when user click on export to excel please help me.
writableStream = fs.createWriteStream("coupons.csv");
This looks to be your problem if I'm understanding you correctly. The current code saves the csv file relative to the app file (basically in the same directory in your case).
Try something like:
writableStream = fs.createWriteStream("./some/directory/coupons.csv");
You should create the csv file in your directory an then delete it in the same way like that
const express = require('express')
const objectstocsv = require('objects-to-csv')
const fs = require("fs")
const app = express()
var data = [
{code: 'CA', name: 'California'},
{code: 'TX', name: 'Texas'},
{code: 'NY', name: 'New York'},
];
const PORT = process.env.PORT || 5000
app.get('/',async(req,res) => {
const csv = new objectstocsv(data);
// Save to file:
await csv.toDisk('./test.csv');
// Download the file
res.download("./test.csv",() => {
//Then delete the csv file in the callback
fs.unlinkSync("./test.csv")
})
})
Very late to the game but wanted to add in case other people were encountering same hurdle. Not sure if this is an ideal solution since I just started learning, but I got around this problem by wrapping the csv creation in an async function, and having that function called when a user clicks on a button.
Essentially, user clicks button > GET request to specific path > export csv and render success page.
index.js or server js file
const exportToCsv = () => {
...some code to get data and generate csv...
};
app.get('/download', async (req, res) => {
exportToCsv();
res.render('<your desired view>');
};
view or html
<button type='submit' onclick='someNavFunction()'>Export</button>
The someNavFunction() can be a helper function to navigate to a new path or some other solution that helps you hit '/download' that you created route for in server file.
This solution worked for me because I wanted to render a success page after download. You can add additional validation to only render if exported successfully etc.