I have this route in my app. Its scanning file names in some directories according to url parameter. Currently my JSON looks like;
[{"chapter":"642","paths":["00.png","01.png","02.png","03.png","04.png","05.png".....
Route:
app.get('/api/oku/:name', function (req, res) {
var files = fs.readdirSync('./public/manga/' + req.params.name);
var files = files.map(function(item){
var subfolders = fs.readdirSync('./public/manga/' + req.params.name + '/' + item);
return {
chapter: item,
paths: subfolders
}
})
res.json(files);
})
Simply I want to add paths to every image. Path that I want to add is;
"manga/" + req.params.name + "/" + item + "/"
So simply it will be looks like manga/Naruto/542/00.png
Try this:
app.get('/api/oku/:name', function (req, res) {
var files = fs.readdirSync('./public/manga/' + req.params.name).map(function(item){
var subfolders = fs.readdirSync('./public/manga/' + req.params.name + '/' + item);
return {
chapter: item,
paths: subfolders.map(function (i) {
return "manga/" + req.params.name + "/" + i + "/";
}
}
})
res.json(files);
})
Also, don't declare the files variable twice.
Related
I try to make a Canva App with a publish extension.
I just follow the Quick start (https://docs.developer.canva.com/apps/extensions/publish-extensions/quick-start) with Glitch and it work well on it.
But when I try to put in on my own public host name, with a other port (like http://mydomaine.com:3000) Canva NEVER call my endpoint. I just write a log file of every action on my app post and I never get a update on it, and when I try the app on Canva.com it just show me a error message.
//Copy from the Quick Start
app.post('/publish/resources/upload', async (request, response) => {
try{
writeLog("Uploading file");
await fs.ensureDir(path.join(__dirname, 'export'));
// Get the first asset from the "assets" array
const [asset] = request.body.assets;
// Download the asset
const image = await jimp.read(asset.url);
const filePath = path.join(__dirname, 'export', asset.name);
await image.writeAsync(filePath);
// Respond with the URL of the published design
response.send({
type: 'SUCCESS',
url: url.format({
protocol: request.protocol,
host: request.get('host'),
pathname: asset.name,
}),
});
} catch (err) {
writeLog("ERROR (app.post('/publish/resources/upload'): " + err);
}
});
//Just log on the log file
function writeLog(log){
// fs.appendFile(path.join(__dirname, '/log/' + `${month}/${date}/${year}` +'log.txt'), dateDisplay + "|" + log + "\n", (err) => {
// if (err) throw err;
// });
var today = new Date();
var time = today.getHours() + ":" + today.getMinutes() + ":" + today.getSeconds();
var date = today.getFullYear() + '-' + (today.getMonth() + 1) + '-' + today.getDate();
var dateTime = date + ' ' + time;
natifFS.appendFile('log.txt', dateTime + '| '+ log + "\n", (err) => {
if (err) throw err;
});
}
Last thing, when I try to call a post request on the same endpoint as Canva (/publish/resources/upload) with Postman, I get a update on my log.txt file
If anyone has idea, thank you.
I think I'm very close to what I want to do. I have the following api get method in node.js that is retrieving a file varbinary(MAX) from an SQL Server database. It was converted from a base64 encoded string before inserted so the Content Type information was stripped from the string.
node.js
router.get('/getFile', (req, res) => {
console.log("Calling getFile for file " + req.query.serialNumber + ".")
var serialNumber = req.query.serialNumber;
let request = new sql.Request(conn);
request.query('SELECT FileName + \'.\' + FileExtension AS \'File\', FileType, ContentType, SerialNumber, Chart ' +
'FROM dbo.ChangeFiles ' +
'WHERE SerialNumber = ' + serialNumber)
.then(function (recordset) {
log("Successfully retrieved file " + recordset[0].SerialNumber + " from database.");
log("Length of blob " + recordset[0].File + " is " + recordset[0].Chart.length)
res.status(200);
res.setHeader('Content-Type', recordset[0].ContentType);
res.setHeader('Content-Disposition', 'attachment;filename=' + recordset[0].File);
res.end(Buffer.from((recordset[0].Chart)));
}).catch(function (err) {
log(err);
res.status(500).send("Issue querying database!");
});
});
That works fine, but what to do in Angular to retrieve it and prompt for a download for the user has not been clear for me, nor has there been a lot as far as help/resources online. Here is what I have so far in my service class.
fileDownload.service.ts
downloadFile(serialNumber: string): Observable<any> {
return this.http.get(this.baseURL + '/getFile', { params: { serialNumber: serialNumber } })
.map(this.extractFile);
}
private extractFile(response: Response) {
const file = new Blob([response.blob]);
FileSaver.saveAs(file);
// const url = window.URL.createObjectURL(file);
// window.open(url);
return file;
}
As you can see I've tried a couple of approaches. The commented out portion of the extractFile method didn't work at all, and using the FileSaver.saveAs function produces a file download of an unknown type, so the headers sent from node.js didn't seem to affect the file itself.
Would someone be able to advise how to proceed in Angular with what is successfully being sent from node.js so that I can successfully download the file, regardless of type?
Thanks so much in advance.
I got it working afterall. I had to rework the api call so that it sent all of the file information separately so that the MIME type, and file name can be assigned to the file on the client side in the component class. For some reason when I tried to do so all in the api, it wouldn't work so that was my work around. So here is what works for me.
node.js api
router.get('/getFile', (req, res) => {
console.log("Calling getFile for file " + req.query.serialNumber + ".")
var serialNumber = req.query.serialNumber;
let request = new sql.Request(conn);
request.query('SELECT FileName + \'.\' + FileExtension AS \'File\', FileType, ContentType, SerialNumber, Chart ' +
'FROM dbo.ChangeFiles ' +
'WHERE SerialNumber = ' + serialNumber)
.then(function (recordset) {
log("Successfully retrieved file " + recordset[0].SerialNumber + " from database.");
log("Length of blob " + recordset[0].File + " is " + recordset[0].Chart.length)
res.send(recordset[0]);
}).catch(function (err) {
log(err);
res.status(500).send("Issue querying database!");
});
});
component class
downloadFile(serialNumber: string): void {
this.changeService.downloadFile(serialNumber).subscribe((res: any) => {
const ab = new ArrayBuffer(res.Chart.data.length);
const view = new Uint8Array(ab);
for (let i = 0; i < res.Chart.data.length; i++) {
view[i] = res.Chart.data[i];
}
const file = new Blob([ab], { type: res.ContentType });
FileSaver.saveAs(file, res.File);
console.log(res);
});
}
service class
downloadFile(serialNumber: string): Observable<any> {
return this.http.get(this.baseURL + '/getFile', { params: { serialNumber: serialNumber } })
.map(this.extractFile);
}
private extractFile(response: Response) {
// const file = new Blob([response.blob]);
// FileSaver.saveAs(file);
// const url = window.URL.createObjectURL(file);
// window.open(url);
const body = response.json();
return body || {};
}
Update your code to call subscribe instead of map
I have a project in node.js for which I want to automate some backup and revision tasks in GULP.
I am able to successfully test following gulp code in terminal and everything working perfectly. The issue comes when I run gulp task from node.js code.
Gulp code:
var gulp = require('gulp');
var runSequence = require('run-sequence');
var rev = require('gulp-rev');
var format = require('date-format');
var dt = (new Date());
var gBkup = __dirname + '/backup/' + format.asString('ddMMyyyy_hhmm', dt);
var config = __dirname + '/gHelper.json', mnf = __dirname + '/rev-manifest.json';
var cssSrc = [], cssSrcO = [], cssSrcI = [];
var dirSrc = [], dirSrcO = [], dirSrcI = [];
gulp.task('init', function (){ // Initialize paths and all arrays containing file paths
var fexists = require('file-exists');
//console.log('Config exists: ' + fexists(config));
if (fexists(config)) {
config = require(config);
}
//console.log('Config object: ' + config);
if (fexists(mnf)) {
mnf = require(mnf);
}
for (var file in config.revision.ext_css) {
var fnm = __dirname + '/preview/' + config.revision.ext_css[file];
cssSrc.push(fnm);
if (mnf[config.revision.ext_css[file]] != "") {
var hnm = __dirname + '/live/' + mnf[config.revision.ext_css[file]];
cssSrcO.push(hnm);
console.log("Manifest: " + hnm);
}
}
for (var dir in config.revision.dir) {
dirSrc.push(__dirname + '/preview/' + config.revision.dir[dir]);
var dirnm = __dirname + '/live/' + config.revision.dir[dir];
dirnm = dirnm.substr(0, dirnm.length-3);
dirSrcO.push(dirnm);
console.log("Directory: " + dirnm);
}
// Files and directories will be ignored in revision
for (var file in config.revision.ext_css) {
cssSrcI.push('!' + __dirname + '/preview/' + config.revision.ext_css[file]);
}
for (var dir in config.revision.dir) {
dirSrcI.push('!' + __dirname + './preview/' + config.revision.dir[dir]);
}
//console.log('Ignore CSS: ' + cssSrcI);
//console.log('Ignore DIR: ' + dirSrcI);
});
// Revisioning Files
gulp.task('revisionCSS', function() { // Revise CSS scripts
var cssDest = __dirname + config.revision.ext_css_dest;
console.log('cssDestination: ' + cssDest);
return gulp.src(cssSrc)
.pipe(rev())
.pipe(gulp.dest(cssDest))
.pipe(rev.manifest({base: cssDest, merge: true}))
.pipe(gulp.dest(cssDest))
});
gulp.task('revInnerScripts', function () { // Revise javascripts
var dirDest = __dirname + config.revision.ext_dir_dest;
var cssDest = __dirname + config.revision.ext_css_dest;
console.log('dirInner: ' + dirDest);
console.log('cssInner: ' + cssDest);
return gulp.src(dirSrc)
.pipe(rev())
.pipe(gulp.dest(dirDest))
.pipe(rev.manifest({base: cssDest, merge: true}))
.pipe(gulp.dest(cssDest));
});
gulp.task('copyIgnoreRevision', function() { // Simply copy other/ignored files from array
var src = [__dirname + '/preview/**']
src = src.concat(cssSrcI);
src = src.concat(dirSrcI);
console.log(src)
return gulp.src(src)
.pipe(gulp.dest(__dirname + '/live'));
});
gulp.task('removeLive', function(callback) { // Removing files
var del = require('del');
var src = cssSrcO.concat(dirSrcO);
console.log("Removing Files: " + src);
return del(src);
});
gulp.task('backupLive', function() { // Backing up revision files before taking revision
// var src = ['./Live/**'];
gulp.src(cssSrcO).pipe(gulp.dest(gBkup));
return gulp.src(dirSrcO).pipe(gulp.dest(gBkup + "/js"));;
/* return gulp.src(cssSrcO, {read: false})
.pipe(clean());*/
});
gulp.task('backup', function(callback) { // Backup tasks list
runSequence('backupLive', 'removeLive', callback);
});
gulp.task('revise', ['copyIgnoreRevision', 'revisionCSS', 'revInnerScripts']);
gulp.task('revback', function (callback) {
runSequence('init', 'backup', 'revreplace', callback);
});
// Replacing references
gulp.task('revreplace', ['revise'], function(callback) { // In callback replace references for revised files
var revReplace = require('gulp-rev-replace');
var mReps = gulp.src(__dirname + '/rev-manifest.json');
return gulp.src(__dirname + '/preview/*.html')
.pipe(revReplace({manifest: mReps}))
.pipe(gulp.dest(__dirname + '/live'));
});
gHelper.json: Listing files which needs to be revised. Everything else will be copied to destination directory.
{
"revision": {
"ext_css" : [
"extie.css",
"responsive.css",
"style.css"
],
"ext_css_dest": "/live",
"dir": [
"js/*.js"
],
"ext_dir_dest": "/live/js"
}
}
Basic folder structure:
MainFolder/
gHelper.json
gulpfile.js
preview/
HTML files which contains references to revision files
Revision files (CSS and JS). CSS files are mentioned in gHelper.json
js/
Revision files (mainly js) which are to be revised as this folder is mentioned in gHelper.json and all files from the folder will be revised
When gulp task revback is invoked a folder live will be generated and added inside MainFolder. Again when revback is invoked, first, backup/{timestamp} folder will be generated taking backup of revised files only and then revision is made for live folder.
Lets see code from Node.js:
/* Publish client */
var gulp = require('gulp');
router.post('/api/:clientName/publish', function(req, res, next) {
var clientName = req.params.clientName;
var filePath = '/gulpfile'; // Full path for gulp file
console.log("Publish client: " + filePath);
try {
var gtask = require(filePath);
if (gulp.tasks.revback) {
console.log('gulp file contains task!');
gulp.start('revback');
}
} catch (err) {
return console.error(err);
}
});
Now the problem comes that sometimes gulp tasks are not being completed, rev-manifest.json is not created at proper position means inside MainFolder but created outside in the folder where this node.js lies.
Please let me know how to resolve the issue, Thanks.
Below is content of rev-manifest.json:
{
"dctConf.js": "dctConf-7c467cb7cb.js",
"extie.css": "extie-a8724bfb0c.css",
"responsive.css": "responsive-76492b9ad4.css",
"style.css": "style-770db73beb.css",
"translation.js": "translation-9687245bfb.js"
}
You could try changing the working directory in your gulpfile.js to the location of the gulpfile. Just add this at the top
process.chdir(__dirname);
Docs
https://nodejs.org/docs/latest/api/process.html#process_process_chdir_directory
https://nodejs.org/docs/latest/api/globals.html#globals_dirname
I have used gulp's native callbacks and removed run-sequence module.
E.g.:
gulp.task('revback', ['revise'], function(callback) {
var revReplace = require('gulp-rev-replace');
var mReps = gulp.src(__dirname + '/rev-manifest.json');
console.log('Manifest content: ' + mReps + ' && ' + __dirname + '/rev-manifest.json');
return gulp.src(__dirname + '/preview/*.html')
.pipe(revReplace({manifest: mReps}))
.pipe(gulp.dest(__dirname + '/live'))
.once('error', function(e) {
console.log('Error at revback: ' + e);
callback(e);
process.exit(1);
})
.once('end', function() {
console.log('Ending process at revback!');
callback();
process.exit();
});
});
gulp.task('revise', ['copyIgnoreRevision', 'revisionCSS', 'revInnerScripts']);
gulp.task('backupLive', ['init'], function() {
// var src = ['./Live/**'];
gulp.src(cssSrcO).pipe(gulp.dest(gBkup));
return gulp.src(dirSrcO).pipe(gulp.dest(gBkup + "/js"));
/* return gulp.src(cssSrcO, {read: false})
.pipe(clean());*/
});
This way, reverse chained to init function.
I am using loopback for storing Image to the server.
I want to modify the file name of the file before getting saved to the server.
Also I want to convert it to another thumbnail form before getting saved.
Here is how I am doing.
At client side
Upload.upload(
{
url: '/api/containers/container_name/upload',
file: file,
fileName: "demoImage.jpg",
//Additional data with file
params:{
orderId: 1,
customerId: 1
}
});
At Server Side I am receiving the query "params" but not getting the "File Name"
My Storage model name is container
Container.beforeRemote('upload', function(ctx, modelInstance, next) {
//OUPTUTS: {orderId:1, customerId:1]}
console.log(ctx.req.query);
//Now I want to change the File Name of the file.
//But not getting how to do that
next();
})
How to change the File name of the File getting saved at the server?
I figured it out.
We have to define a custom function getFileName in boot/configure-storage.js.
Suppose my datasource for loopback-component-storage is presImage.
server/boot/configure-storage.js
module.exports = function(app) {
//Function for checking the file type..
app.dataSources.presImage.connector.getFilename = function(file, req, res) {
//First checking the file type..
var pattern = /^image\/.+$/;
var value = pattern.test(file.type);
if(value ){
var fileExtension = file.name.split('.').pop();
var container = file.container;
var time = new Date().getTime();
var query = req.query;
var customerId = query.customerId;
var orderId = query.orderId;
//Now preparing the file name..
//customerId_time_orderId.extension
var NewFileName = '' + customerId + '_' + time + '_' + orderId + '.' + fileExtension;
//And the file name will be saved as defined..
return NewFileName;
}
else{
throw "FileTypeError: Only File of Image type is accepted.";
}
};
}
common/models/container.js
Now suppose my container model is container.
module.exports = function(Container) {
Container.afterRemote('upload', function(ctx, modelInstance, next) {
var files = ctx.result.result.files.file;
for(var i=0; i<files.length; i++){
var ModifiedfileName = files[i].name;
console.log(ModifiedfileName) //outputs the modified file name.
} //for loop
next();
}); //afterRemote..
};
Now for converting it images to Thumbnail size
Download the quickthumb
Here is how to use it with loopback.
This code is copied directly from Loopback thumbnail view
common/models/container.js
module.exports = function(Container) {
var qt = require('quickthumb');
Container.afterRemote('upload', function(ctx, res, next) {
var file = res.result.files.file[0];
var file_path = "./server/storage/" + file.container + "/" + file.name;
var file_thumb_path = "./server/storage/" + file.container + "/thumb/" + file.name;
qt.convert({
src: file_path,
dst: file_thumb_path,
width: 100
}, function (err, path) {
});
next();
});
};
Piggybacking on the answer above, this configure-storage enables the file name to be set explicitly via req.params.filename and to default to the existing name if none is provided.
configure-storage.js
module.exports = function(app) {
//Function for checking the file type..
app.dataSources.storage.connector.getFilename = function(file, req, ignoreRes) {
if (!req.params.filename) {
return file.name
}
var fileExtension = file.name.split('.').pop()
return req.params.filename + '.' + fileExtension
};
}
I am saving user uploads by renaming it's original name to userID + '_' + new Date().getTime() + fileExt. I am storing the file properties in mongodb collections as :
{
name : String //generated name
, originalName : String //its original name
...
}
Now when the user requests for downloading of file I have to provide the file with its original name (which is persisted in db so no problem to grab it) to the user.
For the following request
GET /users/:userId/uploads/:fileId?type=download
I have this handler
//the mongoose query
UserUpload.findById(req.params.fileId).exec(function(err, doc){
var fileLocation = __dirname + '/public/uploads/users/' + req.params.userId + '/' + doc.name;
if(req.query.type && req.query.type == 'download') {
// I don't know what should I do now
// Downloader.download(fileLocation, newName);
}
});
I read wiki of node-static module but could not figure out how to do that job?
I found the answer here : Download a file from NodeJS Server using Express . Both using express and without using express.
It is too simple if you are using Express. Here is the documentation for res.download. I can't believe that the solution is just one line of code :
res.download('/path/to/file.ext', 'newname.ext');
Here what I use in one of my project, smil is a type of file I need to download, nevermind it.
In that project I have DOWNLOAD_DIR as global variable, which contain the full path to the download folder.
It may make a lot of people cringe (especcially the fileExistSync) but it s a start.
var DOWNLOAD_DIR = '/path/to/download',
url = require('url'),
http = require('http'),
/*
download
IN_:
file_url, url of the file to download
OUT:
file_name, full path to downloaded file, null if the file cound t be downloaded.
COM:
download a file, the file name will be the last part of the url (http://why.so/serious.txt => serious.txt).
WARNING: Do NOT follow redirections.
*/
function download(file_url, callback) {
var options = {host: url.parse(file_url).host, port: 80, path: url.parse(file_url).pathname},
file_name = url.parse(file_url).pathname.split('/').pop(),
//Creating the file
file = fs.createWriteStream(DOWNLOAD_DIR + file_name, {flags: 'w', encoding: 'binary'});
console.log('Downloading file from ' + file_url);
console.log('\tto ' + file_name);
http.get(options, function (res) {
res.pipe(file, {end: 'false'});
//When the file is complete
res.on('end', function () {
//Closing the file
file.end();
console.log('\t\tDownloaded '+ file_name);
callback(DOWNLOAD_DIR + file_name);
});
});
process.on('uncaughtException', function(err) {
console.log('Can t download ' + file_url + '\t(' + err + ')', false);
callback(null);
});
}
/*
download_smil
IN_:
file_url, url of the file to download
OUT:
file_name, full path to downloaded file
COM:
Follow http redirection and then call download function to download it.
You can modify the cad function to use custom names.
*/
function download_smil(file_url, callback) {
function cad(link, callback) {
//Does the file already exist?
var file = url.parse(link).pathname.substr(url.parse(link).pathname.lastIndexOf('/') + 1),
pkmn;
pkmn = fs.existsSync(DOWNLOAD_DIR + '/' + file);
if (pkmn) {
//YES: callback
console.log('File ' + file + ' already exist, skipping');
callback(DOWNLOAD_DIR + file, true);
} else {
//NO: Download it
console.log('Will download ' + link);
download(link, callback);
}
}
//GET the page
http.get(file_url, function (res) {
var link;
//console.log('res.statusCode = ' + res.statusCode + ' res.headers.location = ' + res.headers.location);
//Check if it is a redirect
if (res.statusCode > 300 && res.statusCode < 400 && res.headers.location) {
//console.log('redirect');
//Check if the hostname is in the location
if (url.parse(res.headers.location).hostname) {
//console.log('The link to the smil is res.headers.location');
link = res.headers.location;
cad(link, callback);
} else {
//console.log('The link to the smil is url_parse(file_url).hostname + res.headers.location = ' + url_parse(file_url).hostname + res.headers.location);
link = url_parse(file_url).hostname + res.headers.location;
cad(link, callback);
}
} else {
//console.log('The url is good : ' + file_url);
cad(file_url, callback);
}
});
}