How do you retrieve a file if your node express web service is on a linux server (ubuntu) and you need to download files from a windows server?
2 Valid Options:
Windows Servers do not support SFTP/SSH. There are some ways to do this (with https://winscp.net/eng/docs/guide_windows_openssh_server and https://www.npmjs.com/package/ssh2) but this requires our server administrator to install on windows and it isn't fully supported by Microsoft.
You could use FTP/FTPS. There's this package: https://github.com/Atinux/node-ftps I just wasn't able to get this to work but it should be a viable option.
How can you do this from the operating system directly instead of relying on third-party node packages?
You can use smbget (linux utility included with https://linux.die.net/man/1/smbget ) and then just use node child_process spawn to call the function.
Just replace [workgroup], [username], [password], [serveraddress], and [path] with your own information here.
function getFile(file) {
return new Promise(function(resolve, reject) {
var tempFilePath = `/tmp/${file}`; // the linux machine
var remoteFile = require('child_process').spawn('smbget', ['--outputfile', tempFilePath, '--workgroup=[workgroup]', '-u', '[username]', '-p', '[password]', `smb://[serveraddress]/c$/[path]/[path]/${file}`]);
remoteFile.stdout.on('data', function(chunk) {
// //handle chunk of data
});
remoteFile.on('exit', function() {
//file loaded completely, continue doing stuff
// TODO: make sure the file exists on local drive before resolving.. an error should be returned if the file does not exist on WINDOWS machine
resolve(tempFilePath);
});
remoteFile.on('error', function(err) {
reject(err);
})
})
}
The code snippet above returns a promise. So in node you could send the response to a route like this:
var express = require('express'),
router = express.Router(),
retrieveFile = require('../[filename-where-above-function-is]');
router.route('/download/:file').get(function(req, res) {
retrieveFile.getFile(req.params.file).then(
file => {
res.status(200);
res.download(file, function(err) {
if (err) {
// handle error, but keep in mind the response may be partially sent
// so check res.headersSent
} else {
// remove the temp file from this server
fs.unlinkSync(file); // this delete the file!
}
});
})
.catch(err => {
console.log(err);
res.status(500).json(err);
})
}
The response will be the actual binary for the file to download. Since this file was retrieved from a remote server, we also need to make sure we delete the local file using fs.unlinkSync().
Use res.download to send the proper headers with the response so that most modern web browsers will know to prompt the user to download the file.
Related
So I need to deploy a Google Cloud Function that allow me to make 2 things.
The first one is to DOWNLOAD any files on SFTP/FTP server on /tmp local directory of the Cloud Function. Then, the second step, is to UPLOAD this file in a bucket on the Google Cloud Storage.
Actually I know how to upload but I don't get how to DOWNLOAD files from ftp server to my local /tmp directory.
So, actually I have written a GCF that receive in parameters (on the body), the configuration (config) to allow me to connect on the FTP server, the filename and the path.
For my test I used the following ftp server test: https://www.sftp.net/public-online-sftp-servers with this configuration.
{
config:
{
hostname: 'test.rebex.net',
username: 'demo',
port: 22,
password: 'password'
},
filename: 'FtpDownloader.png',
path: '/pub/example'
}
After my DOWNLOAD, I start my UPLOAD. For that I check if I found the DOWNLOAD file in '/tmp/filename' before to UPLOAD but the file is nerver here.
See the following code:
exports.transferSFTP = (req, res) =>
{
let body = req.body;
if(body.config)
{
if(body.filename)
{
//DOWNLOAD
const Client = require('ssh2-sftp-client');
const fs = require('fs');
const client = new Client();
let remotePath
if(body.path)
remotePath = body.path + "/" + body.filename;
else
remotePath = "/" + body.filename;
let dst = fs.createWriteStream('/tmp/' + body.filename);
client.connect(body.config)
.then(() => {
console.log("Client is connected !");
return client.get(remotePath, dst);
})
.catch(err =>
{
res.status(500);
res.send(err.message);
})
.finally(() => client.end());
//UPLOAD
const {Storage} = require('#google-cloud/storage');
const storage = new Storage({projectId: 'my-project-id'});
const bucket = storage.bucket('my-bucket-name');
const file = bucket.file(body.filename);
fs.stat('/tmp/' + body.filename,(err, stats) =>
{
if(stats.isDirectory())
{
fs.createReadStream('/tmp/' + body.filename)
.pipe(file.createWriteStream())
.on('error', (err) => console.error(err))
.on('finish', () => console.log('The file upload is completed !!!'));
console.log("File exist in tmp directory");
res.status(200).send('Successfully executed !!!')
}
else
{
console.log("File is not on the tmp Google directory");
res.status(500).send('File is not loaded in tmp Google directory')
}
});
}
else res.status(500).send('Error: no filename on the body (filename)');
}
else res.status(500).send('Error: no configuration elements on the body (config)');
}
So, I received the following message: "File is not loaded in tmp Google directory" because after fs.stat() method, stats.isDirectory() is false. Before I use the fs.stats() method to check if the file is here, I have just writen files with the same filenames but without content.
So, I conclude that my upload work but without DONWLOAD files is really hard to copy it in the Google Cloud Storage.
Thanks for your time and I hope I will find a solution.
The problem is that your not waiting for the download to be completed before your code which performs the upload starts running. While you do have a catch() statement, that is not sufficient.
Think of the first part (the download) as a separate block of code. You have told Javascript to go off an do that block asynchronously. As soon as your script has done that, it immediately goes on to do the the rest of your script. It does not wait for the 'block' to complete. As a result, your code to do the upload is running before the download has been completed.
There are two things you can do. The first would be to move all the code which does the uploading into a 'then' block following the get() call (BTW, you could simplify things by using fastGet()). e.g.
client.connect(body.config)
.then(() => {
console.log("Client is connected !");
return client.fastGet(remotePath, localPath);
})
.then(() => {
// do the upload
})
.catch(err => {
res.status(500);
res.send(err.message);
})
.finally(() => client.end());
The other alternative would be to use async/await, which will make your code look a little more 'synchronous'. Something along the lines of (untested)
async function doTransfer(remotePath, localPath) {
try {
let client - new Client();
await client.connect(config);
await client.fastGet(remotePath, localPath);
await client.end();
uploadFile(localPath);
} catch(err) {
....
}
}
here is a github project that answers a similar issue to yours.
here they deploy a Cloud Function to download the file from the FTP and upload them directly to the bucket, skipping the step of having the temporal file.
The code works, the deployment way in this github is not updated so I'll put the deploy steps as I suggest and i verified they work:
Activate Cloud Shell and run:
Clone the repository from github: git clone https://github.com/RealKinetic/ftp-bucket.git
Change to the directory: cd ftp-bucket
Adapt your code as needed
Create a GCS bucket, if you dont have one already you can create one by gsutil mb -p [PROJECT_ID] gs://[BUCKET_NAME]
Deploy: gcloud functions deploy importFTP --stage-bucket [BUCKET_NAME] --trigger-http --runtime nodejs8
In my personal experience this is more efficient than having it in two functions unless you need to do some file editing within the same cloud function
I'm looking to let a user download a file directly from an sftp server, but in the browser. For example, user wants to find a audio file name called 'noise', they enter the parameters and a button shows to download. I am using express as my web application framework and as well as ejs.
I've found methods to download via SFTP like the code below but this is saved through my applications folder rather than the user's disk.
sftp.connect(config).then(() => {
sftp.get('file.wav').then((data) => {
var outFile = fs.createWriteStream('file.wav')
data.on('data',function(response) {
outFile.write(response);
});
data.on('close', function() {
outFile.close();
});
});
})
How can you download directly from sftp to the user's disk by giving them the option through a button?
You pipe the stream to your res. For example:
router.get('/download', (req, res) => {
sftp.connect(config).then(() => {
sftp.get('file.wav').then((data) => {
res.pipe(data)
})
})
})
I have an application that makes a PDF and sends it to the client using node js. The application works perfectly locally but when I host it in a Ubuntu server in Digital Ocean, the endpoint that generates the PDF isn't working
This is the code that sends the PDF to the client :
pdf.create(html, options).toStream((err, stream)=> {
if (err) {
res.json({
message: 'Sorry, we were unable to generate pdf',
});
}
stream.pipe(res)
});
in the client side this how i communicate with the endpoint
genratePdf({ commit }, data) {
axios.post('http://localhost:1337/getpdf', data,{ responseType:'arraybuffer' }).then((response) => {
let blob = new Blob([response.data],{type:'application/pdf'})
var link=document.createElement('a');
link.href=URL.createObjectURL(blob);
link.download="Report_"+new Date()+".pdf";
link.click();
}, (err) => {
console.log(err)
})
When i run it locally it works perfectly:
but when I host in a Ubuntu Digital Ocean droplet the other endpoint work but the generated PDF one is not working and it shows me that error
I think it's a timeout problem the app does not wait for the stream to finish to pipe it in res.
There is actually an error occuring in the generation of your pdf but because you are not returning when handling error, it still executes the stream.pipe statement.
Change your code to this :
pdf.create(html, options).toStream((err, stream)=> {
if (err) {
console.error(err);
return res.json({
message: 'Sorry, we were unable to generate pdf',
});
}
return stream.pipe(res)
});
Notice I added a console.error(err);, it might help you to debug further. But I think the library you are using uses PhantomJS, might be an error with it as PhantomJS must be compiled for the arch.
Try doing rm -Rf ./node_modules && npm i
Im struggling to find material on this
I have a rest API, written in node.js, that uses mongoDB.
I want users to be able to upload images (profile pictures) and have them saved on the server (in mongoDB).
A few questions, Ive seen it is recommended to use GridFS, is this the best solution?
How do i send these files? Ive seen res.sendFile, but again is this the best solution?
If anyone has any material they can link me I would be appreciative
thanks
You won't be able to get the file object on the server directly. To get file object on the server, use connect-multiparty middleware. This will allow you to access the file on the server.
var multipart = require('connect-multiparty');
var multipartmiddleware = multipart();
var mv = require('mv');
var path = require('path');
app.post("/URL",multipartmiddleware,function(req,res){
var uploadedImage = req.files.file;
for (var i = 0; i < uploadedImage.length; i++) {
var tempPath = uploadedImage[i].path;
var targetPath = path.join(__dirname ,"../../../img/Ads/" + i + uploadedImage[i].name);
mv(tempPath, targetPath, function (err) {
if (err) { throw err; }
});
}
})
Use file system
Generally in any database you store the image location in the data as a string that tells the application where the image is stored on the file system.
Unless your database needs to be portable as a single unit, the storing of images inside of the database as binary objects generally adds unnecessary size and complexity to your database.
-Michael Stearne
In MongoDB, use GridFS for storing files larger than 16 MB.
- Mongo Documentation
Therefore unless your images will be over 16 MB, you should either store the file on a CDN (preferable) or the server's own file system and save its URL to user's document on the database.
Local file system implementation
This method uses Busboy to parse the photo upload.
in relevant html file:
<input type="file" title="Choose a file to upload" accept="image/*" autofocus="1">
Handler function for your photo upload route in server file (you will need to fill in the variables that apply to you and require the necessary modules):
function photoUploadHandlerFunction (req, res) {
var busboy = new Busboy({ headers: req.headers })
busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
const saveToDir = path.join(__dirname, uploadsPath, user.id)
const saveToFile = path.join(saveToDir, filename)
const pathToFile = path.join(uploadsPath, user.id, filename)
const writeStream = fs.createWriteStream(saveToFile)
createDirIfNotExist(saveToDir)
.then(pipeUploadToDisk(file, writeStream))
.then(findUserAndUpdateProfilePic(user, pathToFile))
.catch((err) => {
res.writeHead(500)
res.end(`Server broke its promise ${err}`)
})
})
busboy.on('finish', function () {
res.writeHead(200, { 'Connection': 'close' })
res.end("That's all folks!")
})
return req.pipe(busboy)
}
Where the promise functions createDirIfNotExist and pipeUploadToDisk could look like this:
function createDirIfNotExist (directory, callback) {
return new Promise(function (resolve, reject) {
fs.stat(directory, function (err, stats) {
// Check if error defined and the error code is "not exists"
if (err) {
if (err.code === 'ENOENT') {
fs.mkdir(directory, (err) => {
if (err) reject(err)
resolve('made folder')
})
} else {
// just in case there was a different error:
reject(err)
}
} else {
resolve('folder already existed')
}
})
})
}
function pipeUploadToDisk (file, writeStream) {
return new Promise((resolve, reject) => {
const fileWriteStream = file.pipe(writeStream)
fileWriteStream.on('finish', function () {
resolve('file written to file system')
})
fileWriteStream.on('error', function () {
reject('write to file system failed')
})
})
}
To answer your question 'How do I send these files?', I would need to know where to (MongoDB, to the client...). If you mean to the client, you could serve the static folder where they are saved.
If you still want to learn about implementing GridFs tutorialspoint have a good tutorial
More material
Good tutorial on handling form uploads
Tutorial using the node-formidable module
If you're using the mongoose odm you can use the mongoose-crate module and send the file wherever for storage.
Also, this is a good case for shared object storage like AWS S3 or Azure blob storage. If you are running a distributed setup in something like AWS, you usually don't want to store photos on the local server.
Store the url or key name in the database that points to the S3 object. This also integrates with CloudFront CDN pretty easily.
As suggested before. MultiPart for the actual upload.
I'm trying to generate a PDF from an HTML file from the frontend and the users may download the PDF (never be stored on the server).
For this I am using the module: html5-to-pdf
My code is like this:
var pdf = require('html5-to-pdf');
var fs = require('fs');
router.post('/pdf', function(req, res) {
var html = req.body.html;
if (!html) {
return res.sendStatus(400);
}
pdf().from.string(html).to.buffer({
renderDelay: 1000
}, function(err, data) {
if (err) {
return res.sendStatus(500);
}
var file = fs.createWriteStream('myDocument.pdf');
file.write(data, function(err) {
if (err) {
res.sendStatus(500);
}
res.download('myDocument');
});
});
});
Whenever I download a file size of 0Bytes and also creates the file on the server
Someone could help me?
Maybe it send file before write finish
file.on('end',function(){
res.download('myDocument');
})
The problem is that html5-to-pdf used phantom to generate the PDF, so it phantom deploys a little server at "localhost" and the port 0, the fact is that OpenShift not recognize "localhost"[1] and if instead of using "localhost" variable is used: process.env.OPENSHIFT_NODEJS_IP the application works correctly.
[1] https://github.com/amir20/phantomjs-node/tree/v1#use-it-in-restricted-enviroments