I have a bucket on Google Cloud Platform where part of my application adds small text files with unique names (no extension).
A second app needs to retrieve individual text files (only one at a time) for insertion into a template.
I cannot find the correct api call for this.
Configuration is as required:
var gcloud = require('gcloud');
var gcs = gcloud.storage({
projectId: settings.bucketName,
keyFilename: settings.bucketKeyfile
});
var textBucket = gcs.bucket(settings.bucketTitle);
Saving to the bucket works well:
textBucket.upload(fileLocation, function(err, file) {
if(err) {
console.log("File not uploaded: " + err);
} else {
// console.log("File uploaded: " + file);
}
});
The following seems logical but returns only metadata and not the actual file for use in the callback;
textBucket.get(fileName, function(err, file) {
if(err) {
console.log("File not retrieved: " + err);
} else {
callback(file);
}
});
Probably no surprise this doesn't work since it's not actually in the official documentation but then again, neither is a simple asnyc function which returns a document you ask for.
The method get on a Bucket object is documented here: https://googlecloudplatform.github.io/gcloud-node/#/docs/v0.29.0/storage/bucket?method=get
If you want to simply download the file into memory, try the method download on a File object: https://googlecloudplatform.github.io/gcloud-node/#/docs/v0.29.0/storage/file?method=download. You can also use createReadStream if using a stream workflow.
If you have ideas for improving the docs, it would be great if you opened an issue on https://github.com/googlecloudplatform/gcloud-node so we can make it easier for the next person.
Related
I am trying to copy a file from a AWS S3 bucket to another bucket using Node. The problem is if the file name doesn't has the white space for example: "abc.csv", It is working fine.
But in case the file to which I want to copy has the white space in the file name for example: "abc xyz.csv". It is throwing the below error.
"The specified key does not exist."
"NoSuchKey: The specified key does not exist.
at Request.extractError (d:\Projects\Other\testproject\s3filetoarchieve\node_modules\aws-sdk\lib\services\s3.js:577:35)
Below is the code provided.
return Promise.each( files, file => {
var params = {
Bucket: process.env.CR_S3_BUCKET_NAME,
CopySource: `/${ process.env.CR_S3_BUCKET_NAME }/${ prefix }${ file.name}`,
Key: `${ archieveFolder }${ file.name }`
};
console.log(params);
return new Promise(( resolve, reject) => {
s3bucket.copyObject(params, function(err, data) {
if (err){
console.log(err, err.stack);
debugger
} else {
console.log(data);
debugger
}
});
});
}).then( result => {
debugger
});
Early help would be highly appreciable. Thank you.
I think the problem is exactly that space in the filename.
S3 keys must be url encoded, as they need to be accesible in URL form.
There are some packages that helps you with url formatting like speakingUrl
or you can try writting some on your own, maybe just simply replacing spaces (\s) with dashes (_ or -) if you want to keep it friendly.
If you don't mind about that, you can simply encodeURIComponent(file.name)
Hope it helps!
I have an Excel file (.xlsx), which when I have it already in my OneDrive, I can use a REST command like this, to modify it:
/v1.0/me/drive/root:/SpreadSheetName.xlsx:/workbook/worksheets/content_stats/tables('RawStats')/Rows
When I modify my code to first upload the file to OneDrive (rather than using the file that is already there), and I use the REST API, I get the error:
Open navigation properties are not supported on OpenTypes. Property name: 'tables'.
I have searched the web for this message, and cannot find anything related to what I am doing. The REST call for modifying the file which was just uploaded is nearly identical, although I do reference the file by ID instead, as that is what is returned by the upload API. This is the URL I use to modify the file which was just uploaded.
/v1.0/me/drive/items:/<RealExcelSpreadsheetID>:/workbook/worksheets/content_stats/tables('RawStats')/Rows
Both are doing a POST. Exact same file, only difference is it is being uploaded first, rather than already being in OneDrive. The file was definitely uploaded correctly, as when I go through the OneDrive web interface I do find it and can view it online. This is a business account.
It was uploaded as MIME type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.
It uses these scopes delegated via OAuth2:
User.Read
User.ReadWrite
Files.Read
Files.ReadWrite
Files.ReadWrite.All
Sites.ReadWrite.All
Using Node.js and JavaScript, although that should't matter.
Here is the code used to upload the file:
function copyTemplateInOneDrive(res, queryParam, officeAccessToken, callback) {
var fs = require('fs');
var excelExt = ".xlsx";
var excelSpreadsheetFilenameStart = "stats";
var uploadUrl = "https://graph.microsoft.com/v1.0/me/drive/root:/" +
excelSpreadsheetFilename + dateNowFull() + excelExt + ":/content";
var xlsxMimeType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
fs.readFile(excelSpreadsheetTemplateFilename, function read(error, fileContent) {
var request = require('request');
var options = {
url: uploadUrl,
headers: {
'Accept': 'application/json',
'Authorization': 'Bearer ' + officeAccessToken,
'Content-Type': xlsxMimeType,
},
body: fileContent
};
request.put(options, function (error, response, body) {
var result = JSON.parse(body);
if ('id' in result) {
console.log("Successfully uploaded template file to OneDrive");
res.write("Successfully uploaded template file to OneDrive");
var excelSpreadsheetID = result.id;
excelSpreadsheetUrl = result.webUrl;
} else {
console.log("ERROR: unable to upload template file to OneDrive " + body);
res.write("Error: unable to upload template file to OneDrive" + body);
return;
}
callback(null, res, queryParam, officeAccessToken, excelSpreadsheetID);
});
});
}
It uses the async module from node.js (which makes use of callback). It also saves the ID returned, and later passes it into the call to the Microsoft Graph.
Your issue here is the URL you're calling with the id is using syntax expecting the file path (i.e. folder\filename.ext) rather than the id. This is why switching to the file name started working for you.
There are two ways to address a file stored in OneDrive:
drive/items/{item-id}
/drive/root:/path/to/file (note the :)
You correctly switched your URI from drive/root to drive/items but by leaving the : in place you are telling OneDrive to address the file by it's path rather than it's id. In other words, it's looking for a file named "{some-id}".
For addressing a file by it's path, your URL is correct:
/drive/root:/{file-path}:/workbook/worksheets/content_stats/tables('RawStats')/Rows
For addressing a file by it's id however, you need to drop the ::
/drive/items/{file-id}/workbook/worksheets/content_stats/tables('RawStats')/Rows
You can read about how files are addressed in the documentation for DriveItem.
I've been taking a crack at uploading files onto S3 via NodeJS, but with a specific path where they have to be stored.
return s3fsImpl.writeFile(file_name.originalFilename,stream).then(function() {
fs.unlink(file_name.path, function(err) {
if (err) {
console.error(err);
} else { /** sucessess **/ }
I'm not sure how do I give a path like /project_name/file_name.
I have been following this tutorial
In this scenario your are using a stream as the target. When you created that stream you should have specified the path at that point.
I am trying to find some solution to stream file on amazon S3 using node js server with requirements:
Don't store temp file on server or in memory. But up-to some limit not complete file, buffering can be used for uploading.
No restriction on uploaded file size.
Don't freeze server till complete file upload because in case of heavy file upload other request's waiting time will unexpectedly
increase.
I don't want to use direct file upload from browser because S3 credentials needs to share in that case. One more reason to upload file from node js server is that some authentication may also needs to apply before uploading file.
I tried to achieve this using node-multiparty. But it was not working as expecting. You can see my solution and issue at https://github.com/andrewrk/node-multiparty/issues/49. It works fine for small files but fails for file of size 15MB.
Any solution or alternative ?
You can now use streaming with the official Amazon SDK for nodejs in the section "Uploading a File to an Amazon S3 Bucket" or see their example on GitHub.
What's even more awesome, you finally can do so without knowing the file size in advance. Simply pass the stream as the Body:
var fs = require('fs');
var zlib = require('zlib');
var body = fs.createReadStream('bigfile').pipe(zlib.createGzip());
var s3obj = new AWS.S3({params: {Bucket: 'myBucket', Key: 'myKey'}});
s3obj.upload({Body: body})
.on('httpUploadProgress', function(evt) { console.log(evt); })
.send(function(err, data) { console.log(err, data) });
For your information, the v3 SDK were published with a dedicated module to handle that use case : https://www.npmjs.com/package/#aws-sdk/lib-storage
Took me a while to find it.
Give https://www.npmjs.org/package/streaming-s3 a try.
I used it for uploading several big files in parallel (>500Mb), and it worked very well.
It very configurable and also allows you to track uploading statistics.
You not need to know total size of the object, and nothing is written on disk.
If it helps anyone I was able to stream from the client to s3 successfully (without memory or disk storage):
https://gist.github.com/mattlockyer/532291b6194f6d9ca40cb82564db9d2a
The server endpoint assumes req is a stream object, I sent a File object from the client which modern browsers can send as binary data and added file info set in the headers.
const fileUploadStream = (req, res) => {
//get "body" args from header
const { id, fn } = JSON.parse(req.get('body'));
const Key = id + '/' + fn; //upload to s3 folder "id" with filename === fn
const params = {
Key,
Bucket: bucketName, //set somewhere
Body: req, //req is a stream
};
s3.upload(params, (err, data) => {
if (err) {
res.send('Error Uploading Data: ' + JSON.stringify(err) + '\n' + JSON.stringify(err.stack));
} else {
res.send(Key);
}
});
};
Yes putting the file info in the headers breaks convention but if you look at the gist it's much cleaner than anything else I found using streaming libraries or multer, busboy etc...
+1 for pragmatism and thanks to #SalehenRahman for his help.
I'm using the s3-upload-stream module in a working project here.
There is also some good examples from #raynos in his http-framework repository.
Alternatively you can look at - https://github.com/minio/minio-js. It has minimal set of abstracted API's implementing most commonly used S3 calls.
Here is an example of streaming upload.
$ npm install minio
$ cat >> put-object.js << EOF
var Minio = require('minio')
var fs = require('fs')
// find out your s3 end point here:
// http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
var s3Client = new Minio({
url: 'https://<your-s3-endpoint>',
accessKey: 'YOUR-ACCESSKEYID',
secretKey: 'YOUR-SECRETACCESSKEY'
})
var outFile = fs.createWriteStream('your_localfile.zip');
var fileStat = Fs.stat(file, function(e, stat) {
if (e) {
return console.log(e)
}
s3Client.putObject('mybucket', 'hello/remote_file.zip', 'application/octet-stream', stat.size, fileStream, function(e) {
return console.log(e) // should be null
})
})
EOF
putObject() here is a fully managed single function call for file sizes over 5MB it automatically does multipart internally. You can resume a failed upload as well and it will start from where its left off by verifying previously upload parts.
Additionally this library is also isomorphic, can be used in browsers as well.
I'm new with serverside programming with node.js. I'm sticking together a tiny webapp with it right now and having the usual startup learning to do. The following piece of code WORKS. But I would love to know if it's more or less a right way to do a simple file upload from a form and throw it into aws s3:
app.router.post('/form', { stream: true }, function () {
var req = this.req,
res = this.res,
form = new formidable.IncomingForm();
form
.parse(req, function(err, fields, files) {
console.log('Parsed file upload' + err);
if (err) {
res.end('error: Upload failed: ' + err);
} else {
var img = fs.readFileSync(files.image.path);
var data = {
Bucket: 'le-bucket',
Key: files.image.name,
Body: img
};
s3.client.putObject(data, function() {
console.log("Successfully uploaded data to myBucket/myKey");
});
res.end('success: Uploaded file(s)');
}
});
});
Note: I had to turn buffer off in union / flatiron.plugins.http.
What I would like to learn is, when to stream load a file and when to syncload it. It will be a really tiny webapp with little traffic.
If it's more or less good then please consider this as a token of working code which I also would throw into a gist. It's not that easy to find documenation and working examples of this kind of stuff. I like flatiron alot. But it's small module approach leads to lots of splattered docs and examples all over the net, speak alone of tutorials.
You should use other module than formidable because as far as I know formidable does not have s3 storage option , then you must save the files in your server before uploading it.
I would recommend you to use : multiparty
Use this example in order to upload directly to S3 without saving it locally in your server.