I'm want to upload mp4 file to my azure storage without md5 content.
var uploadOptions = {};
uploadOptions.storeBlobContentMD5 = false;
blobSvc.createBlockBlobFromLocalFile('kovach', fileName, files.file.path,uploadOptions, function(error, result, response) {
if(!error){
// file uploaded
console.log(result);
res.end(fileName);
}
});
but in the response I got value in content md5.
When you upload a local file larger than 32MB, the blobSvc.createBlockBlobFromLocalFile method actually invokes several REST calls:
Create a new block blob.
Put blocks.
Commit the block list.
From the Put Block List REST, the Content-MD5 header is returned so that the client can check for message content integrity. This header refers to the content of the request, meaning, in this case, the list of blocks, and not the content of the blob itself.
I think you try blobsvc.getBlobProperties, you won't see content-md5 set on this blob.
Related
this.state.videoBlob is a blob object. I used URL.createObjectURL to generate a blob URL, and passed it to fs.createReadStream, like below:
fs.createReadStream(URL.createObjectURL(this.state.videoBlob))
This blob url looks like:
blobURL: blob:http://localhost:3000/dabe5cdd-00cc-408a-9f3d-b0ba5f2b10b3
But I got an error saying:
TypeError: fs.createReadStream is not a function
The problem won't exist if I passed some online video URL. So how can I read blob from fs.createReadStream? Thanks!
When I look at the code behind fs.createReadStream(), it calls new ReadStream() and passes the path/url to that. On this line of code, it appears that the only type of url that is supported is a file URL. The doc is silent on that topic which is why I went and looked at the code. So, it does not appear to me that fs.createReadStream() supports that type of pseudo-URL.
Since you just want a readstream from that URL and you have the actual URL of the remote resource, I would suggest you just use either http.get() or request() or something similar as those will all contact the remote host and return to you a readStream. Since your objective was to get a readStream, this is one way to achieve that.
http.get('http://localhost:3000/dabe5cdd-00cc-408a-9f3d-b0ba5f2b10b3', (res) => {
// res is a readstream here
}).on('error', (err) => {
// error on the request here
});
FYI, you may find this answer on Blob URLs to be useful. I don't see any evidence that fs.createReadStream() supports blob URLs. In the browser, they are something created only by the internals of the browser and are only useful within that specific web page context (they refer indirectly to some internal storage) and can't be passed outside the web page or even preserved from one web page to the next. If you wanted your server to have access to the actual data from a blob URL created in the browser, you'd have to upload the actual data to your server. Your server can't access a blob URL created in the browser.
I'm trying to upload base64 file/image into Google cloud storage using the signed URL. My server side code (NodeJS) is something like this:
let {Storage} = require('#google-cloud/storage');
storage = new Storage({
projectId,
keyFilename: gcloudServiceAccountFilePath,
});
function generateSignedUrl(){
const options = {
version: 'v4',
action: 'write',
expires: Date.now() + 15 * 60 * 1000, // 15 minutes
//contentType: 'application/octet-stream'
};
}
const [url] = await storage.bucket(gcloudBucket)
.file(`${fileRelativePath}`).getSignedUrl(options);
return url;
}
Now when I try with POSTMAN with below configuration,
Request: PUT
URL: https://storage.googleapis.com/my-signed-url.. //generated from above code
Headers:
x-goog-acl: 'public-read'
Content-Type: 'image/jpeg'
Body:
raw : 'base64-file-conent'
My uploaded file in GCS stays as base64 and file size is also different as you can see in the storage.
1st image is directly uploaded into GCS with drag & drop.
2nd image is uploaded with POSTMAN
Not sure if I'm missing something while generating signed-url or any headers while uploading file through postman.
Thanks :)
The reason for the difference in the Object sizes uploaded to the Google Cloud Storage is actually the difference in the Metadata of the Object. When you upload the image Object with POSTMAN by using REST APIs, the API header is added as part of the image's metadata. This Google Documentation clearly states that “the Cloud Storage stores these headers as part of the object's metadata”.
The first line of the Object metadata Introduction also confirms that Objects stored in Cloud Storage have metadata associated with them. Hence, the API headers are added as the Metadata of your Image object and consequently increase the Size of the Object.
Image Objects uploaded via the Console do not have Object metadata, except they are explicitly set.
So I want to pipe a file straight to the client; how I am currently doing it is create a file to disk, then sending that file straight to the client.
router.get("/download/:name", async (req, res) => {
const s3 = new aws.S3();
const dir = "uploads/" + req.params.name + ".apkg"
let file = fs.createWriteStream(dir);
await s3.getObject({
Bucket: <bucket-name>,
Key: req.params.name + ".apkg"
}).createReadStream().pipe(file);
await res.download(dir);
});
I just looked up that res.download() only serves locally. Is there a way you can do it directly from AWS S3 to Client download? i.e. pipe files straight to user. Thanks in advance
As described in this SO thread:
You can simply pipe the read stream into the response instead of the piping it to the file, just make sure to supply the correct Content-Type and to set it as an attachment, so the browser will know how to handle the response properly.
res.attachment(req.params.name);
await s3.getObject({
Bucket: <bucket-name>,
Key: req.params.name + ".apkg"
}).createReadStream().pipe(res);
On more pattern for this is to create a signed url directly to the S3 object and then let the client download straight from S3, instead of streaming it from your node webserver. This will reduce the workload from your web server.
You will need to use the getSignedUrl method from the AWS S3 SDK for JS.
Then, Once you have the URL, just return it to your client to download the file by themselves.
You should take into account that once you give the client a signed URL that has download permissions for, say, 5 minutes, they will only be able to download that file during those next 5 minutes. And you should also take into account that they will be able to pass that URL to anyone else for download during those 5 minutes, so it is dependant on how secure you need this to be.
S3 can be used to content so I would do the following.
Add CORS headers on your node response. This will enable browser to download from another origin i.e. S3.
Enable S3 web server on your bucket.
Script to download redirect from S3 - this you could achieve in JS.
Use signed URL as suggested in the other post if you need to protect S3 content.
How can I get uploaded file content using multiparty in node.js? I don't need temp file, I need to redirect all stream into Google Cloud Storage in order to save the file content, but I can't find the way to get this content with events.
Found the answer. We need to use part stream a subscribe to standard streams events, like, data and end in order to receive file's data.
part.on("data", chunk => {
writeStream.write(chunk);
});
part.on("end", chunk => {
writeStream.end(chunk);
});
writeStream - is a another stream where you want to put your data. In my case that was Google Cloud Storage file PUT request via signedUrl.
part - is a part object of a form part event
I'm working on a project using Google Cloud Storage to allow users to upload media files into a predefined bucket using Node.js. I've been testing with small .jpg files. I also used gsutil to set bucket permissions to public.
At first, all files generated links that downloaded the file. Upon investigation of the docs, I learned that I could explicitly set the Content-Type of each file after upload using the gsutil CLI. When I used this procedure to set the filetype to 'image/jpeg', the link behavior changed to display the image in the browser. But this only worked if the link had not been previously clicked prior to updating the metadata with gsutil. I thought that this might be due to browser caching, but the behavior was duplicated in an incognito browser.
Using gsutil to set the mime type would be impractical at any rate, so I modified the code in my node server POST function to set the metadata at upload time using an npm module called mime. Here is the code:
app.post('/api/assets', multer.single('qqfile'), function (req, res, next) {
console.log(req.file);
if (!req.file) {
return ('400 - No file uploaded.');
}
// Create a new blob in the bucket and upload the file data.
var blob = bucket.file(req.file.originalname);
var blobStream = blob.createWriteStream();
var metadata = {
contentType: mime.lookup(req.file.originalname)
};
blobStream.on('error', function (err) {
return next(err);
});
blobStream.on('finish', function () {
blob.setMetadata(metadata, function(err, response){
console.log(response);
// The public URL can be used to directly access the file via HTTP.
var publicUrl = format(
'https://storage.googleapis.com/%s/%s',
bucket.name, blob.name);
res.status(200).send(
{
'success': true,
'publicUrl': publicUrl,
'mediaLink': response.mediaLink
});
});
});
blobStream.end(req.file.buffer);
});
This seems to work, from the standpoint that it does actually set the Content-Type on upload, and that is correctly reflected in the response object as well as the Cloud Storage console. The issue is that some of the links returned as publicUrl cause a file download, and others cause a browser load of the image. Ideally I would like to have both options available, but I am unable to see any difference in the stored files or their metadata.
What am I missing here?
Google Cloud Storage makes no assumptions about the content-type of uploaded objects. If you don't specify, GCS will simply assign a type of "application/octet-stream".
The command-line tool gsutil, however, is smarter, and will attach the right Content-Type to files being uploaded in most cases, JPEGs included.
Now, there are two reasons why your browser is likely to download images rather than display them. First, if the Content-Type is set to "application/octet-stream", most browsers will download the results as a file rather than display them. This was likely happening in your case.
The second reason is if the server responds with a 'Content-Disposition: attachment' header. This doesn't generally happen when you fetch GCS objects from the host "storage.googleapis.com" as you are doing above, but it can if you, for instance, explicitly specified a contentDisposition for the object that you've uploaded.
For this reason I suspect that some of your objects don't have an "image/jpeg" content type. You could go through and set them all with gsutil like so: gsutil -m setmeta 'Content-Type:image/jpeg' gs://myBucketName/**