I'm trying to generate a signed upload url to GCS, using golang, that can be used from browser.
In nodejs, I can create resumableUploadURL by calling this code:
const { Storage } = require('#google-cloud/storage')
const bucketName = "mybucketname"
const path = "path/to/file.png";
const contentType = "image/png";
const bucket = storage.bucket(bucketName);
const file = bucket.file(path);
const [uploadURL] = await file.createResumableUpload({
origin: params.origin,
'Content-Type': contentType,
metadata: { 'Content-Type': contentType } // ?important?
});
And that will give uploadURL
https://storage.googleapis.com/upload/storage/v1/b/mybucketname/o?name=path%2Fto%2Ffile.png&uploadType=resumable&upload_id=ADPycdvi1mH34PtIUEyVO9SzezIAkdlw3suis2WpB1KrK44FJ9_wGCKRHDyRBk9fZP1Aadqqy-3u5BpnHwIRZyJyl1bDFGmuTLPn
how can I do that in golang?
I've tried:
import ("cloud.google.com/go/storage")
opts := storage.SignedURLOptions{
ContentType: u.ContentType,
Method: "PUT",
Expires: time.Now().Add(15 * time.Minute),
Headers: []string{"host: %s", u.Origin},
Scheme: storage.SigningSchemeV4, // or V2 doesnt matter
}
uploadURL, err := bucket.SignedURL(path, &opts);
And it gave me:
https://storage.googleapis.com/mybucketname/path/to/file.png?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx.iam.gserviceaccount.com%2F20230206%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230206T144142Z&X-Goog-Expires=899&X-Goog-Signature=11715d0552d8ed1ee3d303bce95d6fee1c04a2dd12e3f300b5cce5298b4b1045cf89217cd929d5de4cc5d8ac3bd6d906c059ff85d9abad35c9d85d65a1e4f0162d0d0d719738323cb935605ff34c6d0bccb6026290c8b61707dc015316748d8542e67d0d5c839177cd27c1e0822b252f340b62fe65.....6c343abd2a40b21561026812a81fb22ed57e714b689169d823fa8954f80184a5d04086ab032edd6da288523985568293501aa57efddc36b9f0c58cb39afe1d4da9f863f1b6b87057503683df6b3ebc0bc7f83bcd580a5abb9236acf167972952ede0e4ad3d1b22e9293d9f614433105381f2f234b3b0f1ea0559d&X-Goog-SignedHeaders=content-type%3Bhost%3Bhost
so, I guess my problem is, when I send my file to the uploadURL returned by golang from browser, it doesn't include Origin in its request headers, and therefore experiencing CORS issues. And also, I can't just manually add it, because most browser refused to set that header because it was considered unsafe.
I think, the browser doesn't send origin headers because the header response from the preflight OPTIONS to the golang URL, doesn't have access-control-allow-origin, access-control-allow-methods, etc etc.
so yeah, how do I generate a signed upload URL that can be used from browser in golang?
Related
I tried
const myData = uri; // this looks like file:///data/expo/...
const myDataResponse = await API.SendFile({
myData: myData
});
But file is not being recieved in the backend. Is there any other way to do this?
IN the backend I get
console.log(req.body)
// I get a string same as uri of file
console.log(req.file)
// undefined
Backend is setup perfectly. I am getting successful response when using postman.
Any help is appreciated. I am a begginer.
You should use FormData.
const data = new FormData();
// first argument is the key name received on you API
// second argument is your file path
data.append('file', filePath)
// then your request should look like this :
await fetch(url, {
method: 'post',
headers: {
'Content-Type': 'multipart/form-data',
},
body: data,
})
I am trying to deliver hls media from a lambda function using nodejs to an AWS MediaPackage input endpoint.
I am doing the following and seems to be pushing the media file chunks (ts files):
const auth = 'Basic ' + Buffer.from(endpoints[0].Username + ':' + endpoints[0].Password).toString('base64');
const options = {
hostname: endpoints[0].Url.hostname,
path: endpoints[0].Url.path,
headers: {
'Content-Type': 'application/json',
'Authorization': auth,
'Content-Length': Buffer.byteLength(data.Body)
},
body: data.Body // body of ts file
};
console.log(options);
const res = await httpsPut(options); //Promise using https put to send the body in options
However, I don't see any logs in CloudWatch to the MediaPackage Channel
Is this the right way to send inject media to MediaPackage? I couldn't find any documentation
Thanks,
Well, I finally got it working following the documentation and doing a bit of try and error. In this MediaPackage documentation they mention that it only supports webdav auth so I ended up using a webdav module to handle the puts of files.
Assuming you are passing valid inputs, the working solution ended up looking like this:
/**
* Helper PUT with Webdav and digest authentication to AWS MediaPackage
* #param mediaPackageEndpoints
* #param fileName
* #param data
*/
const putToMediaPackage = async function(mediaPackageEndpoints, fileName, data) {
const url = new URL(mediaPackageEndpoints[0].Url);
const client = createClient(
url.toString().replace('/channel', ''),
{
username: mediaPackageEndpoints[0].Username,
password: mediaPackageEndpoints[0].Password,
digest: true
}
);
await client.putFileContents(fileName, data);
return client;
}
I am trying to use this tutorial to upload files directly from the browser in a bucket using presigned URL.
I tryed this:
export const myFunction = async (imageType: string) => {
const s3 = new AWS.S3();
const imageName = "somename";
const s3Params = {
Bucket: bucketname,
Key: imageName,
ContentType: 'image/' + imageType,
Expires: 300,
};
const uploadURL = await s3.getSignedUrlPromise('putObject', s3Params);
return JSON.stringify({
uploadURL,
imageName,
});
};
And I do get back an URL in postman, I accessed it and then it redirected me to a new request, I set it to put and try to send the request without any file and with a file with the same name from the URL (the name of the image appear in the URL) in the form-data, sent it, but both my tries resulted in this error :
SignatureDoesNotMatch. The request signature we calculated does not match the signature you provided. Check your key and signing method.
I read something on stack, found oput it could be due to the file name if it has exotic characters, but it isn't my case.
How can I solve this problem?
I have this payload format in my hapi route that accepts only multipart/form-data data and the output is set as a stream:
payload: {
maxBytes: 1024,
output: 'stream',
parse: true,
allow: 'multipart/form-data',
}
I want to test that route and my payload is this:
const FD = new FormData();
FD.append('field1', 'value');
FD.append('field2', 'value');
The hapi inject method looks like this:
const res = await server.inject({
method,
url,
headers: {
...
'Content-Type': 'multipart/form-data; boundary=--SEPARATION--',
},
payload: FD,
});
I am getting
Invalid multipart payload format
I tried to set a stream using a Steam object too but it doesn't work. Also I tried to send a File object.
At this point I just want to send something to the route that will not result in an error. It's not necessary to be a FormData. Anything that will be accepted by the route is fine as long as I can add some custom fields to test it further.
Thanks in advance.
As far as I know, FormData doesn’t exist in node.js, so I assume you’re using the form-data module?
In that case, you need to ask form-data for the buffer and headers separately, like this:
const FormData = require('form-data');
const FD = new FormData();
FD.append('field1', 'value');
FD.append('field2', 'value');
const response = await server.inject({
method,
url,
headers: {
<your other headers>
...FD.getHeaders(),
},
payload: FD.getBuffer(),
});
If you want to test sending files, you also need to provide the filename property to append(…) for the binary data to be decoded correctly by the server:
const fileContent = <any Buffer>;
FD.append('form_file_property', fileContent, {filename: 'a testfile'});
I'm trying to give a temporary access to a service account that has only read permission to upload an mp3 to a Google Cloud Storage bucket.
For that, I'm generating a signedurl but i can't get it to work.
What I'm doing to generate the URL and upload the file :
const config = {
action: 'write',
expires: '03-12-2019',
contentType: 'audio/mpeg'
}
const storage = new Storage({
projectId: projectId,
keyFilename: './service-account-key.json'
})
let bucketSubDirectory = storage.bucket(bucketName).file('subdirectory/in/bucket/' + songName)
bucketSubDirectory.getSignedUrl(config, (err, url) => {
if (err) {
console.error(err)
return
}
console.log(url)
// The file is now available to be written to.
let writeStream = request.put(url, {
headers: {
'Content-Type': 'audio/mpeg'
}
})
writeStream.end('New data');
writeStream.on('complete', function (resp) {
console.log(resp.statusCode + " " + resp.statusMessage)
})
})
When I execute the script I get a 403 forbidden response and when I try to access the generated URL from the browser I get the following :
<Error>
<Code>MalformedSecurityHeader</Code>
<Message>Your request has a malformed header.</Message><ParameterName>signature</ParameterName>
<Details>Signature was not base64 encoded</Details>
</Error>
Any idea how can I solve this problem?
According to the node.js client library documentation, the method getSignedUrl has a parameter config which as a parameter contentType:
contentType
Optional
string
If you provide this value, the client must provide this HTTP header set to the same value.
So you might need to modify your PUT request to include this header with the value contentType: audio/mpeg.
Also, if you enter the url in the browser you are not doing a PUT request but a GET one.
EDIT:
Also check that your service account which creates the signedurl has the right permissions granted. For the code your are running you need at least the roles/storage.objectAdmin role. To grant it:
gcloud projects add-iam-policy-binding yourProjectID --member yourServiceAccount --role "roles/storage.objectAdmin"
Once this is done, users will be able to write to the file with just a PUT request to the url.
const request = require('request');
let url = 'yourSignedURL'
let writeStream = request.put(url, {
headers: {
'Content-Type': 'audio/mpeg'
}
})
writeStream.end('New data');
writeStream.on('complete', function (resp) {
console.log(resp.statusCode + " " + resp.statusMessage)
})