I am trying to deliver hls media from a lambda function using nodejs to an AWS MediaPackage input endpoint.
I am doing the following and seems to be pushing the media file chunks (ts files):
const auth = 'Basic ' + Buffer.from(endpoints[0].Username + ':' + endpoints[0].Password).toString('base64');
const options = {
hostname: endpoints[0].Url.hostname,
path: endpoints[0].Url.path,
headers: {
'Content-Type': 'application/json',
'Authorization': auth,
'Content-Length': Buffer.byteLength(data.Body)
},
body: data.Body // body of ts file
};
console.log(options);
const res = await httpsPut(options); //Promise using https put to send the body in options
However, I don't see any logs in CloudWatch to the MediaPackage Channel
Is this the right way to send inject media to MediaPackage? I couldn't find any documentation
Thanks,
Well, I finally got it working following the documentation and doing a bit of try and error. In this MediaPackage documentation they mention that it only supports webdav auth so I ended up using a webdav module to handle the puts of files.
Assuming you are passing valid inputs, the working solution ended up looking like this:
/**
* Helper PUT with Webdav and digest authentication to AWS MediaPackage
* #param mediaPackageEndpoints
* #param fileName
* #param data
*/
const putToMediaPackage = async function(mediaPackageEndpoints, fileName, data) {
const url = new URL(mediaPackageEndpoints[0].Url);
const client = createClient(
url.toString().replace('/channel', ''),
{
username: mediaPackageEndpoints[0].Username,
password: mediaPackageEndpoints[0].Password,
digest: true
}
);
await client.putFileContents(fileName, data);
return client;
}
Related
I'm trying to generate a signed upload url to GCS, using golang, that can be used from browser.
In nodejs, I can create resumableUploadURL by calling this code:
const { Storage } = require('#google-cloud/storage')
const bucketName = "mybucketname"
const path = "path/to/file.png";
const contentType = "image/png";
const bucket = storage.bucket(bucketName);
const file = bucket.file(path);
const [uploadURL] = await file.createResumableUpload({
origin: params.origin,
'Content-Type': contentType,
metadata: { 'Content-Type': contentType } // ?important?
});
And that will give uploadURL
https://storage.googleapis.com/upload/storage/v1/b/mybucketname/o?name=path%2Fto%2Ffile.png&uploadType=resumable&upload_id=ADPycdvi1mH34PtIUEyVO9SzezIAkdlw3suis2WpB1KrK44FJ9_wGCKRHDyRBk9fZP1Aadqqy-3u5BpnHwIRZyJyl1bDFGmuTLPn
how can I do that in golang?
I've tried:
import ("cloud.google.com/go/storage")
opts := storage.SignedURLOptions{
ContentType: u.ContentType,
Method: "PUT",
Expires: time.Now().Add(15 * time.Minute),
Headers: []string{"host: %s", u.Origin},
Scheme: storage.SigningSchemeV4, // or V2 doesnt matter
}
uploadURL, err := bucket.SignedURL(path, &opts);
And it gave me:
https://storage.googleapis.com/mybucketname/path/to/file.png?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx.iam.gserviceaccount.com%2F20230206%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230206T144142Z&X-Goog-Expires=899&X-Goog-Signature=11715d0552d8ed1ee3d303bce95d6fee1c04a2dd12e3f300b5cce5298b4b1045cf89217cd929d5de4cc5d8ac3bd6d906c059ff85d9abad35c9d85d65a1e4f0162d0d0d719738323cb935605ff34c6d0bccb6026290c8b61707dc015316748d8542e67d0d5c839177cd27c1e0822b252f340b62fe65.....6c343abd2a40b21561026812a81fb22ed57e714b689169d823fa8954f80184a5d04086ab032edd6da288523985568293501aa57efddc36b9f0c58cb39afe1d4da9f863f1b6b87057503683df6b3ebc0bc7f83bcd580a5abb9236acf167972952ede0e4ad3d1b22e9293d9f614433105381f2f234b3b0f1ea0559d&X-Goog-SignedHeaders=content-type%3Bhost%3Bhost
so, I guess my problem is, when I send my file to the uploadURL returned by golang from browser, it doesn't include Origin in its request headers, and therefore experiencing CORS issues. And also, I can't just manually add it, because most browser refused to set that header because it was considered unsafe.
I think, the browser doesn't send origin headers because the header response from the preflight OPTIONS to the golang URL, doesn't have access-control-allow-origin, access-control-allow-methods, etc etc.
so yeah, how do I generate a signed upload URL that can be used from browser in golang?
having issues uploading file from NodeJs server side, found 100 posts and reasearches but nothing works, would appreciate any help.
Structure of the App
Front App - React Admin framework receving file and i encode in base64 the content of the image to send to API
Backend - NestJS App - receving base64 image in API
From my backend API need to send file to an external backend (Python API) to upload - here is the problem
Please see below my code, something wrong with the file from JS
i have tried several methods and all of them ends in same error
1 solution
converting base64 image in buffer and send to external backend to upload the file
have tried to pass as well cleanImageBuffer but no changes
import axios from 'axios';
import FormData from 'form-data';
export async function upload(
fileBase64: string,
filename: string
): Promise<any> {
const buffer = Buffer.from(fileBase64, 'base64')
const extension = fileBase64.substring(fileBase64.indexOf('/') + 1, fileBase64.indexOf(";base64"))
const cleanBase64 = fileBase64.replace(/^data:image\/png;base64,/, '')
const cleanImageBuffer = Buffer.from(cleanBase64, 'base64')
const formData = new FormData();
// have tried to pass as well cleanImageBuffer but no changes
formData.append('file', buffer);
formData.append('fileName', filename + '.' + extension);
formData.append('namespace', 'test');
return await axios
.post('external_api_url', JSON.stringify(formData), {
headers: {
Authorization: `Bearer token`,
ContentType: 'multipart/form-data'
}
})
.then((response) => {
console.log('response = ' + JSON.stringify(response))
})
result 1 solution
{
"status": "error",
"error": {
"code": "bad_request",
"message": "file Expected UploadFile, received: <class 'str'>"
}
}
2 solution
from base64 image received saving on my disk
after creating a stream and sending the image
export async function upload (
fileBase64: string,
filename: string
): Promise<any> {
const extension = fileBase64.substring(fileBase64.indexOf('/') + 1, fileBase64.indexOf(";base64"))
const cleanBase64 = fileBase64.replace(/^data:image\/png;base64,/, '')
const TMP_UPLOAD_PATH = '/tmp'
if (!fs.existsSync(TMP_UPLOAD_PATH)) {
fs.mkdirSync(TMP_UPLOAD_PATH);
}
fs.writeFile(TMP_UPLOAD_PATH + '/' + filename + '.' + extension, cleanBase64, 'base64', function(err) {
console.log(err);
})
const fileStream = fs.createReadStream(TMP_UPLOAD_PATH + '/' + filename + '.' + extension)
const formData = new FormData();
formData.append('file', fileStream, filename + '.' + extension);
formData.append('fileName', filename + '.' + extension);
formData.append('namespace', 'test');
return await axios
.post('external_api_url', formData, {
headers: {
Authorization: `Bearer token`,
ContentType: 'multipart/form-data'
}
})
.then((response) => {
console.log('response = ' + JSON.stringify(response))
})
}
result 2 solution
{
"status": "error",
"error": {
"code": "bad_request",
"message": "file Expected UploadFile, received: <class 'str'>"
}
}
other solution that ended in same result
tried to use fetch from node-fetch - same result
found out that some people had an outdated version of axios and having this issues, i have installed latest axios version 1.1.3 but same result
best scenario that i need
from base64 image received
convert in buffer and send file to external Python API so to avoid saving the file on local disk
would appreciate any help
below is a python example that works but not JS (JS nothing works)
import requests
url = "http://127.0.0.1:8000/external_api"
payload={'namespace': 'test'}
files=[
('file',('lbl-pic.png',open('/local/path/lbl-pic.png','rb'),'image/png'))
]
headers = {
'Authorization': 'Bearer token'
}
response = requests.request("POST", url, headers=headers, data=payload, files=files)
print(response.text)
Just a suggestion:
Here's a line which returns mentioned error https://github.com/tiangolo/fastapi/blob/41735d2de9afbb2c01541d0f3052c718cb9f4f30/fastapi/datastructures.py#L20, you might find it useful.
First see if you can make it work with regular HTML file input (don't complicate with Base64 yet), as described here https://stackoverflow.com/a/70824288/2347084
If (1) works, then try converting base64 into a File object as suggested here https://stackoverflow.com/a/47497249/2347084
Combine (2) and (1)
I want to post my solution that worked, because as i can see in internet everybody have issues with FormData on nodejs
i was using axios to send the buffer for uploading file
issue is with axios and specially with FormData, it does not add Content-Lenght in headers, any version of axios does not do this
python API had required Content-Lenght
So kindly was asking to make this header optionally in python API and the code started to work
The solution is if anybody goes in similar issues
axios does not add Content-Lenght when working with FormData (could not find any version of axios that works)
if you work with a buffer without having the file on local disk than will be issue because of Content-Lenght
if u have the file locally than using the module fs u are able to read the file and add all hedears and Content-Lenght
on axios GitHub issue is saying that this bug is fixed in latest axios, but it was still not working in my case
below is a code by using buffer and Content-Lenght is not required in 3rd API
function upload (image: {imageBase64: string, fileName: string}) {
const { imageBase64, fileName } = image;
const cleanBase64 = imageBase64.substr(imageBase64.indexOf(',') + 1);
// buffer should be clean base64
const buffer = Buffer.from(cleanBase64, 'base64');
const formData = new FormData();
// filename as option is required, otherwise will not work, will say that received file is string and UploadFile
formData.append('file', buffer, { filename: fileName });
return client
.post('url', formData, {
headers: {
...formData.getHeaders(),
},
})
.then((response) => response.data)
.catch((error) => {
return {
status: 'error',
error,
};
});
}
I have a React component where I ask the user to insert an image using react-dropze. On drop, I save the image into an image state.
Like this:
const handleOnDrop = (files) => {
setimage(files[0]);
}
Once I submit, I send a request to my back-end in order to get the URL with this function:
export const generateUploadURL = async () => {
const rawBytes = await randomBytes(16);
const imageName = rawBytes.toString('hex');
const params = ({
Bucket: process.env.S3_BUCKET_NAME,
Key: imageName,
ContentType: 'image/*',
Expires: 60
})
const uploadUrl = await s3.getSignedUrlPromise('putObject', params);
return uploadUrl;
}
I get the URL and finally execute a put into the s3 with the URL:
await axios.put(url, {
headers: {
"Content-Type": "multipart/form-data"
},
body: image
});
And then I save the data into my database but that's not important.
The thing is, after that, I can't render the image from the link I stored so I went into the link and encountered this:
{"headers":{"Content-Type":"multipart/form-data"},"body":{"path":"asdasdsadtest.jpg"}}
I tried setting the content-type to the imageType but that didn't work either. I have no clue on how I could make it work.
Why are you using the s3.getSignedUrlPromise('putObject ") api? Using s3.upload would allow you to send the file in one go and would make it much simpler in my opinion. See: https://stackabuse.com/uploading-files-to-aws-s3-with-node-js/ for an example of this solution
Solved it the following way:
url = await getS3Url(image);
await axios.put(url, image, {
headers: {
"Content-Type": image.type
}
});
Insted of placing it into the body, this worked.
I need to upload an image/video to Linkedin through API. I am using axios and have Content-Type set to multipart/form-data and all my images/videos that need to be uploaded are stored with a url to the file. All files are stored remotely on cloudinary.
let bodyFormData = new FormData();
bodyFormData.append(
"fileupload",
request(file.url).pipe(fs.createWriteStream("video.mp4"))
);
axios
.post("https://api.linkedin.com/media/upload", bodyFormData, {
headers: {
Authorization: "Bearer " + account.accessToken,
"Content-Type": "multipart/form-data"
}
})
.then(linkedinImageResult => {
I am following this documentation here:
https://learn.microsoft.com/en-us/linkedin/marketing/integrations/community-management/shares/rich-media-shares#
One of the common errors I have gotten is:
UnhandledPromiseRejectionWarning: TypeError: source.pause is not a function
If I change
request(file.url).pipe(fs.createWriteStream("video.mp4"))
to just
file.url
I get this error:
'java.io.IOException: Missing initial multi part boundary'
If I remove
"Content-Type": "multipart/form-data"
I get this error:
"Unable to parse form content"
Note:
file.url is a url to a cloudinary file, example of file.url: "https://res.cloudinary.com/dnc1t9z9o/video/upload/v1555527484/mn3tyjcpg1u4anlma2v7.mp4"
Any help is greatly appreciated :)
Please note that using Rich Media is being deprecated:
Uploading image using https://api.linkedin.com/media/upload is being
deprecated. Partners are recommended to use assets API that returns
response such as urn:li:digitalmediaAsset:C5522AQHn46pwH96hxQ to post
shares.
If you need help with the asset API, please see my answer here
I found the solution for this specific situation!
Here is my example code.
const postRichMedia = async (mediaURL, accessToken, fileName) => {
const formData = new FormData();
const key = S3Service.getSocialKeyFromUrl(mediaURL);
const file = await S3Service.downloadFileFromS3(key);
formData.append("fileupload", file.Body, fileName);
try {
const {data} = await axios.post('https://api.linkedin.com/media/upload', formData, {
headers: {
...formData.getHeaders(),
Authorization: `Bearer ${accessToken}`
},
});
const {location} = data;
return location;
} catch (err) {
console.error(err);
}
};
I'm trying to give a temporary access to a service account that has only read permission to upload an mp3 to a Google Cloud Storage bucket.
For that, I'm generating a signedurl but i can't get it to work.
What I'm doing to generate the URL and upload the file :
const config = {
action: 'write',
expires: '03-12-2019',
contentType: 'audio/mpeg'
}
const storage = new Storage({
projectId: projectId,
keyFilename: './service-account-key.json'
})
let bucketSubDirectory = storage.bucket(bucketName).file('subdirectory/in/bucket/' + songName)
bucketSubDirectory.getSignedUrl(config, (err, url) => {
if (err) {
console.error(err)
return
}
console.log(url)
// The file is now available to be written to.
let writeStream = request.put(url, {
headers: {
'Content-Type': 'audio/mpeg'
}
})
writeStream.end('New data');
writeStream.on('complete', function (resp) {
console.log(resp.statusCode + " " + resp.statusMessage)
})
})
When I execute the script I get a 403 forbidden response and when I try to access the generated URL from the browser I get the following :
<Error>
<Code>MalformedSecurityHeader</Code>
<Message>Your request has a malformed header.</Message><ParameterName>signature</ParameterName>
<Details>Signature was not base64 encoded</Details>
</Error>
Any idea how can I solve this problem?
According to the node.js client library documentation, the method getSignedUrl has a parameter config which as a parameter contentType:
contentType
Optional
string
If you provide this value, the client must provide this HTTP header set to the same value.
So you might need to modify your PUT request to include this header with the value contentType: audio/mpeg.
Also, if you enter the url in the browser you are not doing a PUT request but a GET one.
EDIT:
Also check that your service account which creates the signedurl has the right permissions granted. For the code your are running you need at least the roles/storage.objectAdmin role. To grant it:
gcloud projects add-iam-policy-binding yourProjectID --member yourServiceAccount --role "roles/storage.objectAdmin"
Once this is done, users will be able to write to the file with just a PUT request to the url.
const request = require('request');
let url = 'yourSignedURL'
let writeStream = request.put(url, {
headers: {
'Content-Type': 'audio/mpeg'
}
})
writeStream.end('New data');
writeStream.on('complete', function (resp) {
console.log(resp.statusCode + " " + resp.statusMessage)
})