I am trying to send a wav file to my ruby server using libcurl. It is a multi-form post, here's the main snippet of code:
curl_easy_setopt(curl, CURLOPT_URL, "192.168.0.136:3000/upload");
struct curl_httppost *post = NULL;
struct curl_httppost *last = NULL;
//Added longer timeout in case of large files
//Two forms, one for file and another for data
curl_formadd(&post, &last, CURLFORM_COPYNAME, "file", CURLFORM_FILECONTENT, "log1.wav", CURLFORM_END);
curl_formadd(&post, &last, CURLFORM_COPYNAME, "data", CURLFORM_PTRCONTENTS, data, CURLFORM_END);
curl_easy_setopt(curl, CURLOPT_HTTPPOST, post);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&respons_buf);
res = curl_easy_perform(curl);
I receive the file as a encoded text, that I am not sure what to do with. This is the same method I sent text files. Am I missing some header information?
libcurl doesn't encode or transform the data in any way when it sends a multipart form post. It sends the exact bytes it read from the file on your disk.
If you don't receive the exact same data in the server end, the server software probably performed some magic tricks on it for you. Possibly based on the non-existing Content-Type: header in that particular section of the post. You could try adding CURLFORM_CONTENTTYPE and the type for the file.
Related
From a node.js back end, I need to send an HTTP message to a REST endpoint. The endpoint requires some parameters that it will expect to find in the HTTP message. Some of the parameters are simple enough, just requiring a number or a string as an argument. But one of the parameters is to be "the raw binary file content being uploaded" and this has puzzled me. As far as I understand, the parameters need to be gathered together into a string to put in the body of the HTTP request; How do I add raw binary data to a string? Obviously, for it to be in the string, it cannot be raw binary data; it needs to be encoded into characters.
The endpoint in question is the Twitter media upload API. The "raw binary data" parameter is called media. Below is an incomplete code snippet showing the basic gist of what I've tried. Specifically, the line where I build the requestBody string. I don't believe it is anywhere near correct, because the endpoint is returning a "bad request" message.
var https = require("https");
var base64ImageData = /* (some base 64 string here) */;
var options = {
host: "api.twitter.com",
path: "/1.1/media/upload.json",
method: "POST",
headers: {
"Content-Type": "multipart/form-data"
}
};
var request = https.request(options, function(response) {});
var requestBody = "media_id=18283918294&media=" + Buffer.from(base64ImageData, "base64").toString("binary");
request.write(requestBody);
request.end();
Also worth noting, Twitter themselves note the following extremely confusing statement:
"When posting base64 encoded images, be sure to set the “Content-Transfer-Encoding: base64” on the image part of the message."
Source: https://developer.twitter.com/en/docs/media/upload-media/uploading-media/media-best-practices
That might be part of the answer to my question, but what I don't understand is: How do I apply different headers to different parts of the HTTP message? Because apparently, the image data needs to have a Content-Transfer-Encoding header of "base64" while the rest of the HTTP message does not...
How do I apply different headers to different parts of the HTTP message?
This is the point of the multipart/form-data content type. A multi-part message looks like this:
Content-Type: multipart/form-data; boundary=---foo---
---foo---
Content-Disposition: form-data; name="datafile1"; filename="r.gif"
Content-Transfer-Encoding: base64
Content-Type: image/gif
// data goes here
---foo---
Content-Disposition: form-data; name="datafile2"; filename="g.png"
Content-Transfer-Encoding: base64
Content-Type: image/png
// another file's data goes here
---foo---
You probably don't want to put all this together yourself. There are a bunch of good libraries for putting together complex POSTs. For example: https://www.npmjs.com/package/form-data
I have a simple uploading code by node.js.
var http = require('http')
var fs = require('fs')
var server = http.createServer(function(req, res){
if(req.url == '/upload') {
var a = fs.createWriteStream('a.jpg', { defaultEncoding: 'binary'})
req.on('data', function(chunk){
a.write(chunk)
})
.on('end', function()){
a.end()
res.end('okay')
})
}
else {
fs.createReadStream('./index.html').pipe(res);
// just show <form>
}
})
server.listen(5000)
when I upload some image, I cannot get exact same file.
Always files are broken.
When I try to do this using formidable, I can get a fine file.
So I studied formidable but I cannot understand how did it catch data and save.
I could find formidable use parser to calculate something about chunk from request but I did not get it all.
(It is definitely my brain issue :( ).
Anyway, what is the difference between my code and formidable?
What am I missing?
Is it a wrong way to just add all chunks from http request and save it by
fs.createWriteStream or fs.writeFile ?
What concepts am I missing?
First, req is a Readable stream. You can simply do:
req.pipe(fs.createWriteStream('a.jpg'))
for the upload part. This is copying all byte data from request stream to file.
This will work when you send raw file data as the request body:
curl --data-binary #"/home/user/Desktop/a.jpg" http://localhost:8080/upload
Because this sends request body exactly as image binary data, that gets streamed to a file on server.
But, there is another request format called multipart/form-data. This is what web browsers use with <form> to upload files.
curl -form "image=#/home/user1/Desktop/a.jpg" http://localhost:8080/upload
Here the request body contains multiple "parts", one for each file attachment or form field, separated by special "boundary" characters:
--------------------------e3f25f5319cd6624
Content-Disposition: form-data; name="image"; filename="a.jpg"
Content-Type: application/octet-stream
JPG IHDRH-ÑtEXtSoftwareAdobe.....raw file data
--------------------------e3f25f5319cd6624
Hence you will need much more complicated code to extract the file part data from it. NPM Modules like busboy and formidable do exactly that.
im downloading a zip file from the internet. I recieve it using a XHR request (using node-webkit) and this means that the content of the zip comes as a string in xhr.responseText. I now want to save this file to the disk, however, i cant seem to get it saved as a noncurrupted zip archive.
I have basically used fs.writeFile, fs.write, fs.createWriteStream, but I cant seem to get it right.
I am using a node module named AdmZip which accepts a file buffer that then can be saved as a zip archive. So, I guess, this could be one way to go, but how to I make a buffer out the the string that i recieve?
btw: i can't use the http module to recieve the file from the internet due to a bug in node.js, therefore im using the xhr request.
So, I found a soulution, by first and foremost setting the xhr.responseType = 'arraybuffer' and then turning the response into a Uint8Array. From there I converted the Uint8Array to a nodejs buffer which I then could save.
var arrayBuffer = xhr.response,
byteArray = new Uint8Array(arrayBuffer);
var buffer = new Buffer(byteArray.length);
for (var i = 0; i < byteArray.length; i++) {
buffer.writeUInt8(byteArray[i], i);
}
fs.writeFileSync(fname, buffer);
I need to implement a HTTP server in node.js without using http module. How fun!
I'm having trouble with sending the response socket.
I'm trying to fetch a file and so my code looks as follows:
fileStream = fs.createReadStream('example.jpg');
fileStream.on("end", function (close) {
var str = "HTTP/1.1 200 OK\r\Content-Type: image/jpeg\r\n" //and some more headers.
socket.write(str);
socket.pipe(fileStream);
socket.end("\r\n");
});
What am I missing?
I am of course using net module to get the socket and fs as well.
There are two main issues with the code you have.
Responses as formatted as headers followed by two sets of "\r\n", and then the content. The "\r\n" in your end() call should be in str. You also are missing an 'n' from the first "\r\n".
You are trying to pipe the contents of the readstream, which is great, but you are doing all of this inside of the steam's end(), so the pipe as nothing left to send because the data was all already emitted.
Try something like this instead. Create the read stream, send the response and pipe then rest.
var str = "HTTP/1.1 200 OK\r\nContent-Type: image/jpeg\r\n\r\n";
socket.write(str);
var fileStream = fs.createReadStream('example.jpg');
fileStream.pipe(socket);
I'm new to using groovy and have started to use it to test some REST services. I'm having an issue parsing my XML response from our service due to 'Content not allowed in prolog.' After awhile searching I came across a post saying there might be a Byte Order Marker at the beginning. To compensate I followed their approach to trim the characters before the first < and then parse the response. While this works, I was also told the issue is that the response is coming back as 'Transfer-Encoding: chunked'.
Using HTTPBuilder, is there a way to handle chunked responses without trimming characters off?
If I try:
def http = new HTTPBuilder('url')
http.request( Method.valueOf("GET"), XML )
I get the 'Content not allowed in prolog message. However:
http.request( Method.valueOf("GET"), TEXT )
Works, but requires trimming the text until the first < before sending the response to XmlParser.
I had the same issue when I needed to interact with an IIS server. The XML returned had a bogus character in front of the actual XML returned by the web server. I worked around it like this:
StringReader reader = builder.get( path: 'rcserver/systeminfo.xml', contentType: ContentType.TEXT )
def text = reader.getText()
def xml = new XmlSlurper().parseText(text.substring(1));
The HTTPBuilder class has a setContentEncoding() method that allows you to specify the response's content-type.
Maybe something like:
http.contentEncoding = ContentEncoding.Type.GZIP
http.request( Method.GET, XML)
Hope this helps.
I was having this problem as well hitting an IIS server over https. Here is a little addition to Wim Deblauwe's answer for a POST request. You have to send a different type in the request than you expect in the response.
Send a POST with XML as the request type and TEXT as the response type. Then, parse the text response into XML. This worked for me.
In Groovy:
def reader = http.request(Method.POST, ContentType.TEXT){
uri.path = "myPath.api"
send ContentType.XML, postBodyXml
}
def text = reader.getText()
def resultxml = new XmlSlurper().parseText(text.substring(1));