Vanilla node.js response encoding - weird behaviour (heroku) - node.js

Question

I'm developing my understanding of servers by writing a webdev framework using vanilla node.js. For the first time a situation arose where a french character was included in a json response from the server, and this character showed up as an unrecognized symbol (a question-mark within a diamond on chrome).
The problem was the encoding, which was being specified here:
/*
At this stage we have access to "response", which is an object of the
following format:
response = {
data: 'A string which contains the data to send',
encoding: 'The encoding type of the data, e.g. "text/html", "text/json"'
}
*/
var encoding = response.encoding;
var length = response.data.length;
var data = response.data;
res.writeHead(200, {
'Content-Type': encoding,
'Content-Length': length
});
res.end(data, 'binary'); // Everything is encoded as binary
The problem was that everything sent by the server is encoded as binary, which ruins the ability to display certain characters. The fix seemed simple; include a boolean binary value in response, and adjust the 2nd parameter of res.end accordingly:
/*
At this stage we have access to "response", which is an object of the
following format:
response = {
data: 'A string which contains the data to send',
encoding: 'The encoding type of the data, e.g. "text/html", "text/json"',
binary: 'a boolean value determining the transfer encoding'
}
*/
.
.
.
var binary = response.binary;
.
.
.
res.end(data, binary ? 'binary' : 'utf8'); // Encode responses appropriately
Here is where I have produced some very, very strange behavior. This modification causes french characters to appear correctly, but occasionally causes the last character of a response to be omitted on the client-side!!!
This bug only happens once I host my application on heroku. Locally, the last character is never missing.
I noticed this bug because certain responses (not all of them!) now break the JSON.parse call on the client-side, although they are only missing the final } character.
I have a horrible band-aid solution right now, which works:
var length = response.data.length + 1;
var data = response.data + ' ';
I am simply appending a space to every single response sent by the server. This actually causes all text/html, text/css, text/json, and application/javascript responses to work because they can tolerate the unnecessary whitespace, but I hate this solution and it will break other Content-Types!
My question is: can anyone give me some insight into this problem?

Answer 1

If you're going to explicitly set a Content-Length, you should always use Buffer.byteLength() on the body to calculate the length, since that method returns the actual number of bytes in the string and not the number of characters like the string .length property will return.