I have the following problems. I have an integer and a string. Both of them need to be converted into binary format. For the integer I found a solution that, as far as I can tell, works. The string on the other hand, I don't have a solid understanding of it.
String(16), as far as I understand, means something like Array<UInt8> and has a fixed length of 16. Am I correct? If so, is there a better way to converting them by hand built in in NodeJS?
const myNumber = 2
const myString = 'MyString'
const myNumberInBinary = toUInt16(myNumber) // eg. 0000000000000101
const myStringinBinary = toString16(myString) // I honestly don't know
function toUInt16(number) {
let binaryString = Number(number).toString(2)
while (binaryString.length < 16) {
binaryString = '0' + binaryString
}
return binaryString
}
// TODO: implement
function toString16(string) {
...
return binaryString
}
best regards
EDIT:
Thanks for all the comments and the answer. They helped me understand this process better. The solution I ended up with is this one:
const bitString = "00000101"
const buffer = new Buffer.alloc(bitString.length / 8)
for (let i = 0; i < bitString.length; i++) {
const value = bitString.substring(i * 8, (i * 8) + 8)
buffer[i] = Number(value)
}
fs.writeFileSync('my_file', buffer, 'binary')
Thanks again!
You should loop through the string and do this on each character:
let result = ""
for (let i = 0; i < myString.length; i++) {
result += myString[i].charCodeAt(0).toString(2) + " ";
}
I am learning node.js
for example, I try
var server = require("./myserver.js");
and
var server = require("./myserver");
why do both work? What is this kind of things call?
If you take a look at node's source code for the require-function, you can see that require's load-function calls a function findLongestRegisteredExtension to determine the file-extension. This function returns .js as its default value if no extension could be determined:
function findLongestRegisteredExtension(filename) {
const name = path.basename(filename);
let currentExtension;
let index;
let startIndex = 0;
while ((index = name.indexOf('.', startIndex)) !== -1) {
startIndex = index + 1;
if (index === 0) continue; // Skip dotfiles like .gitignore
currentExtension = name.slice(index);
if (Module._extensions[currentExtension]) return currentExtension;
}
return '.js';
}
That's why using both require("./myserver.js"); or require("./myserver"); will resolve the same module.
I have been developing in PHP for many years and now I´m trying to learn NodeJS, and I´m having problems with asynchronous functions...
I have this function in NodeJS (source is an array of users):
//userformat.js
var age = module.exports.age = function(date){
var diff = new Date - new Date(date);
var diffdays = diff / 1000 / (60 * 60 * 24);
var age = Math.floor(diffdays / 365.25);
return age;
}
...etc.
//index.js
function format(source, me, callback){
var len = source.length
for (var i = 0; i < len; i++){
source[i]['_age'] = userFormat.age(source[i]['birthday']);
source[i]['_rating'] = userFormat.rating(source[i]['votes']);
source[i]['_is_followed'] = userFormat.followed(source[i]['fllws'], me);
}
callback(null, source);
}
I know that function is wrong in Node, because is possible that the function calls the callback before the functions in the for() had finished, and the values in the array could be undefined, for this reason I changed the function using callbacks:
//userformat.js
var age = module.exports.age = function(date, callback){
var diff = new Date - new Date(date);
var diffdays = diff / 1000 / (60 * 60 * 24);
var age = Math.floor(diffdays / 365.25);
callback(age);
}
...etc.
//index.js
function format(source, me, callback){
var len = source.length
for (var i = 0; i < len; i++){
userFormat.age(source[i]['birthday'], function(resul){
source[i]['_age'] = resul;
userFormat.rating(source[i]['votes'], function(resul){
source[i]['_rating'] = resul;
userFormat.followed(source[i]['fllws'], me, function(resul){
source[i]['_is_followed'] = resul;
//Callback
if (i == len-1){
callback(null, source);
}
})
})
})
}
}
That is correct? I have another way to do it more efficient or elegant?
Thank you so much!
I think you misunderstood the concept of asynchronous in nodeJS. Your code isn't asynchronous and your callback will always be called after the loop.
Check this question : how do i create a non-blocking asynchronous function in node.js?
In my node.js application I did an npm install btoa-atob so that I could use the btoa() and atob() functions which are native in client-side javascript but for some reason weren't included in node. The new directory showed up in my node_modules folder, which itself is in root alongside app.js. Then I made sure to add btoa-atob as a dependency in my package.json file which is in root.
However, for some reason, it still will not work.
console.log(btoa("Hello World!"));
^ should output "SGVsbG8gV29ybGQh" to the console, but instead, I get the error:
btoa is not defined.
Did I not do the install properly? What did I overlook?
The 'btoa-atob' module does not export a programmatic interface, it only provides command line utilities.
If you need to convert to Base64 you could do so using Buffer:
console.log(Buffer.from('Hello World!').toString('base64'));
Reverse (assuming the content you're decoding is a utf8 string):
console.log(Buffer.from(b64Encoded, 'base64').toString());
Note: prior to Node v4, use new Buffer rather than Buffer.from.
The solutions posted here don't work in non-ascii characters (i.e. if you plan to exchange base64 between Node.js and a browser). In order to make it work you have to mark the input text as 'binary'.
Buffer.from('Hélló wórld!!', 'binary').toString('base64')
This gives you SOlsbPMgd/NybGQhIQ==. If you make atob('SOlsbPMgd/NybGQhIQ==') in a browser it will decode it in the right way. It will do it right also in Node.js via:
Buffer.from('SOlsbPMgd/NybGQhIQ==', 'base64').toString('binary')
If you don't do the "binary part", you will decode wrongly the special chars.
I got it from the implementation of the btoa npm package:
My team ran into this problem when using Node with React Native and PouchDB. Here is how we solved it...
NPM install buffer:
$ npm install --save buffer
Ensure Buffer, btoa, and atob are loaded as a globals:
global.Buffer = global.Buffer || require('buffer').Buffer;
if (typeof btoa === 'undefined') {
global.btoa = function (str) {
return new Buffer(str, 'binary').toString('base64');
};
}
if (typeof atob === 'undefined') {
global.atob = function (b64Encoded) {
return new Buffer(b64Encoded, 'base64').toString('binary');
};
}
export const universalBtoa = str => {
try {
return btoa(str);
} catch (err) {
return Buffer.from(str).toString('base64');
}
};
export const universalAtob = b64Encoded => {
try {
return atob(b64Encoded);
} catch (err) {
return Buffer.from(b64Encoded, 'base64').toString();
}
};
I found that although the shims from answers above worked, they did not match the behaviour of desktop browsers' implementations of btoa() and atob():
const btoa = function(str){ return Buffer.from(str).toString('base64'); }
// returns "4pyT", yet in desktop Chrome would throw an error.
btoa('✓');
// returns "fsO1w6bCvA==", yet in desktop Chrome would return "fvXmvA=="
btoa(String.fromCharCode.apply(null, new Uint8Array([0x7e, 0xf5, 0xe6, 0xbc])));
As it turns out, Buffer instances represent/interpret strings encoded in UTF-8 by default. By contrast, in desktop Chrome, you can't even input a string that contains characters outside of the latin1 range into btoa(), as it will throw an exception: Uncaught DOMException: Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range.
Therefore, you need to explicitly set the encoding type to latin1 in order for your Node.js shim to match the encoding type of desktop Chrome:
const btoaLatin1 = function(str) { return Buffer.from(str, 'latin1').toString('base64'); }
const atobLatin1 = function(b64Encoded) {return Buffer.from(b64Encoded, 'base64').toString('latin1');}
const btoaUTF8 = function(str) { return Buffer.from(str, 'utf8').toString('base64'); }
const atobUTF8 = function(b64Encoded) {return Buffer.from(b64Encoded, 'base64').toString('utf8');}
btoaLatin1('✓'); // returns "Ew==" (would be preferable for it to throw error because this is undecodable)
atobLatin1(btoa('✓')); // returns "\u0019" (END OF MEDIUM)
btoaUTF8('✓'); // returns "4pyT"
atobUTF8(btoa('✓')); // returns "✓"
// returns "fvXmvA==", just like desktop Chrome
btoaLatin1(String.fromCharCode.apply(null, new Uint8Array([0x7e, 0xf5, 0xe6, 0xbc])));
// returns "fsO1w6bCvA=="
btoaUTF8(String.fromCharCode.apply(null, new Uint8Array([0x7e, 0xf5, 0xe6, 0xbc])));
I have a code shared between server and client and I needed an implementation of btoa inside it.
I tried doing something like:
const btoaImplementation = btoa || (str => Buffer.from(str).toString('base64'));
but the Server would crush with:
ReferenceError: btoa is not defined
while Buffer is not defined on the client.
I couldn't check window.btoa (it's a shared code, remember?)
So I ended up with this implementation:
const btoaImplementation = str => {
try {
return btoa(str);
} catch(err) {
return Buffer.from(str).toString('base64')
}
};
I was able to use btoa for binary data to base 64 string conversion using below npm package:
https://www.npmjs.com/package/btoa
As described in their documentation, I did below steps in node JS application:
Install => npm install --save btoa
Declare at top => const btoa = require('btoa');
Use => const b64 = btoa("stringToEncode");
Here's a concise universal solution for base64 encoding:
const nodeBtoa = (b) => Buffer.from(b).toString('base64');
export const base64encode = typeof btoa !== 'undefined' ? btoa : nodeBtoa;
Anybody looking to decode:
let decoded = Buffer.from(<encoded string>, 'base64').toString()
Because I came here looking for decoding, ended up figuring it out from an answer here.
Same problem with the 'script' plugin in the Atom editor, which is an old version of node, not having btoa(), nor atob(), nor does it support the Buffer datatype. Following code does the trick:
var Base64 = new function() {
var keyStr = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/="
this.encode = function(input) {
var output = "";
var chr1, chr2, chr3, enc1, enc2, enc3, enc4;
var i = 0;
input = Base64._utf8_encode(input);
while (i < input.length) {
chr1 = input.charCodeAt(i++);
chr2 = input.charCodeAt(i++);
chr3 = input.charCodeAt(i++);
enc1 = chr1 >> 2;
enc2 = ((chr1 & 3) << 4) | (chr2 >> 4);
enc3 = ((chr2 & 15) << 2) | (chr3 >> 6);
enc4 = chr3 & 63;
if (isNaN(chr2)) {
enc3 = enc4 = 64;
} else if (isNaN(chr3)) {
enc4 = 64;
}
output = output + keyStr.charAt(enc1) + keyStr.charAt(enc2) + keyStr.charAt(enc3) + keyStr.charAt(enc4);
}
return output;
}
this.decode = function(input) {
var output = "";
var chr1, chr2, chr3;
var enc1, enc2, enc3, enc4;
var i = 0;
input = input.replace(/[^A-Za-z0-9\+\/\=]/g, "");
while (i < input.length) {
enc1 = keyStr.indexOf(input.charAt(i++));
enc2 = keyStr.indexOf(input.charAt(i++));
enc3 = keyStr.indexOf(input.charAt(i++));
enc4 = keyStr.indexOf(input.charAt(i++));
chr1 = (enc1 << 2) | (enc2 >> 4);
chr2 = ((enc2 & 15) << 4) | (enc3 >> 2);
chr3 = ((enc3 & 3) << 6) | enc4;
output = output + String.fromCharCode(chr1);
if (enc3 != 64) {
output = output + String.fromCharCode(chr2);
}
if (enc4 != 64) {
output = output + String.fromCharCode(chr3);
}
}
output = Base64._utf8_decode(output);
return output;
}
this._utf8_encode = function(string) {
string = string.replace(/\r\n/g, "\n");
var utftext = "";
for (var n = 0; n < string.length; n++) {
var c = string.charCodeAt(n);
if (c < 128) {
utftext += String.fromCharCode(c);
} else if ((c > 127) && (c < 2048)) {
utftext += String.fromCharCode((c >> 6) | 192);
utftext += String.fromCharCode((c & 63) | 128);
} else {
utftext += String.fromCharCode((c >> 12) | 224);
utftext += String.fromCharCode(((c >> 6) & 63) | 128);
utftext += String.fromCharCode((c & 63) | 128);
}
}
return utftext;
}
this._utf8_decode = function(utftext) {
var string = "";
var i = 0;
var c = 0,
c1 = 0,
c2 = 0,
c3 = 0;
while (i < utftext.length) {
c = utftext.charCodeAt(i);
if (c < 128) {
string += String.fromCharCode(c);
i++;
} else if ((c > 191) && (c < 224)) {
c2 = utftext.charCodeAt(i + 1);
string += String.fromCharCode(((c & 31) << 6) | (c2 & 63));
i += 2;
} else {
c2 = utftext.charCodeAt(i + 1);
c3 = utftext.charCodeAt(i + 2);
string += String.fromCharCode(((c & 15) << 12) | ((c2 & 63) << 6) | (c3 & 63));
i += 3;
}
}
return string;
}
}()
var btoa = Base64.encode;
var atob = Base64.decode;
console.log("btoa('A') = " + btoa('A'));
console.log("btoa('QQ==') = " + atob('QQ=='));
console.log("btoa('B') = " + btoa('B'));
console.log("btoa('Qg==') = " + atob('Qg=='));
If you end up here, looking for atob is not defined solution (like me). Try upgrade your nodejs version - it helps me
I understand this is a discussion point for a node application, but in the interest of universal JavaScript applications running on a node server, which is how I arrived at this post, I have been researching this for a universal / isomorphic react app I have been building, and the package abab worked for me. In fact it was the only solution I could find that worked, rather than using the Buffer method also mentioned (I had typescript issues).
(This package is used by jsdom, which in turn is used by the window package.)
Getting back to my point; based on this, perhaps if this functionality is already written as an npm package like the one you mentioned, and has it's own algorithm based on W3 spec, you could install and use the abab package rather than writing you own function that may or may not be accurate based on encoding.
---EDIT---
I started having weird issues today with encoding (not sure why it's started happening now) with package abab. It seems to encode correctly most of the time, but sometimes on front end it encodes incorrectly. Spent a long time trying to debug, but switched to package base-64 as recommended, and it worked straight away. Definitely seemed to be down to the base64 algorithm of abab.
I am working on a Node.js application and I need to serialize and deserialize instances of the structs defined in an .thrift file, like the following:
struct Notification {
1: string subject,
2: string message
}
Now this is easy doable in Java, according to the tutorial at http://www.gettingcirrius.com/2011/03/rabbitmq-with-thrift-serialization.html :
Notification notification = new Notification();
TDeserializer deserializer = new TDeserializer();
deserializer.deserialize(notification, serializedNotification);
System.out.println("Received "+ notification.toString());
But I can't find how this is done using the nodejs library of Thrift. Can anyone help, please?
Ok, after wasting a lot of time on research and trying different solutions, I finally came to the answer to my own question:
//SERIALIZATION:
var buffer = new Buffer(notification);
var transport = new thrift.TFramedTransport(buffer);
var binaryProt = new thrift.TBinaryProtocol(transport);
notification.write(binaryProt);
where notification is the object I wish to serialize. At this point, the byte array can be found in the transport.outBuffers field:
var byteArray = transport.outBuffers;
For deserialization:
var tTransport = new thrift.TFramedTransport(byteArray);
var tProtocol = new thrift.TBinaryProtocol(tTransport);
var receivedNotif = new notification_type.Notification();
receivedNotif.read(tProtocol);
Assuming that the following lines have been added to the index.js file from the nodejs library for thrift:
exports.TFramedTransport = require('./transport').TFramedTransport;
exports.TBufferedTransport = require('./transport').TBufferedTransport;
exports.TBinaryProtocol = require('./protocol').TBinaryProtocol;
Here is my TypeScript version which runs in a browser. npm install buffer before use.
It should work on node if you remove import { Buffer }.
/*
Thrift serializer for browser and node.js
Author: Hirano Satoshi
Usage:
let byteArray = thriftSerialize(thriftObj);
let thriftObj2 = thriftDeserialize(byteArray, new ThriftClass())
let mayBeTrue = byteArrayCompare(byteArray, thriftSerialize(thriftObj2))
*/
import { TBufferedTransport, TFramedTransport, TJSONProtocol, TBinaryProtocol } from 'thrift';
import { Buffer } from 'buffer';
export function thriftSerialize(thriftObj: any): Buffer {
let transport = new TBufferedTransport(null);
let protocol = new TBinaryProtocol(transport);
thriftObj.write(protocol);
// copy array of array into byteArray
let source = transport.outBuffers;
var byteArrayLen = 0;
for (var i = 0, len = source.length; i < len; i++)
byteArrayLen += source[i].length;
let byteArray = new Buffer(byteArrayLen);
for (var i = 0, len = source.length, pos = 0; i < len; i++) {
let chunk = source[i];
chunk.copy(byteArray, pos);
pos += chunk.length;
}
return byteArray;
}
export function thriftDeserialize(byteArray: Buffer, thriftObj: any): any {
let transport = new TBufferedTransport(byteArray);
let callback = (transport_with_data) => {
var proto = new TBinaryProtocol(transport_with_data);
// var proto = new TJSONProtocol(transport);
thriftObj.read(proto);
}
// var buf = new Buffer(byteArray);
TBufferedTransport.receiver(callback)(byteArray);
return thriftObj;
}
export function byteArrayCompare(array1, array2): boolean {
if (!array1 || !array2)
return false;
let val = array1.length === array2.length && array1.every((value, index) => value === array2[index])
return val;
}
Somehow i did not find the the byte array at:
transport.outBuffers
i needed to do the following:
var transport = new Thrift.TFramedTransport(null, function(bytes){
dataWrapper.out = bytes;
cb(dataWrapper)
})
var binaryProt = new Thrift.TCompactProtocol(transport);
notification.write(binaryProt) ;
transport.flush() ; //important without the flush the transport callback will not be invoked