How to Write a String to Memory in WebAssembly - string

I started off working like this:
// example.wast
(module
(memory (export "mem") 1))
// example.js
WebAssembly.instantiateStreaming(fetch('example.wasm'), {}).then(mod => {
var exports = mod.instance.exports
var i8 = new Uint8Array(exports.mem)
var ra = 100
var rb = 500
var rc = 1000
var rd = 1200
var re = 2000
exports.copy(0, ra)
exports.copy(ra, rb)
exports.copy(rb, rc)
exports.copy(rc, rd)
exports.copy(rd, re)
console.log(ra, getString(ra, i8))
console.log(rb, getString(rb, i8))
console.log(rc, getString(rc, i8))
console.log(rd, getString(rd, i8))
console.log(re, getString(re, i8))
})
function getString(index, buffer) {
var s = ''
for (var i = index; i < index + size; i++) {
s += String.fromCharCode(buffer[i])
}
return s
}
But in order to implement that copy function, I first need to load strings into the WebAssembly memory. Wondering how I do that (using as little JS as possible, using mostly WebAssembly instead). Wondering if you can do this:
exports.loadString(index, 'Hello World', i8)
Or if not, you have to do something like this instead:
function loadString(index, string, buffer) {
for (var i = 0, n = string.length; i < n; i++) {
buffer[index + i] = string[i]
}
}
Or better yet:
function loadString(index, string, buffer) {
for (var i = 0, n = string.length; i < n; i++) {
exports.loadChar(index + i, string[i])
}
}
Wondering how to do this exports.loadChar or exports.loadString in WebAssembly, where it loads to a specific place in memory the string.
I would like to avoid using JS, i.e. buffer[index + i] = string[i]. Perhaps there is a way to dynamically load into the data segment, which would allow for exports.loadString.

WebAssembly does not have any of its own utility functions for reading / writing to linear memory, therefore there is no exported exports.loadString function.
WebAssembly exports a reference to linear memory itself, which you can read / write to as a typed array. You've already obtained a reference to it here:
var i8 = new Uint8Array(exports.mem)
This creates a byte array that allows you to read / write to the linear memory that was exported from your module with the name mem. You need to encode your string and write it to this array:
var encoder = new TextEncoder();
var encodedString = encoder.encode(str);
var i8 = new Uint8Array(exports.mem)
// Copy the UTF-8 encoded string into the WASM memory.
i8.set(encodedString);

Related

How to convert string and integer to binary in nodejs?

I have the following problems. I have an integer and a string. Both of them need to be converted into binary format. For the integer I found a solution that, as far as I can tell, works. The string on the other hand, I don't have a solid understanding of it.
String(16), as far as I understand, means something like Array<UInt8> and has a fixed length of 16. Am I correct? If so, is there a better way to converting them by hand built in in NodeJS?
const myNumber = 2
const myString = 'MyString'
const myNumberInBinary = toUInt16(myNumber) // eg. 0000000000000101
const myStringinBinary = toString16(myString) // I honestly don't know
function toUInt16(number) {
let binaryString = Number(number).toString(2)
while (binaryString.length < 16) {
binaryString = '0' + binaryString
}
return binaryString
}
// TODO: implement
function toString16(string) {
...
return binaryString
}
best regards
EDIT:
Thanks for all the comments and the answer. They helped me understand this process better. The solution I ended up with is this one:
const bitString = "00000101"
const buffer = new Buffer.alloc(bitString.length / 8)
for (let i = 0; i < bitString.length; i++) {
const value = bitString.substring(i * 8, (i * 8) + 8)
buffer[i] = Number(value)
}
fs.writeFileSync('my_file', buffer, 'binary')
Thanks again!
You should loop through the string and do this on each character:
let result = ""
for (let i = 0; i < myString.length; i++) {
result += myString[i].charCodeAt(0).toString(2) + " ";
}

Is it possible to bundle a wasm binary into a JavaScript bundle to avoid a second request?

I have an extremely start time sensitive application where the html has the JavaScript inlined to ensure only one request is made.
I'm investigating writing parts of the application in Rust/wasm but I can't find any resources describing how to inline the wasm binary.
Is it possible to and how would I go about doing that?
I can convert the .wasm file to base64 and load it as an ArrayBuffer. This will result in a file-size penalty of 30%
function _base64ToArrayBuffer(base64) {
var binary_string = window.atob(base64);
var len = binary_string.length;
var bytes = new Uint8Array(len);
for (var i = 0; i < len; i++) {
bytes[i] = binary_string.charCodeAt(i);
}
return bytes.buffer;
}
void async function main() {
const file = 'AGFzbQEAAAABCAJgAX8AYAAAAg8BB2NvbnNvbGUDbG9nAAADAgEBBxEBDWV4cG9ydGVkX2Z1bmMAAQoIAQYAQSoQAAsAFARuYW1lAQQBAAFpAgcCAAEAAAEA'
const bytes = _base64ToArrayBuffer(file)
const obj = await WebAssembly.instantiate(bytes, window)
obj.instance.exports.exported_func()
}()

Why does array this take up 1GB of memory?

I have files that store game map data, and I am using this function to read the map file and load it into memory when the server starts. Basically 4 bytes at the start tell the server how big the map is, and then each tile on the map has 6 bytes of data. I have roughly 33MB of data stored in multiple files, but when I read them into an array to access from memory, it takes up almost 1GB of RAM. I'm just wondering if something I am doing here is redundant or not needed and causing too much memory to be allocated.
Example: a 256x256 map would have 256 * 256 * 6 + 4 bytes of data
let mapData = [];
function loadMapFiles() {
fs.readdir("./maps", (err, files) => {
for (let file of files) {
let pointer = 4;
fs.readFile("./maps/" + file, (error, data) => {
if (error) throw error;
let sizex = data.readUInt16BE(0);
let sizey = data.readUInt16BE(2);
let mapNameNumber = Number(file);
mapData[mapNameNumber] = [];
for (let y = 0; y < sizey; y++) {
mapData[mapNameNumber][y] = [];
for (let x = 0; x < sizex; x++) {
mapData[mapNameNumber][y][x] = [];
mapData[mapNameNumber][y][x][0] = data.readUInt16BE(pointer);
mapData[mapNameNumber][y][x][1] = data.readUInt16BE(pointer + 2);
mapData[mapNameNumber][y][x][2] = data.readUInt16BE(pointer + 4);
pointer = pointer + 6;
}
}
});
}
});
}

Serialization-deserialization with Apache Thrift in nodejs

I am working on a Node.js application and I need to serialize and deserialize instances of the structs defined in an .thrift file, like the following:
struct Notification {
1: string subject,
2: string message
}
Now this is easy doable in Java, according to the tutorial at http://www.gettingcirrius.com/2011/03/rabbitmq-with-thrift-serialization.html :
Notification notification = new Notification();
TDeserializer deserializer = new TDeserializer();
deserializer.deserialize(notification, serializedNotification);
System.out.println("Received "+ notification.toString());
But I can't find how this is done using the nodejs library of Thrift. Can anyone help, please?
Ok, after wasting a lot of time on research and trying different solutions, I finally came to the answer to my own question:
//SERIALIZATION:
var buffer = new Buffer(notification);
var transport = new thrift.TFramedTransport(buffer);
var binaryProt = new thrift.TBinaryProtocol(transport);
notification.write(binaryProt);
where notification is the object I wish to serialize. At this point, the byte array can be found in the transport.outBuffers field:
var byteArray = transport.outBuffers;
For deserialization:
var tTransport = new thrift.TFramedTransport(byteArray);
var tProtocol = new thrift.TBinaryProtocol(tTransport);
var receivedNotif = new notification_type.Notification();
receivedNotif.read(tProtocol);
Assuming that the following lines have been added to the index.js file from the nodejs library for thrift:
exports.TFramedTransport = require('./transport').TFramedTransport;
exports.TBufferedTransport = require('./transport').TBufferedTransport;
exports.TBinaryProtocol = require('./protocol').TBinaryProtocol;
Here is my TypeScript version which runs in a browser. npm install buffer before use.
It should work on node if you remove import { Buffer }.
/*
Thrift serializer for browser and node.js
Author: Hirano Satoshi
Usage:
let byteArray = thriftSerialize(thriftObj);
let thriftObj2 = thriftDeserialize(byteArray, new ThriftClass())
let mayBeTrue = byteArrayCompare(byteArray, thriftSerialize(thriftObj2))
*/
import { TBufferedTransport, TFramedTransport, TJSONProtocol, TBinaryProtocol } from 'thrift';
import { Buffer } from 'buffer';
export function thriftSerialize(thriftObj: any): Buffer {
let transport = new TBufferedTransport(null);
let protocol = new TBinaryProtocol(transport);
thriftObj.write(protocol);
// copy array of array into byteArray
let source = transport.outBuffers;
var byteArrayLen = 0;
for (var i = 0, len = source.length; i < len; i++)
byteArrayLen += source[i].length;
let byteArray = new Buffer(byteArrayLen);
for (var i = 0, len = source.length, pos = 0; i < len; i++) {
let chunk = source[i];
chunk.copy(byteArray, pos);
pos += chunk.length;
}
return byteArray;
}
export function thriftDeserialize(byteArray: Buffer, thriftObj: any): any {
let transport = new TBufferedTransport(byteArray);
let callback = (transport_with_data) => {
var proto = new TBinaryProtocol(transport_with_data);
// var proto = new TJSONProtocol(transport);
thriftObj.read(proto);
}
// var buf = new Buffer(byteArray);
TBufferedTransport.receiver(callback)(byteArray);
return thriftObj;
}
export function byteArrayCompare(array1, array2): boolean {
if (!array1 || !array2)
return false;
let val = array1.length === array2.length && array1.every((value, index) => value === array2[index])
return val;
}
Somehow i did not find the the byte array at:
transport.outBuffers
i needed to do the following:
var transport = new Thrift.TFramedTransport(null, function(bytes){
dataWrapper.out = bytes;
cb(dataWrapper)
})
var binaryProt = new Thrift.TCompactProtocol(transport);
notification.write(binaryProt) ;
transport.flush() ; //important without the flush the transport callback will not be invoked

ActionScript 3: ByteArray to binary String

I've been asked to implement and MD5 hasher ActionScript-3 and as I was in the middle of debugging how I formatted my input I came across a problem. When I try and output the ByteArray as a binary string using .toString(2), the toString(2) method will perform some short cuts that alter how the binary should look.
For Example
var bytes:ByteArray = new ByteArray();
bytes.endian = Endian.LITTLE_ENDIAN;
bytes.writeUTFBytes("a");
bytes.writeByte(0x0);
var t1:String = bytes[0].toString(2); // is 1100001 when it should be 01100001
var t2:String = bytes[1].toString(2); // is 0 when it should be 00000000
so I guess my question is, might there a way to output a binary String from a ByteArray that will always shows each byte as a 8 bit block?
All you need is to pad the output of toString(2) with zeros on the left to make its length equal to 8. Use this function for padding
function padString(str:String, len:int, char:String, padLeft:Boolean = true):String{
var padLength:int = len - str.length;
var str_padding:String = "";
if(padLength > 0 && char.length == 1)
for(var i:int = 0; i < padLength; i++)
str_padding += char;
return (padLeft ? str_padding : "") + str + (!padLeft ? str_padding: "");
}
With this function the code looks like this and gives the correct output
var bytes:ByteArray = new ByteArray();
bytes.endian = Endian.LITTLE_ENDIAN;
bytes.writeUTFBytes("a");
bytes.writeByte(0x0);
var t1:String = padString(bytes[0].toString(2), 8, "0"); // is now 01100001
var t2:String = padString(bytes[1].toString(2), 8, "0"); // is now 00000000
Update
If you want to get a string representation of complete byteArray you can use a function which iterates on the byteArray. I have wrote the following function and it seems to work correctly. Give it a try
// String Padding function
function padString(str:String, len:int, char:String, padLeft:Boolean = true):String{
// get no of padding characters needed
var padLength:int = len - str.length;
// padding string
var str_padding:String = "";
// loop from 0 to no of padding characters needed
// Note: this loop will not run if padLength is less than 1
// as i < padLength will be false from begining
for(var i:int = 0; i < padLength; i++)
str_padding += char;
// return string with padding attached either to left or right depending on the padLeft Boolean
return (padLeft ? str_padding : "") + str + (!padLeft ? str_padding: "");
}
// Return a Binary String Representation of a byte Array
function byteArrayToBinaryString(bArray:ByteArray):String{
// binary string to return
var str:String = "";
// store length so that it is not recomputed on every loop
var aLen = bArray.length;
// loop over all available bytes and concatenate the padded string to return string
for(var i:int = 0; i < aLen; i++)
str += padString(bArray[i].toString(2), 8, "0");
// return binary string
return str;
}
Now you can simply use the byteArrayToBinaryString() function like this:
// init byte array and set Endianness
var bytes:ByteArray = new ByteArray();
bytes.endian = Endian.LITTLE_ENDIAN;
// write some data to byte array
bytes.writeUTFBytes("a");
bytes.writeByte(0x0);
// convert to binaryString
var byteStr:String = byteArrayToBinaryString(bytes); // returns 0110000100000000
Here is a function extended on the Hurlant library to handle hashing byteArray.
This class has a learning curve but once you get it you will love it.
As far as your ByteArray issue with toString. I know the toString method is not accurate For this very reason.
You might want to look into byteArray.readMultiByte that will give you the 01 you are looking for. Although I can't seem top get it to work on my sample code either lol
I just always get a and empty string.
var bytes:ByteArray = new ByteArray();
bytes.endian = Endian.LITTLE_ENDIAN;
bytes.writeUTFBytes("a");
bytes.writeByte(0x0);
bytes.position = 0
var t1:String = bytes.readMultiByte(1,'us-ascii'); // is 1100001 when it should be 01100001
trace(t1)
var t2:String = bytes.readMultiByte(1,'iso-8859-01'); // is 0 when it should be 00000000
trace(t2)

Resources