Need to create a buffer from UNIX timestamp to save it on bsv blockchain.
Tried this solution:
let buffer = Buffer.allocUnsafe(10);
buffer.writeUInt16BE(Date.now());
But getting an error:
RangeError [ERR_OUT_OF_RANGE]: The value of "value" is out of range. It must be >= 0 and <= 65535. Received 1568909911723
at checkInt (internal/buffer.js:35:11)
at writeU_Int16BE (internal/buffer.js:653:3)
at Buffer.writeUInt16BE (internal/buffer.js:661:10)
Try this:
let bignum = require('bignum');
let opts= {endian:'big',size:6 /*6-byte / 48-bit*/}
let dt = Date.now();
console.log(dt);
let num = bignum(dt.toString());
var buf = num.toBuffer(opts);
let a = bignum.fromBuffer(buf,opts)
console.log(a.toNumber());
Read this also:nodejs write 64bit unsigned integer to buffer. Will be helpful to understand the core concept.
Unix timestamp usually implies seconds while timestamps in JS contain milliseconds. If you need the milliseconds use Sandeep Patels solution.
If seconds are enough it can be simplified to this:
const seconds = Math.floor(Date.now() / 1000);
const timeBuffer = Buffer.alloc(4, 0x00);
timeBuffer.writeUInt32BE(seconds, 0);
Related
I have already read the following link but still get some errors with my current attempt:
let data = &[37u8, 42u8];
let data_two = &[0x34u8, 0x32u8];
let res:Vec<u8> = [data, data_two].concat();
Also, ideally I would like to avoid concatenation, and write an array of u8 to a buffer, where I reserve the first two bytes for storing length and index like:
let nb:u8 = get_chunks_nb();
let index:u8 = get_chunk_index();
let header = &[nb, index];
// this kind of things in C:
memcpy(&buffer, header, 2);
memcpy(&buffer[2], chunk, chunk_len);
Thank you for your help!
I took a shot at it, but I'm not 100% sure as to why, I'm still new to Rust.
It looks like the compiler is seeing data and data_two as arrays, and so [data, data_two] is then an array of array and not an array of slice. Which is probably why it couldn't find the concat method on it.
By explicitely saying that data is a slice, everything seems to fall into place:
let data:&[u8] = &[37u8, 42u8];
let data_two = &[0x34u8, 0x32u8];
let mut res:Vec<u8> = [data, data_two].concat();
Summary:
I'm aware there are CSV file reading libraries for Node.js. I wanted to make my own function to implement the most optimization possible based on the shortcuts I can take knowing the properties of my specific CSV format.
I'm working with a variable entry-count-per line file, so I can't simply scrape the buffer for a specified number of comma matches and load my field data with the correct buffers, I need to find the end of the line first. So this code is strictly aiming to go line by line through the file buffer by finding the next \n as efficiently as possible.
Problem:
binaryUtils.findBuffer is the issue in this code.
Question:
Is there a way to write binaryUtils.findBuffer more efficiently? Is it optimal for the given scenario?
Code:
I load the file into a buffer (or at least as large of a chunk of it as is possible):
// I know that the minimum length of a line in this CSV
// so I can skip ahead to find the exact location of the
// newline without testing every character pair for the
// end of the line:
const minOffset = 208 // minimum byte length of a line
const maxReadChunk = 2147483647
const csvPath = sourceFolderPath + "/" + processQueue[index]
console.log("loading source data path:", csvPath)
const fd = fs.openSync(csvPath, "r+")
const stats = fs.statSync(csvPath)
let optionCount = 0
let fileSize = stats["size"]
let totalFileSize = fileSize
let chunk = 0;
const chunks = Math.ceil(fileSize / maxReadChunk)
let bufferSize = Math.min(maxReadChunk, fileSize)
let readOffset = bufferSize
fileSize -= bufferSize
let fileBuffer = Buffer.allocUnsafe(bufferSize)
fs.readSync(fd, fileBuffer, 0, bufferSize, 0)
Then I loop through line by line, and benchmark along the way
let linePointer = 0
let nextLinePointer
let line
let findStart
let findEnd
let findDuration
let findTotal = 0
let loadStart = moment()
while (true) {
// `performance` imported from require('perf_hooks') to get fractions
// of milliseconds whereas moment() only captures nearest ms
findStart = performance.now()
// the logs that I show from the benchmarking on binaryUtils.findBuffer
// will demonstrates, as expected, that it is taking up 97%+ of the
// processing time. See below for binaryUtils.findBuffer code
nextLinePointer = binaryUtils.findBuffer(
newLineBuffer,
newLineBufferSize,
fileBuffer,
bufferSize,
linePointer,
minOffset
)
findEnd = performance.now()
findDuration = findEnd - findStart
findTotal += findDuration
// console.log("found nextLinePointer", nextLinePointer)
if (nextLinePointer === false) {
if (chunk < chunks) {
chunk++
console.log("progressed to chunk", chunk, "/", chunks)
bufferSize = Math.min(maxReadChunk, fileSize)
fileBuffer = Buffer.allocUnsafe(bufferSize)
fs.readSync(fd, fileBuffer, 0, bufferSize, readOffset)
readOffset += bufferSize
} else {
console.log("reached end of file")
break
}
} else {
optionCount++
if (optionCount % 5000 === 0) {
line = fileBuffer.slice(linePointer + 1, nextLinePointer)
let now = moment();
duration = now - loadStart;
let rate = optionCount / duration;
let timeDiff = now.diff(loadStart)
console.clear()
console.log("found line:", binaryUtils.readStr(line))
console.log("Initializing option chains at rate of",
rate.toFixed(2), "items / ms")
console.log("Chunk: \t", chunk + 1, "/", chunks)
console.log("Start time: \t", loadStart.format("HH:m:ss"))
console.log("Current time: \t", now.format("HH:m:ss"))
console.log("Duration: \t", moment.utc(timeDiff).format("HH:mm:ss"))
console.log("Processed: \t", nextLinePointer, "/", totalFileSize,
"bytes, (", ((nextLinePointer / totalFileSize) * 100).toFixed(2), "% )")
console.log("Finding line has taken:",
moment.utc(findTotal).format("HH:mm:ss"), "/",
moment.utc(timeDiff).format("HH:mm:ss"), "total duration (",
((findTotal / timeDiff) * 100).toFixed(2), "% )")
}
linePointer = nextLinePointer
}
}
And the culprit of the theoretically slow (35 line / ms) processing of this buffer is:
binaryUtils.findBuffer = (
targetBuffer,
targetBufferSize,
sourceBuffer,
sourceBufferSize,
pointer,
minOffset
) => {
pointer += minOffset
while (true) {
newPointer = pointer + targetBufferSize
if (newPointer > sourceBufferSize) return false
slice = sourceBuffer.slice(newPointer, newPointer + targetBufferSize)
if (slice.equals(targetBuffer)) {
return newPointer
}
pointer = newPointer
}
}
And here's the console output:
found line: FHR,2004-01-21 16:00:00,FHR,2004-02-21,30.000,P,0.0000,0.0000,0.0000,0.0000,0,20,2.7000,20,3.1000,27.2300,27.2500,1,9,20,2.7000,20,3.1000
Initializing option chains at rate of 87.33 items / ms
Chunk: 1 / 2
Start time: 08:47:40
Current time: 08:54:13
Duration: 00:06:33
Processed: 2147152582 / 3461193224 bytes, ( 62.04 % )
Finding line has taken: 00:06:25 / 00:06:33 total duration ( 97.90 % )
The processing rate is obviously not really useful for categorizing the operation as slow without knowing system specs and comparison benchmarks, but my question is about whether or not the fundamental design of binaryUtils.findBuffer is optimal.
A secondary benchmark just to make sure slicing buffers from a large buffer isn't slow for some reason:
let fileBuffer = Buffer.allocUnsafe(bufferSize)
fs.readSync(fd, fileBuffer, 0, bufferSize, 0)
let sliceCount = 100000
let sliceStart = performance.now()
for (let i = 0; i < sliceCount; i++) {
let newBuffer = fileBuffer.slice(i, i + 300)
}
let sliceEnd = performance.now()
let duration = sliceEnd - sliceStart
let rate = sliceCount / duration
console.log(sliceCount, "slices in", duration, "ms for rate of", rate, "/ ms")
output:
100000 slices in 7.09850001335144 ms for rate of 14087.483244616722 / ms
Ok, I keep making the mistake of forgetting just how much faster low level operations are than anything we can do manually with JS...
indexOf written in C is just so much faster than using a loop to search the buffer, that was the solution.
let start
const findBuffer = (
targetBuffer,
targetBufferSize,
sourceBuffer,
sourceBufferSize,
pointer,
minOffset
) => {
start = pointer + minOffset
return start + sourceBuffer.slice(start, sourceBufferSize).indexOf(targetBuffer)
}
750% faster, for a rate of 600 lines / ms
2500% faster, for a rate of 2400 lines / ms if you remove the variable declaration within the function and reuse start defined with let outside the function
I was trying:
r = [];
for (i = 0; i < 1e3; i++) {
a = (i+'').repeat(1e6);
r[i] = a.slice(64, 128);
}
and got an OutOfMemory. From here we see it's because all the as are kept in GC cuz a part of them are used.
How to make the slice don't keep the memory? I tried r[i]=''+a.slice(64, 128)+'' but still OOM. Do I have to a[64]+...+a[127] (loops also count as brute force)?
Is it so hard to slice and keep only necessary part of the old large string? The problem here only mentioned "copying every substring as a new string", but not "freeing part of the string remaining the necessary part assessible"
In this case it makes sense for the application code to be more aware of system constraints:
const r = [];
for (let i = 0; i < 1e3; ++i) {
const unitStr = String(i);
// choose something other than "1e6" here:
const maxRepeats = Math.ceil(128 / unitStr.length); // limit the size of the new string
// only using the last 64 characters...
r[i] = unitStr.repeat(maxRepeats).slice(64, 128);
}
...The application improvement is: to no longer construct 1000 strings of up to 3,000,000 bytes each when only 64 bytes are needed for each output string.
Your hardware and other constraints are not specified, but sometimes allowing the program more memory is appropriate:
node --max-old-space-size=8192 my-script.js
An analytic approach. Use logic to more precisely determine the in-memory state required for each working data chunk. With the constraints provided, minimize the generation of unneeded in-memory string data.
const r = new Array(1e3).fill().map((e,i) => outputRepeats(i));
function outputRepeats(idx) {
const OUTPUT_LENGTH = 128 - 64;
const unitStr = String(idx); // eg, '1', '40' or '286'
// determine from which character to start output from "unitStr"
const startIdxWithinUnit = (64 + 1) % unitStr.length; // this can be further optimized for known ranges of the "idx" input
// determine the approximate output string (may consume additional in-memory bytes: up to unitStr.length - 1)
// this can be logically simplified by unconditionally using a few more bytes of memory and eliminating the second arithmetic term
const maxOutputWindowStr = unitStr.repeat(Math.ceil(OUTPUT_LENGTH / unitStr.length) + Math.floor(Math.sign(startIdxWithinUnit)));
// return the exact resulting string
return maxOutputWindowStr.slice(startIdxWithinUnit, OUTPUT_LENGTH);
}
I see that BigInt is supported in node 10. However, there's no ReadBigInt() functionality in the Buffer class.
Is it possible to somehow go around it? Perhaps read 2 ints, cast them to BigInt, shift the upper one and add them to reconstruct the bigint?
A little late to the party here, but as the BigInt ctor accepts a hex string we can just convert the Buffer to a hex string and pass that in to the BigInt ctor. This also works for numbers > 2 ** 64 and doesn't require any dependencies.
function bufferToBigInt(buffer, start = 0, end = buffer.length) {
const bufferAsHexString = buffer.slice(start, end).toString("hex");
return BigInt(`0x${bufferAsHexString}`};
}
I recently had encountered the need to do this as well, and managed to find this npm library: https://github.com/no2chem/bigint-buffer ( https://www.npmjs.org/package/bigint-buffer ) which can read from a buffer as a BigInt.
Example Usage (reading, there is more examples on the linked github/npm):
const BigIntBuffer = require('bigint-buffer');
let testBuffer = Buffer.alloc(16);
testBuffer[0] = 0xff; // 255
console.log(BigIntBuffer.toBigIntBE(testBuffer));
// -> 338953138925153547590470800371487866880n
That will read the 16byte (128bit) number from the buffer.
If you wish to read only part of it as a BigInt, then slicing the buffer should work.
With Node v12, functions for reading bigint from buffers was added, so if possible, you should try to use Node v12 or later.
But these functions are just pure math based on reading integers from the buffer, so you can pretty much copy them into your Node 10-11 code.
https://github.com/nodejs/node/blob/v12.6.0/lib/internal/buffer.js#L78-L152
So modifying these methods to not be class methods could look something like this
function readBigUInt64LE(buffer, offset = 0) {
const first = buffer[offset];
const last = buffer[offset + 7];
if (first === undefined || last === undefined) {
throw new Error('Out of bounds');
}
const lo = first +
buffer[++offset] * 2 ** 8 +
buffer[++offset] * 2 ** 16 +
buffer[++offset] * 2 ** 24;
const hi = buffer[++offset] +
buffer[++offset] * 2 ** 8 +
buffer[++offset] * 2 ** 16 +
last * 2 ** 24;
return BigInt(lo) + (BigInt(hi) << 32n);
}
EDIT: For anyone else having the same issue, I created a package for this.
https://www.npmjs.com/package/read-bigint
One liner: BigInt('0x'+buffer.toString('hex'))
i'm trying to convert an string like "0:13:30", which consist of h:mm:ss, to an integer which will be an answer of (m*60)+(s), working only with the minutes and seconds in greasemonkey or jscript.
What i curently have is:
var t_str = ''; var t_int =0;
var str1='';var str2='';var t_int1=0;var t_int2=0;
t_str="0:13:30";
alert(t_str);
str1=t_str[4]+t_str[5];
str2=t_str[2]+t_str[3];
t_int1=parseInt (str1);
t_int2=parseInt (str2);
t_int2=t_int2 * 60;
t_int=t_int1+t_int2;
alert(t_int);
I get up to the first alert. how do i get it to assign the values "13" and "30" to str2 and str1? Sorry for a basic question, but im not used to this language :)
var time = t_str.split(":"),
h = 3600 * parseInt(time[0], 10),
m = 60 * parseInt(time[1], 10),
s = parseInt(time[2], 10);
alert(h+m+s);
I assume you are using javascript. If yes, then you can use split() method to split the t_str first and then carry on with your parsing of integers.
`str = t_str.split(":");` //- array of {0,13,30}
and then use this array to access your numbers.