Exit from Array.map() loop - node.js

I have following code
const getCompanies = async (searchURL) => {
const html = await rp(baseURL + searchURL);
const businessMap = cheerio('a.business-name', html).map(async (i, e) => {
const phone = cheerio('p.phone', innerHtml).text();
if (phone == 123) {
exit;
}
return {
phone,
}
}).get();
return Promise.all(businessMap);
};
I want to exit in loop if my condition match. is there any way so if condition match then return the data right away. and stop execution of the loop

Your use case will be more suited to Array.some instead of Array.map
The some() method tests whether at least one element in the array passes the test implemented by the provided function. It returns a Boolean value.
So it stops execution as soon as any of the item in the array matches the condition.
You can use an external variable alongside to catch the matched value, e.g:
const getCompanies = async (searchURL) => {
const html = await rp(baseURL + searchURL);
let businessMap;
cheerio('a.business-name', html).some(async (i, e) => {
const phone = cheerio('p.phone', innerHtml).text();
if(phone == 123) {
// condition matched so assign data to businessMap here
// and return true so that execution stops
return true;
}
});
return Promise.all(businessMap);
};

businessMap is an array and there's nothing to await.
A simpler way to do this is:
const getCompanies = async (searchURL) => {
const html = await rp(baseURL + searchURL)
let $ = cheerio.load(html)
const ps = $('a.business-name p.phone').get()
return ps.map(p => $(p).text()).filter(phone => phone !== '123')
};

Related

Solana Web3.js getParsedTransaction sometimes return null

I am trying to get transaction values using signature id from any of the NFT's signature. For a test, I used the same signature and loop it 100 times to make sure that it is a valid signature with existing value
I did console.log on the index of the for loop. For some reason, it is returning null sometimes within the loop.
I am using Quicknode's RPC. ($9/month)
Am I missing something that caused this issue?
Code
async function test() {
for (let i = 0; i < 100; i++) {
signatures = await connection.getSignaturesForAddress(new PublicKey('BUPzNBDy3gVRRz1AqyzCDSnp15uAxh5j61dEj5GLLfx6'));
let signatures2 = signatures.map(({ signature }) => signature)
for (let a = 0; a < signatures2.length; a++) {
let txns = await connection.getParsedTransaction(signatures2[a], 'finalized')
if (!txns) {
console.log('null')
} else {
console.log('ok')
}
}
}
}
exports.get_collection_volume = (req, res) => {
test()
}
Output
0
1
2
3
null
null
6
7
null
null
null
null
12
null
14
null
16
I cannot reproduce the issue (also using Quiknode), I suspect it's probably an RPC/network issue.
However, looping 100 times on getParsedTransaction is probably not the best way to verify the validity of a signature. Instead you can use getSignatureStatus and verify that the transaction has a confirmationStatus of finalized and that err is null.
For instance:
const isValidSignature = async (connection: Connection, sig: string) => {
const status = await connection.getSignatureStatus(sig, {
searchTransactionHistory: true,
});
return (
status.value?.err === null &&
status.value?.confirmationStatus === "finalized"
);
};
If you don't just want to verify that the signature is valid but also extract parsed transaction details you can do something like:
const getParsedTx = async (connection: Connection, sig: string) => {
const parsed = await connection.getParsedTransaction(sig, "finalized");
if (!parsed || parsed?.meta?.err !== null) {
throw new Error("Invalid signature");
}
return parsed;
};

Why is one of my variables returning undefined in a switch and case?

I'm trying to create a currency converter from USD to whatever other currency. I have simplified the code to the part I am having issues with.
const toCurrency = document.getElementById("toCurrency");
const returnValue = document.getElementById("returnValue");
let usdInput = document.getElementById("inputValue");
let submit = {
conversionSubmit: () => {
conversion.conversion()
}
}
let conversion = {
conversion: () => {
switch(toCurrency.value) {
case USD: returnValue.innerText = usdInput.value;
return usdInput.value;
}
}
};
The issue here is that my usdInput.value returns undefined when I run the function even though when I see what the variable is through the console, it is perfectly defined.

nodejs function().anotherFunction() how they do it?

I install a library/module but I am curious how they do it
const modules = require('abc');
const app = new modules('123');
const client = app.f1('abcdef').f2('ghi');
console.log(client.to());
const client2 = app.f1('1111111').f2('2222');
console.log(client2.to());
console.log(client.to());
result is 123:abcdef-ghi
result is 123:1111111-2222
result is 123:abcdef-ghi
how they did it?
i want to create that example lib/module
please give me sample code
Depending on your exact requirements, something like this seems to work:
let state = Symbol("private")
class App {
constructor(arg) {
this[state] = arg
}
f1 = arg => new App(this[state] + ":" + arg)
f2 = arg => new App(this[state] + "-" + arg)
to = () => this[state]
}
Here each of the chainable method returns a new instance of the class, each keeping the temporary result as a local state. The to() method returns that state.
This is what I use for doing function().anotherFunction()
function func() {
console.log("ran func")
return {
"anotherFunc": function () {
console.log("ran another func")
}
}
}
func().anotherFunc()
You can also just return a object
const returnData = {
"anotherFunc": function () {
console.log("ran another func")
}
}
function func() {
console.log("ran func")
return returnData
}
func().anotherFunc()
Example of use:
function word1(in_1) {
return {
"word2": function (in_2) {
console.log(`Inputs: ${in_1} ${in_2}`)
}
}
}
word1("Hello").word2("World")
// Expected output: Inputs: Hello World

Why is the first function call is executed two times faster than all other sequential calls?

I have a custom JS iterator implementation and code for measuring performance of the latter implementation:
const ITERATION_END = Symbol('ITERATION_END');
const arrayIterator = (array) => {
let index = 0;
return {
hasValue: true,
next() {
if (index >= array.length) {
this.hasValue = false;
return ITERATION_END;
}
return array[index++];
},
};
};
const customIterator = (valueGetter) => {
return {
hasValue: true,
next() {
const nextValue = valueGetter();
if (nextValue === ITERATION_END) {
this.hasValue = false;
return ITERATION_END;
}
return nextValue;
},
};
};
const map = (iterator, selector) => customIterator(() => {
const value = iterator.next();
return value === ITERATION_END ? value : selector(value);
});
const filter = (iterator, predicate) => customIterator(() => {
if (!iterator.hasValue) {
return ITERATION_END;
}
let currentValue = iterator.next();
while (iterator.hasValue && currentValue !== ITERATION_END && !predicate(currentValue)) {
currentValue = iterator.next();
}
return currentValue;
});
const toArray = (iterator) => {
const array = [];
while (iterator.hasValue) {
const value = iterator.next();
if (value !== ITERATION_END) {
array.push(value);
}
}
return array;
};
const test = (fn, iterations) => {
const times = [];
for (let i = 0; i < iterations; i++) {
const start = performance.now();
fn();
times.push(performance.now() - start);
}
console.log(times);
console.log(times.reduce((sum, x) => sum + x, 0) / times.length);
}
const createData = () => Array.from({ length: 9000000 }, (_, i) => i + 1);
const testIterator = (data) => () => toArray(map(filter(arrayIterator(data), x => x % 2 === 0), x => x * 2))
test(testIterator(createData()), 10);
The output of the test function is very weird and unexpected - the first test run is constantly executed two times faster than all the other runs. One of the results, where the array contains all execution times and the number is the mean (I ran it on Node):
[
147.9088459983468,
396.3472499996424,
374.82447600364685,
367.74555300176144,
363.6300039961934,
362.44370299577713,
363.8418449983001,
390.86111199855804,
360.23125199973583,
358.4788999930024
]
348.6312940984964
Similar results can be observed using Deno runtime, however I could not reproduce this behaviour on other JS engines. What can be the reason behind it on the V8?
Environment:
Node v13.8.0, V8 v7.9.317.25-node.28,
Deno v1.3.3, V8 v8.6.334
(V8 developer here.) In short: it's inlining, or lack thereof, as decided by engine heuristics.
For an optimizing compiler, inlining a called function can have significant benefits (e.g.: avoids the call overhead, sometimes makes constant folding possible, or elimination of duplicate computations, sometimes even creates new opportunities for additional inlining), but comes at a cost: it makes the compilation itself slower, and it increases the risk of having to throw away the optimized code ("deoptimize") later due to some assumption that turns out not to hold. Inlining nothing would waste performance, inlining everything would waste performance, inlining exactly the right functions would require being able to predict the future behavior of the program, which is obviously impossible. So compilers use heuristics.
V8's optimizing compiler currently has a heuristic to inline functions only if it was always the same function that was called at a particular place. In this case, that's the case for the first iterations. Subsequent iterations then create new closures as callbacks, which from V8's point of view are new functions, so they don't get inlined. (V8 actually knows some advanced tricks that allow it to de-duplicate function instances coming from the same source in some cases and inline them anyway; but in this case those are not applicable [I'm not sure why]).
So in the first iteration, everything (including x => x % 2 === 0 and x => x * 2) gets inlined into toArray. From the second iteration onwards, that's no longer the case, and instead the generated code performs actual function calls.
That's probably fine; I would guess that in most real applications, the difference is barely measurable. (Reduced test cases tend to make such differences stand out more; but changing the design of a larger app based on observations made on a small test is often not the most impactful way to spend your time, and at worst can make things worse.)
Also, hand-optimizing code for engines/compilers is a difficult balance. I would generally recommend not to do that (because engines improve over time, and it really is their job to make your code fast); on the other hand, there clearly is more efficient code and less efficient code, and for maximum overall efficiency, everyone involved needs to do their part, i.e. you might as well make the engine's job simpler when you can.
If you do want to fine-tune performance of this, you can do so by separating code and data, thereby making sure that always the same functions get called. For example like this modified version of your code:
const ITERATION_END = Symbol('ITERATION_END');
class ArrayIterator {
constructor(array) {
this.array = array;
this.index = 0;
}
next() {
if (this.index >= this.array.length) return ITERATION_END;
return this.array[this.index++];
}
}
function arrayIterator(array) {
return new ArrayIterator(array);
}
class MapIterator {
constructor(source, modifier) {
this.source = source;
this.modifier = modifier;
}
next() {
const value = this.source.next();
return value === ITERATION_END ? value : this.modifier(value);
}
}
function map(iterator, selector) {
return new MapIterator(iterator, selector);
}
class FilterIterator {
constructor(source, predicate) {
this.source = source;
this.predicate = predicate;
}
next() {
let value = this.source.next();
while (value !== ITERATION_END && !this.predicate(value)) {
value = this.source.next();
}
return value;
}
}
function filter(iterator, predicate) {
return new FilterIterator(iterator, predicate);
}
function toArray(iterator) {
const array = [];
let value;
while ((value = iterator.next()) !== ITERATION_END) {
array.push(value);
}
return array;
}
function test(fn, iterations) {
for (let i = 0; i < iterations; i++) {
const start = performance.now();
fn();
console.log(performance.now() - start);
}
}
function createData() {
return Array.from({ length: 9000000 }, (_, i) => i + 1);
};
function even(x) { return x % 2 === 0; }
function double(x) { return x * 2; }
function testIterator(data) {
return function main() {
return toArray(map(filter(arrayIterator(data), even), double));
};
}
test(testIterator(createData()), 10);
Observe how there are no more dynamically created functions on the hot path, and the "public interface" (i.e. the way arrayIterator, map, filter, and toArray compose) is exactly the same as before, only under-the-hood details have changed. A benefit of giving all functions names is that you get more useful profiling output ;-)
Astute readers will notice that this modification only shifts the issue away: if you have several places in your code that call map and filter with different modifiers/predicates, then the inlineability issue will come up again. As I said above: microbenchmarks tend to be misleading, as real apps typically have different behavior...
(FWIW, this is pretty much the same effect as at Why is the execution time of this function call changing? .)
Just to add to this investigation, I compared the OP's original code with the predicate and selector functions declared as separate functions as suggested by jmrk to two other implementations. So, this code has three implementations:
OP's code with predicate and selector functions declared separately as named functions (not inline).
Using standard array.map() and .filter() (which you would think would be slower because of the extra creation of intermediate arrays)
Using a custom iteration that does both filtering and mapping in one iteration
The OP's attempt at saving time and making things faster is actually the slowest (on average). The custom iteration is the fastest.
I guess the lesson here is that it's not necessarily intuitive how you make things faster with the optimizing compiler so if you're tuning performance, you have to measure against the "typical" way of doing things (which may benefit from the most optimizations).
Also, note that in the method #3, the first two iterations are the slowest and then it gets faster - the opposite effect from the original code. Go figure.
The results are here:
[
99.90320014953613,
253.79690098762512,
271.3091011047363,
247.94990015029907,
247.457200050354,
261.9487009048462,
252.95090007781982,
250.8520998954773,
270.42809987068176,
249.340900182724
]
240.59370033740998
[
222.14270091056824,
220.48679995536804,
224.24630093574524,
237.07260012626648,
218.47070002555847,
218.1493010520935,
221.50559997558594,
223.3587999343872,
231.1618001461029,
243.55419993400574
]
226.01488029956818
[
147.81360006332397,
144.57479882240295,
73.13350009918213,
79.41700005531311,
77.38950109481812,
78.40880012512207,
112.31539988517761,
80.87990117073059,
76.7899010181427,
79.79679894447327
]
95.05192012786866
The code is here:
const { performance } = require('perf_hooks');
const ITERATION_END = Symbol('ITERATION_END');
const arrayIterator = (array) => {
let index = 0;
return {
hasValue: true,
next() {
if (index >= array.length) {
this.hasValue = false;
return ITERATION_END;
}
return array[index++];
},
};
};
const customIterator = (valueGetter) => {
return {
hasValue: true,
next() {
const nextValue = valueGetter();
if (nextValue === ITERATION_END) {
this.hasValue = false;
return ITERATION_END;
}
return nextValue;
},
};
};
const map = (iterator, selector) => customIterator(() => {
const value = iterator.next();
return value === ITERATION_END ? value : selector(value);
});
const filter = (iterator, predicate) => customIterator(() => {
if (!iterator.hasValue) {
return ITERATION_END;
}
let currentValue = iterator.next();
while (iterator.hasValue && currentValue !== ITERATION_END && !predicate(currentValue)) {
currentValue = iterator.next();
}
return currentValue;
});
const toArray = (iterator) => {
const array = [];
while (iterator.hasValue) {
const value = iterator.next();
if (value !== ITERATION_END) {
array.push(value);
}
}
return array;
};
const test = (fn, iterations) => {
const times = [];
let result;
for (let i = 0; i < iterations; i++) {
const start = performance.now();
result = fn();
times.push(performance.now() - start);
}
console.log(times);
console.log(times.reduce((sum, x) => sum + x, 0) / times.length);
return result;
}
const createData = () => Array.from({ length: 9000000 }, (_, i) => i + 1);
const cache = createData();
const comp1 = x => x % 2 === 0;
const comp2 = x => x * 2;
const testIterator = (data) => () => toArray(map(filter(arrayIterator(data), comp1), comp2))
// regular array filter and map
const testIterator2 = (data) => () => data.filter(comp1).map(comp2);
// combine filter and map in same operation
const testIterator3 = (data) => () => {
let result = [];
for (let value of data) {
if (comp1(value)) {
result.push(comp2(value));
}
}
return result;
}
const a = test(testIterator(cache), 10);
const b = test(testIterator2(cache), 10);
const c = test(testIterator3(cache), 10);
function compareArrays(a1, a2) {
if (a1.length !== a2.length) return false;
for (let [i, val] of a1.entries()) {
if (a2[i] !== val) return false;
}
return true;
}
console.log(a.length);
console.log(compareArrays(a, b));
console.log(compareArrays(a, c));

publishing observable to make it *hot*

I am trying to create a hot observable that does not need to be subscribed too. This is for a library and I would like to give users the ability to call certain methods and avoid the need to call subscribe() to get the observables to fire. Initially I have this:
const q = new Queue();
q.add('foo bar baz').subscribe(); // <<< need to call subscribe
Queue.prototype.add = Queue.prototype.enqueue = function (lines) {
lines = _.flattenDeep([lines]);
var lockAcquired = false;
return this.init()
.flatMap(() => {
return acquireLock(this)
})
.flatMap(() => {
lockAcquired = true;
return appendFile(this, lines)
})
.flatMap(() => releaseLock(this))
.catch(err => {
if (lockAcquired) {
return releaseLock(this);
}
else {
return makeGenericObservable();
}
})
};
in order to make the observable hot, I thought I could do something like this:
const q = new Queue();
q.add('foo bar baz'); // <<< don't call subscribe
Queue.prototype.add = Queue.prototype.enqueue = function (lines) {
lines = _.flattenDeep([lines]);
var lockAcquired = false;
return this.init()
.flatMap(() => {
return acquireLock(this)
})
.flatMap(() => {
lockAcquired = true;
return appendFile(this, lines)
})
.flatMap(() => releaseLock(this))
.catch(err => {
if (lockAcquired) {
return releaseLock(this);
}
else {
return makeGenericObservable();
}
})
.publish()
.share() // this too?
};
however the problem is when I call publish(), nothing happens, and the add method never seems to get fully invoked (I assume the first observable in the sequence never fires at all, because effectively subscribe is not called). But I though publish() would automatically invoke the observable chain?
How can I make the observable returned from the add method, hot?
You misunderstood. Hot observable does require subscription like a cold one. Difference is that hot uses some external producer (like dome element) and start listen to it on subscription. On the other hand cold observable creates producer internally on subscription.
It results in that you can miss some events with hot observable because external producer knows nothing about subscription and emits independently. And with cold observable you can't miss anything because producer is created on subscription.
Long story short you can build any chain of observable on top of either hot or cold observable but until you subscribe to it nothing will happen.
PS. No need to use publish with share because latter is alias for .publish().refCount().
If you don't want to use subscribe, you could manually .connect() your stream or subscribe it inside your method:
const q = new Queue();
q.add('foo bar baz');
Queue.prototype.add = Queue.prototype.enqueue = function (lines) {
lines = _.flattenDeep([lines]);
var lockAcquired = false;
let add$ = this.init()
.flatMap(() => {
return acquireLock(this)
})
.flatMap(() => {
lockAcquired = true;
return appendFile(this, lines)
})
.flatMap(() => releaseLock(this))
.catch(err => {
if (lockAcquired) {
return releaseLock(this);
}
else {
return makeGenericObservable();
}
})
.publish();
add$.connect();
return add$; // optional, depends if you even need the stream outside of the add-method
};
or as an alternative to use an internal subscribe:
const q = new Queue();
q.add('foo bar baz'); // <<< don't call subscribe
Queue.prototype.add = Queue.prototype.enqueue = function (lines) {
lines = _.flattenDeep([lines]);
var lockAcquired = false;
let add$ = this.init()
...
.share();
add$.subscribe();
return add$;
};

Resources