I use testcafe with Cucumber.
After update testcafe version to 1.20.0 some requests doesn't get response.
I implement a request hook like this:
class CustomRequestHook extends RequestHook {
constructor(requestFilterRules, responseEventConfigureOpts) {
super(requestFilterRules, responseEventConfigureOpts);
this.requests = {};
}
async onRequest(event) {
let requestInfo = event._requestInfo;
this.requests[requestInfo.requestId] = {
url: requestInfo.url,
method: requestInfo.method,
contentType: requestInfo.headers['content-type'],
body: requestInfo.body,
startTime: process.hrtime(),
startDate: new Date().toUTCString()
};
}
async onResponse(event) {
let request = this.requests[event.requestId];
const diffArray = process.hrtime(request.startTime);
request.duration = Math.ceil(diffArray[0] * 1000 + diffArray[1] / 1000000);
request.statusCode = event.statusCode;
}
getRequests() {
return Object.values(this.requests);
}
}
In the end of scenario I write info about requests to a file, but statusCode and duration are undefined for some requests. So the ui page doesn't loaded and test fails. This is a floating bug. For the same requests in different scenarios, the response may or may not recieved.
For testcafe 14.0.0 it works ok.
Has anyone encountered a similar problem?
I tried to update tescafe version to 2.2.0, but it did not help
Related
I am looking to implement a retry mechanism using retry-axios. I have successfully installed the package in the node project.
const baseUrl = `https://mock.codes/500`
const myAxiosInstance = axios.create();
myAxiosInstance.defaults.raxConfig = {
retry: 5,
retryDelay: 5000,
backoffType: 'static',
instance:myAxiosInstance,
onRetryAttempt: err => {
const cfg = rax.getConfig(err);
console.log(`Retry attempt #${cfg.currentRetryAttempt}`);
}
};
const interceptorId = rax.attach(myAxiosInstance);
const res = await myAxiosInstance.get(`${baseUrl}`);
The retry operation has been attempted only once. afterward, I got Invalid character in header content [\"0\"] error.
I need to start retrying the operation if the response is 500 or 400.
Thanks is advance
As Phil mentioned in his comment, this is a bug with Axios itself. It is known to affect Axios 1.1.0 and 1.1.2 at the very least. It was affecting me in 1.1.3 as well.
The fix is here but has not yet been approved. https://github.com/axios/axios/pull/5090
Commenters in the github issues suggested downgrading Axios to pre-1.0.0. I am no longer experiencing this issue after switching to:
axios 0.27.2
axios-retry 3.3.1
Hope that helps!
Since the above answer is highlighting the bug in the axios package itself, I recommend to manually resolving the issue as followed:
const axios = require("axios");
const baseUrl = `https://mock.codes/500`
let retryCount = 0; // Track the count of requests with retry count.
async function test() {
try {
const res = await axios.get(`${baseUrl}`); // Make request
console.log(res); // Log the successful response
} catch(err) {
if(err.response.status !== 200 && retryCount < 10) { // Check if the response code is not equal to 200, log and retry
console.log(`Retrying ${retryCount} time......`);
retryCount += 1;
await test();
} else {
console.log(`Retry count exceeded 10 times.`); // Log the maximum retry count, to avoid an infinite loop
}
}
}
test();
To add to what Appstronaut Studios said above,
axios version 1.3.2 just came out 2 days ago and that seems to fixed the issue as well. https://github.com/axios/axios/releases/tag/v1.3.2
In Node.js there is the cluster module to utilize all available cores on the machine which is pretty great, especially when used with the node module pm2. But I am pretty stoked about some features of Deno but I have wondered about how to best run it on a multi-core machine.
I understand that there is workers which works great for a specific task but for normal web requests it seems like performance of multi-core machines is wasted somewhat? What is the best strategy to get maximum availability and utilization of my hardware in Deno?
I am a bit worried that if you only have a single process going on and there is some CPU intensive task for whatever reason it will "block" all other requests coming in. In node.js the cluster module would solve this, since another process would handle the request but I am unsure on how to handle this in Deno?
I think you could run several instances in Deno on different ports and then have some kind of load balancer in front of it but that seems like quite a complex setup in comparison. I also get that you could use some kind of service like Deno Deploy or whatever, but I already have hardware that I want to run it on.
What are the alternatives for me?
Thanks in advance for you sage advice and better wisdom.
In Deno, like in a web browser, you should be able to use Web Workers to utilize 100% of a multi-core CPU.
In a cluster you need a "manager" node (which can be a worker itself too as needed/appropriate). In a similar fashion the Web Worker API can be used to create however many dedicated workers as desired. This means the main thread should never block as it can delegate all tasks that will potentially block to its workers. Tasks that won't block (e.g. simple database or other I/O bound calls) can be done directly on the main thread like normal.
Deno also supports navigator.hardwareConcurrency so you can query about available hardware and determine the number of desired workers accordingly. You might not need to define any limits though. Spawning a new dedicated worker from the same source as a previously spawned dedicated worker may be fast enough to do so on demand. Even so there may be value in reusing dedicated workers rather than spawning a new one for every request.
With Transferable Objects large data sets can be made available to/from workers without copying the data. This along with messaging makes it pretty straight forward to delegate tasks while avoiding performance bottlenecks from copying large data sets.
Depending on your use cases you might also use a library like Comlink "that removes the mental barrier of thinking about postMessage and hides the fact that you are working with workers."
e.g.
main.ts
import { serve } from "https://deno.land/std#0.133.0/http/server.ts";
import ComlinkRequestHandler from "./ComlinkRequestHandler.ts";
serve(async function handler(request) {
const worker = new Worker(new URL("./worker.ts", import.meta.url).href, {
type: "module",
});
const handler = ComlinkRequestHandler.wrap(worker);
return await handler(request);
});
worker.ts
/// <reference no-default-lib="true"/>
/// <reference lib="deno.worker" />
import ComlinkRequestHandler from "./ComlinkRequestHandler.ts";
ComlinkRequestHandler.expose(async (request) => {
const body = await request.text();
return new Response(`Hello to ${request.url}\n\nReceived:\n\n${body}\n`);
});
ComlinkRequestHandler.ts
import * as Comlink from "https://cdn.skypack.dev/comlink#4.3.1?dts";
interface RequestMessage extends Omit<RequestInit, "body" | "signal"> {
url: string;
headers: Record<string, string>;
hasBody: boolean;
}
interface ResponseMessage extends ResponseInit {
headers: Record<string, string>;
hasBody: boolean;
}
export default class ComlinkRequestHandler {
#handler: (request: Request) => Promise<Response>;
#responseBodyReader: ReadableStreamDefaultReader<Uint8Array> | undefined;
static expose(handler: (request: Request) => Promise<Response>) {
Comlink.expose(new ComlinkRequestHandler(handler));
}
static wrap(worker: Worker) {
const { handleRequest, nextResponseBodyChunk } =
Comlink.wrap<ComlinkRequestHandler>(worker);
return async (request: Request): Promise<Response> => {
const requestBodyReader = request.body?.getReader();
const requestMessage: RequestMessage = {
url: request.url,
hasBody: requestBodyReader !== undefined,
cache: request.cache,
credentials: request.credentials,
headers: Object.fromEntries(request.headers.entries()),
integrity: request.integrity,
keepalive: request.keepalive,
method: request.method,
mode: request.mode,
redirect: request.redirect,
referrer: request.referrer,
referrerPolicy: request.referrerPolicy,
};
const nextRequestBodyChunk = Comlink.proxy(async () => {
if (requestBodyReader === undefined) return undefined;
const { value } = await requestBodyReader.read();
return value;
});
const { hasBody: responseHasBody, ...responseInit } = await handleRequest(
requestMessage,
nextRequestBodyChunk
);
const responseBodyInit: BodyInit | null = responseHasBody
? new ReadableStream({
start(controller) {
async function push() {
const value = await nextResponseBodyChunk();
if (value === undefined) {
controller.close();
return;
}
controller.enqueue(value);
push();
}
push();
},
})
: null;
return new Response(responseBodyInit, responseInit);
};
}
constructor(handler: (request: Request) => Promise<Response>) {
this.#handler = handler;
}
async handleRequest(
{ url, hasBody, ...init }: RequestMessage,
nextRequestBodyChunk: () => Promise<Uint8Array | undefined>
): Promise<ResponseMessage> {
const request = new Request(
url,
hasBody
? {
...init,
body: new ReadableStream({
start(controller) {
async function push() {
const value = await nextRequestBodyChunk();
if (value === undefined) {
controller.close();
return;
}
controller.enqueue(value);
push();
}
push();
},
}),
}
: init
);
const response = await this.#handler(request);
this.#responseBodyReader = response.body?.getReader();
return {
hasBody: this.#responseBodyReader !== undefined,
headers: Object.fromEntries(response.headers.entries()),
status: response.status,
statusText: response.statusText,
};
}
async nextResponseBodyChunk(): Promise<Uint8Array | undefined> {
if (this.#responseBodyReader === undefined) return undefined;
const { value } = await this.#responseBodyReader.read();
return value;
}
}
Example usage:
% deno run --allow-net --allow-read main.ts
% curl -X POST --data '{"answer":42}' http://localhost:8000/foo/bar
Hello to http://localhost:8000/foo/bar
Received:
{"answer":42}
There's probably a better way to do this (e.g. via Comlink.transferHandlers and registering transfer handlers for Request, Response, and/or ReadableStream) but the idea is the same and will handle even large request or response payloads as the bodies are streamed via messaging.
It all depends on what workload you would like to push to the threads. If you are happy with the performance of the built in Deno HTTP server running on the main thread but you need to leverage multithreading to create the responses more efficiently then it's simple as of Deno v1.29.4.
The HTTP server will give you an async iterator server like
import { serve } from "https://deno.land/std/http/server.ts";
const server = serve({ port: 8000 });
Then you may use the built in functionality pooledMap like
import { pooledMap } from "https://deno.land/std#0.173.0/async/pool.ts";
const ress = pooledMap( window.navigator.hardwareConcurrency - 1
, server
, req => new Promise(v => v(respondWith(req))
);
for await (const res of ress) {
// respond with res
}
Where respondWith is just a function which handles the recieved request and generates the respond object. If respondWith is already an async function then you don't even need to wrap it into a promise.
However, in case you would like to run multiple Deno HTTP servers on separate therads then that's also possible but you need a load balancer like GoBetween at the head. In this case you should instantiate multiple Deno HTTP servers at separate threads and receive their requsets at the main thread as separate async iterators. To achieve this, per thread you can do like;
At the worker side i.e. ./servers/server_800X.ts;
import { serve } from "https://deno.land/std/http/server.ts";
const server = serve({ port: 800X });
console.log("Listening on http://localhost:800X/");
for await (const req of server) {
postMessage({ type: "request", req });
}
and at the main thread you can easily convert the correspodning worker http server into an async iterator like
async function* server_800X() {
worker_800X.onmessage = event => {
if (event.data.type === "request") {
yield event.data.req;
}
};
}
for await (const req of server_800X()) {
// Handle the request here in the main thread
}
You should also be able to multiplex either the HTTP (req) or the res async iterators by using the MuxAsyncIterators functionality in to a single stream and then spawn by pooledMap. So if you have 2 http servers working on server_8000.ts and server_8001.ts then you can multiplex them into a single async iterator like
const muxedServer = new MuxAsyncIterator<Request>();
muxedServer.add(server_8000);
muxedServer.add(server_8001);
for await (const req of muxedServer) {
// repond accordingly(*)
}
Obviously you should also be able to spawn new threads to process requests received from the muxedServer by utilizing pooledMap as shown above.
(*) In case you choose to use a load balancer and multiple Deno http servers then you should assign special headers to the requests at the load balancer, designating the server ID that it's been diverted to. This way, by inspecting this speical header you can decide from which server to respond for any particular request.
I'm trying to use the web3 Batch in order to call token balances all together. When I call batch.execute() it returns undefined instead of the resolved requests that have been added to the batch.
Can someone enlighten me where I am messing things up?
Here is my code.
async generateContractFunctionList(
address: Address,
tokens: Token[],
blockNumber: number
) {
const batch = new this.web3.BatchRequest();
for (let i = 0; i < tokens.length; i++) {
const contract = new this.web3.eth.Contract(balanceABI as AbiItem[]);
contract.options.address = tokens[i].address;
batch.add(
contract.methods
.balanceOf(address.address)
.call.request({}, blockNumber)
);
}
return batch;
}
async updateBalances() {
try {
const addresses = await this.addressService.find();
const tokens = await this.tokenService.find();
const blockNumber = await this.web3.eth.getBlockNumber();
for (let i = 0; i < addresses.length; i++) {
const address = addresses[i];
const batch = this.generateContractFunctionList(address, tokens, blockNumber);
const response = await (await batch).execute();
console.log(response); //returns undefined
}
} catch (error: unknown) {
if (error instanceof Error) {
console.log(`UpdateBalanceService updateBalances`, error.message);
}
}
}
why does batch.execute() not return anything and is void? I went by this example from this article and modified it to my needs but did not change too much of the stuff that could be messing it up.
https://chainstack.com/the-ultimate-guide-to-getting-multiple-token-balances-on-ethereum/
when I add a callback function to the "batch.add" and console log, balance get logged to console. But I am trying to use async await on the .execute() so how can I get a result from the method calls with await batch.execute() and have all the callback results in there like its written in the blog post.
A quick and dirty solution is to use an outdated version of the package.
package.json:
....
"dependencies": {
...
"web3": "^2.0.0-alpha.1",
...
}
....
I'm a developer advocate at Chainstack.
batch.add()requires a callback function as the last parameter. You can either change your code to pass a callback or, as mentioned above use version 2.0.0-alpha as used in the article.
We'll update the article soon to use the latest version of web3.js
I have a Javascript application that spawns Electron and does a bunch of stuff in it.
I'm trying to debug a strange network issue I'm having, and to do this, I'd like to use a HAR file to store a log of all the HTTP requests being made by Electron.
Is this possible?
Yes, it can be done using chrome-har-capturer - you can pass a bunch of events from the webContents.debugger and then chrome-har-capturer will transform them into a HAR for you.
Example code:
const chromeHarCapturer = require('chrome-har-capturer')
let log = []
const webContents = browserWindow.webContents
webContents.debugger.on("message", function(event, method, params) {
// https://github.com/cyrus-and/chrome-har-capturer#fromlogurl-log-options
if (!["Page.domContentEventFired", "Page.loadEventFired", "Network.requestWillBeSent", "Network.dataReceived",
"Network.responseReceived", "Network.resourceChangedPriority", "Network.loadingFinished",
"Network.loadingFailed"].includes(method)) {
// not relevant to us
return
}
log.push({method, params})
if (method === 'Network.responseReceived') { // the chrome events don't include the body, attach it manually if we want it in the HAR
webContents.debugger.sendCommand('Network.getResponseBody', {
requestId: params.requestId
}, function(err, result) {
result.requestId = params.requestId
log.push({
method: 'Network.getResponseBody',
params: result
})
})
}
})
webContents.debugger.once("detach", function() {
// on detach, write out the HAR
return chromeHarCapturer.fromLog("http://dummy-url-for-whole-session", log).then(function(har) {
const path = `/tmp/${Number(new Date())}-har.json`
fs.writeJson(path, log)
log = []
})
})
// subscribe to the required events
webContents.debugger.attach()
webContents.debugger.sendCommand('Network.enable')
webContents.debugger.sendCommand('Page.enable')
My problem is the following: I want to test a method that uploads a buch of data into an AWS S3 bucket. The problem is: I don't want to really upload data every time I am testing and I don't want to care about credentials sitting in the env. So I want to setup Sinon's fake-server module to simulate the upload and return the same results then S3 would. Sadly, it seems to be difficult to find a working example with code using async/await.
My test looks like this:
import {skip, test, suite} from "mocha-typescript";
import Chai from "chai";
import {S3Uploader} from "./s3-uploader.class";
import Sinon from "sinon";
#suite
class S3UploaderTest {
public server : Sinon.SinonFakeServer | undefined;
before() {
this.server = Sinon.fakeServer.create();
}
after() {
if (this.server != null) this.server.restore();
}
#test
async "should upload a file to s3 correctly"(){
let spy = Sinon.spy();
const uploader : S3Uploader = new S3Uploader();
const upload = await uploader.send("HalloWelt").toBucket("onetimeupload.test").toFolder("test/hw.txt").upload();
Chai.expect(upload).to.be.a("object");
}
}
Inside of the uploader.upload() method, I resolved a promise out of a callback. So how can I simulate the uploading-process?
Edit: Here is the code of the s3-uploader:
import AWS from "aws-sdk";
export class S3Uploader {
private s3 = new AWS.S3({ accessKeyId : process.env.ACCESS_KEY_ID, secretAccessKey : process.env.SECRET_ACCESS_KEY });
private params = {
Body: null || Object,
Bucket: "",
Key: ""
};
public send(stream : any) {
this.params.Body = stream;
return this;
}
public toBucket(bucket : string) {
this.params.Bucket = bucket;
return this;
}
public toFolder(path : string) {
this.params.Key = path;
return this;
}
public upload() {
return new Promise((resolve, reject) => {
if (process.env.ACCESS_KEY_ID == null || process.env.SECRET_ACCESS_KEY == null) {
return reject("ERR_NO_AWS_CREDENTIALS");
}
this.s3.upload(this.params, (error : any, data : any) => {
return error ? reject(error) : resolve(data);
});
});
}
}
Sinon fake servers are something you might use to develop a client that itself makes requests, instead of a wrapper around an existing client like AWS.S3, like you're doing. In this case, you're better off just stubbing the behavior of AWS.S3 instead of testing the actual requests it makes. That way you can avoid testing the implementation details of AWS.S3.
Since you're using TypeScript and you've made your s3 client private, you're going to need to make some changes to expose it to your tests. Otherwise, you won't be able to stub its methods without the TS compiler complaining about it. You also won't be able to write assertions using the params object, for similar reasons.
Since I don't use TS regularly, I'm not too familiar with it's common dependency injection techniques, but one thing you could do is add optional constructor arguments to your S3Uploader class that can overwrite the default s3 and arguments properties, like so:
constructor(s3, params) {
if (s3) this.s3 = s3;
if (params) this.params = params;
}
After which, you can create a stub instance and pass it to your test instance like this:
const s3 = sinon.createStubInstance(AWS.S3);
const params = { foo: 'bar' };
const uploader = new S3Uploader(s3, params);
Once you have the stub instance in place, you can write assertions to make sure the upload method was called the way you want it to be:
sinon.assert.calledOnce(s3.upload);
sinon.assert.calledWith(s3.upload, sinon.match.same(params), sinon.match.func);
You can also affect the behavior the upload method using the sinon stub api. For example, to make it fail like so:
s3.upload.callsArgWith(1, null);
Or make it succeed like so:
const data = { whatever: 'data', you: 'want' };
s3.upload.callsArgWith(1, null, data);
You'll probably want a completely separate test for each of these cases, using an instance before hook to avoid duplicating the common setup stuff. Testing for success will involve simply awaiting the promise and checking that its result is the data. Testing for failure will involve a try/catch that ensures the promise was rejected with the proper error.
Also, since you seem to be doing actual unit tests here, I'll recommend testing each S3Uploader method separately instead of calling them all in once big test. This drastically reduces the number of possible cases you need to cover, making your tests a lot more straightforward. Something like this:
#suite
class S3UploaderTest {
params: any; // Not sure the best way to type this.
s3: any; // Same. Sorry, not too experienced with TS.
uploader: S3Uploader | undefined;
before() {
this.params = {};
this.s3 = sinon.createStubInstance(AWS.S3);
this.uploader = new S3Uploader(this.s3, this.params);
}
#test
"send should set Body param and return instance"() {
const stream = "HalloWelt";
const result = this.uploader.send(stream);
Chai.expect(this.params.Body).to.equal(stream);
Chai.expect(result).to.equal(this.uploader);
}
#test
"toBucket should set Bucket param and return instance"() {
const bucket = "onetimeupload.test"
const result = this.uploader.toBucket(bucket);
Chai.expect(this.params.Bucket).to.equal(bucket);
Chai.expect(result).to.equal(this.uploader);
}
#test
"toFolder should set Key param and return instance"() {
const path = "onetimeupload.test"
const result = this.uploader.toFolder(path);
Chai.expect(this.params.Key).to.equal(path);
Chai.expect(result).to.equal(this.uploader);
}
#test
"upload should attempt upload to s3"() {
this.uploader.upload();
sinon.assert.calledOnce(this.s3.upload);
sinon.assert.calledWith(
this.s3.upload,
sinon.match.same(this.params),
sinon.match.func
);
}
#test
async "upload should resolve with response if successful"() {
const data = { foo: 'bar' };
s3.upload.callsArgWith(1, null, data);
const result = await this.uploader.upload();
Chai.expect(result).to.equal(data);
}
#test
async "upload should reject with error if not"() {
const error = new Error('Test Error');
s3.upload.callsArgWith(1, error, null);
try {
await this.uploader.upload();
throw new Error('Promise should have rejected.');
} catch(err) {
Chai.expect(err).to.equal(err);
}
}
}
If I were doing this with mocha proper, I'd group each method's tests into a nested describe block. I'm not sure if that's encouraged or even possible with mocha-typescript, but if so you might consider it.