We want to execute invoke 10 tasks in parallel and handle each of the 10 results in parallel.
to achieve, created a list of tasks and used continuewith each of which are associated to async methods,
snippet
private async Task<List<bool>> TransformJobsAsync(
List<T> jobs)
{
var result = new List<bool>() { true };
var tasks = new List<Task<bool>>(jobs.Count);
try
{
foreach (var j in jobs)
{
tasks .Add(InvokeSomeAsync(j).ContinueWith(x => HandleResultAsync(x, j)).Unwrap());
}
await Task.WhenAll(tasks);
return tasks.Select(x => x.Result).ToList();
}
catch (Exception ex)
{
result = new List<bool>() { false };
}
return result;
}
Task<(T response, T job)> InvokeSomeAsync (T job)
{
var cts = new CancellationTokenSource();
try
{
cts.CancelAfter(30000);
var response = await SomeThirdPartyApi(request, cts.Token);
if (response.HttpStatusCode == System.Net.HttpStatusCode.OK)
{
}
return (response, job);
}
catch (OperationCanceledException opexException)
{
contextMessage = Messages.TranformationExecutionTimedOut;
}
catch (Exception ex)
{
contextMessage = Messages.UnHandledException;
}
finally
{
cts = null; //why? suggested pattern? review.
}
return await Task.FromException<(response, T Job)>(
throw new Exception());
}
async Task<bool> HandleResultAsync(Task<(T response, T job)> task, T job)
{
try
{
if (task.Status == TaskStatus.RanToCompletion)
{
if (task.Result.Status)
{
response = await CallMoreAsync(task.Result.reponse,
job, currentServiceState);
}
else
{
//log returned response = false
}
}
else
{
//log task failed
}
}
catch (Exception ex)
{
response = false;
}
finally
{
await DoCleanUpAsync();
}
return response;
}
I wanted to know if there is any better pattern and continuewith is not appropriate to use!
Sometimes We get this error, System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
You should not use ContinueWith. ContinueWith is a low-level, dangerous way to do the same thing as await. Modern code should use await instead.
To combine two asynchronous operations (e.g., InvokeSomeAsync and HandleResultAsync), introduce an async method:
async Task<bool> InvokeAndHandleResultAsync<T>(T job)
{
var task = InvokeSomeAsync(job);
return await HandleResultAsync(task, job);
}
This can then be used in your foreach:
foreach (var j in jobs)
{
tasks.Add(InvokeAndHandleResultAsync(j));
}
Other notes:
CancellationTokenSource should be disposed.
await Task.From* is usually a yellow flag.
Using Task.Status is a red flag.
Use exceptions for exceptional situations, rather than having a bool result with a contextMessage state somewhere else.
I'd write it something like:
async Task InvokeAndHandleResultAsync<T>(T job)
{
using (var cts = new CancellationTokenSource(30000))
{
try
{
var response = await SomeThirdPartyApi(request, cts.Token);
if (!response.Status)
{
//log returned response = false
return;
}
await CallMoreAsync(response, job, currentServiceState);
}
catch (Exception ex)
{
//log task failed
}
finally
{
await DoCleanUpAsync();
}
}
}
Also, instead of building a list of tasks, you can simplify that code:
private async Task TransformJobsAsync(List<T> jobs)
{
return Task.WhenAll(jobs.Select(j => InvokeAndHandleResult(j)));
}
Related
Async programming in node.js is usually done with callbacks. I find callback-based code hard to read and reason about, which is why I'm using async & await whenever I can. This almost always works well and leads to robust code. However, rarely I'm wondering whether I'm making things more difficult than necessary. For example, how do you create a stream such that you can await its creation? More specifically, the result of the await should of course be the stream itself when things go well. When they don't, an appropriate exception should be thrown.
The best I could come up with is the function below. It feels very clunky and I'm wondering whether there is an easier way to do this?
import type { EventEmitter } from "events";
export const createStreamAsync = async <T extends EventEmitter>(create: () => T): Promise<T> => {
const result = create();
let onOpen = (): void => void 0;
let onError = (reason?: unknown): void => void reason;
try {
await new Promise<void>((resolve, reject) => {
onOpen = resolve;
onError = reject;
result.on("open", onOpen).on("error", onError);
});
return result;
} finally {
result.removeListener("open", onOpen);
result.removeListener("error", onError);
}
};
You'd use the function as follows:
import { createWriteStream } from "fs";
import { createStreamAsync } from "./createStreamAsync.js";
try {
const stream = await createStreamAsync(() => createWriteStream("~/whatever.txt"));
// Use stream ...
} catch (error: unknown) {
// Handle error
}
Readable streams have an AsyncIterator symbol, so they can be processed with for await ... of:
const readable = fs.createReadStream('file.txt');
for await (const chunk of readable) {
console.log(chunk);
}
You can listen to both an event and catch the error event (if any) with events.once:
const { once } = require('node:events');
const writable = fs.createWriteStream('file.txt');
try {
await once(writable, 'open');
// If the event emits data, `once` returns it as an array:
// const x = await once(...);
} catch (err) {
console.error('Could not open file', err);
}
This will automatically remove the listener, just like EventEmitter.once does.
event.on returns an AsyncIterator to handle multiple events:
const { on } = require('node:events');
const { exec } = require('node:child_process');
const child = exec(...);
try {
for await (const event of on(child, 'message')) {
// `event` is an array of 0 or more values.
console.log(event);
}
} catch (err) {
console.error(err);
}
I am performing the useMutation operation in the innermost loop and want to check the remaining cost upon every mutation. But it gets checked after all the mutations which is a problem because for some reason even if all the mutations get done(When the cost is under limits), It calls the .then() part for cost-checking and waiting for unknown reason.
Edit: I also noticed that even though the program is waiting again and again, the network status of chrome shows that all the mutations have happened and only the query of handleDiscountMore i.e. fetchMore is pending
const { loading, error, data, fetchMore, extensions, refetch } = useQuery(GET_COLLECTION, {
variables: { "id": coll.collection.id }
});
const [updatePrice] = useMutation(UPDATE_PRICE);
const redirectToModify = async (data, totalProducts) => {
wait(20000);
var cursor, fetchCount;
fetchCount = data.collection.products.edges.length;
totalProducts -= fetchCount;
data.collection.products.edges.map(async(product) => {
const results = await Promise.all(product.node.variants.edges.map(variant => {
if (selected == 'curr_price') {
//do stuff
}
else {
//do stuff
}
const productVariableInput = {
//Object
};
updatePrice({ variables: { input: productVariableInput } }).then(({ data, extensions }) => {
console.log("Remaining", extensions.cost.throttleStatus.currentlyAvailable)
console.log(data)
if (extensions.cost.throttleStatus.currentlyAvailable < 100) {
console.log("WAITING")
wait(18000);
}
}).catch(e => {
console.log(e)
})
console.log("AFTER")
return 0;
}))
})
if (totalProducts > 0) {
console.log("Calling")
wait(15000);
handleDiscountMore(data, cursor, totalProducts)
}
};
//Below function is Just for reference. It gets called before checking the throttleStatus above. afaik there's no problem with this
const handleDiscountMore = (data, cursor, pc) => {
console.log("Call received")
fetchMore({
variables: {
"id": data.collection.id,
"cursor": cursor
},
updateQuery: (
previousResult,
{ fetchMoreResult }
) => {
console.log("adding", fetchMoreResult);
redirectToModify(fetchMoreResult, pc);
// return fetchMoreResult;
}
})
}
Your map of maps is evaluating all promises at exactly the same time. Here's a cleaned up example that uses a nested for loop instead, which will wait for each request to finish before starting the next (note: I couldn't run it to test, so there's probably some bugs, but the idea is there):
const id = coll.collection.id;
const { loading, error, data, fetchMore, extensions, refetch } = useQuery(GET_COLLECTION, {
variables: { id }
});
const [updatePrice] = useMutation(UPDATE_PRICE);
// Given a product, returns a promise that resolves when all variants are processed
async function process_product(product){
const variants = product.node.variants.edges;
for (let i = 0; i < variants.length; i++){
await process_variant(variants[i]);
}
}
// Given a variant, returns a promise after the product is processed
async function process_variant(variant){
if (variant) {
console.log('doing stuff')
}
else {
console.log('doing other stuff')
}
const productVariableInput = {};
const variables = { input: productVariableInput };
try {
const {data, extensions} = await updatePrice({ variables });
const remaining_throttle = extensions.cost.throttleStatus.currentlyAvailable;
console.log("Remaining", remaining_throttle)
console.log(data)
// Change to a while loop to make sure you actually wait until resources are available
if (remaining_throttle < 100) {
console.log("WAITING")
await wait(18000);
}
} catch (e) {
console.log('error:', e);
}
console.log("AFTER")
return 0;
}
const redirectToModify = async (data, totalProducts) => {
await wait(20000);
let cursor;
const products = data.collection.product.edges;
totalProducts = totalProducts - products.length;
// Wait for all products to finish processing
for (var i = 0; i < products.length; i++){
await process_product(products[i]);
}
if (totalProducts > 0) {
console.log("Calling")
await wait(15000);
handleDiscountMore(data, cursor, totalProducts)
}
};
function updateQuery(previousResult, { fetchMoreResult }){
console.log("adding", fetchMoreResult);
redirectToModify(fetchMoreResult, pc);
return fetchMoreResult;
}
//Below function is Just for reference. It gets called before checking the throttleStatus above. afaik there's no problem with this
function handleDiscountMore(data, cursor, pc) {
console.log("Call received")
const variables = { id: data.collection.id, cursor };
fetchMore({ variables, updateQuery })
}
I have a class which has 10 methods, and only 8 of those 10 should be exported, the other 2 are for authenticating tokens and I don't want them to be exported since the user won't need to call them ever ( I'm writing a package for npmjs ), my question is, how to only export the class with those 8 methods while the class itself is using the other two?
Edit: added some code
class StackClass {
constructor(p1, p2) {
this.data = {
p1,
p2,
accessToken: null,
expiresIn: null,
tokenType: null
};
}
async getToken() {
return new Promise(async (resolve, reject) => {
try {
let fetchResponse = await fetch(
"url with this.data.p1 and this.data.p2"
);
let fetchData = await fetchResponse.json();
if (fetchData.status != 200) {
reject("ERROR");
} else {
// get the token and save it in this.data
this.data.accessToken = fetchData.access_token;
this.data.expiresIn = fetchData.expires_in;
this.data.tokenType = fetchData.token_type;
resolve(fetchData);
}
} catch (err) {
console.log(err);
}
});
}
async isTokenValid() {
if (new Date() >= this.data.expiresIn) {
// Generate a new Token
try {
await this.getToken();
} catch (err) {
console.log(err);
}
}
// else - Do nothing and use current token saved in this.data.accessToken
}
async getData() {
try {
await this.isTokenValid();
await fetch("url2 with this.data.accessToken");
} catch (err) {
console.log(err);
}
}
}
let user = new StackClass("username", "password");
user.isTokenValid(); // SHOULD BE PRIVATE
user.getToken(); // SHOULD BE PRIVATE
user.getData(); // SHOULD BE PUBLIC
Define the isTokenValid and getToken as standalone functions instead. You should also avoid the explicit Promise construction antipattern, and only catch in a place where you can sensibly handle the error, which is usually in the caller of the asynchronous function (perhaps getData, perhaps in the consumer of the StackClass class):
class StackClass {
constructor(p1, p2) {
this.data = {
p1,
p2,
accessToken: null,
expiresIn: null,
tokenType: null
};
}
async getData() {
try {
await isTokenValid(this);
await fetch("url2 with this.data.accessToken");
} catch (err) {
// or you can just avoid the `try`/`catch` altogether,
// and have the consumer of getData handle problems
console.log(err);
}
}
}
function isTokenValid(stack) {
if (new Date() >= stack.data.expiresIn) {
// Generate a new Token
return getToken(stack);
}
// else - Do nothing and use current token saved in this.data.accessToken
}
async function getToken(stack) {
const fetchResponse = await fetch(
"url with this.data.p1 and this.data.p2"
);
const fetchData = await fetchResponse.json();
if (fetchData.status != 200) {
throw new Error('error');
} else {
// get the token and save it in this.data
stack.data.accessToken = fetchData.access_token;
stack.data.expiresIn = fetchData.expires_in;
stack.data.tokenType = fetchData.token_type;
}
}
Why don't you export a function that takes a list of arguments, creates an object of your class with those arguments, and then return a new object literal with only public methods? Take a look:
my-class.js
class MyClass {
constructor(f1, f2) {
this.field1 = f1;
this.field2 = f2;
}
m1() {
this.m2();
}
m2() {
console.log('this.field1', this.field1);
this.m3();
}
m3() {
console.log('this.field2', this.field2);
this._m4();
}
_m4() {
console.log('private m4');
this._m5();
}
_m5() {
console.log('private m5');
}
}
module.exports = (f1, f2) => {
const my = new MyClass(f1, f2);
return {
m1: my.m1.bind(my),
m2: my.m2.bind(my),
m3: my.m3.bind(my),
}
}
index.js
const myObj = require('./my-class')('f1', 'f2');
myObj.m1();
Live demo
This way you are hiding the private methods which should not be accessible outside of the module.
I need to iterate through an array. With each iteration, I need to update my database. I need to wait for the first update to be complete and then make the second update.
After searching through several answers, I found ASYNC/AWAIT feature of ES2017. However, I have not been able to implement it so far. The updates are happening randomly and not in a sequence. Please let me know how to implement ASYNC/AWAIT in this situation
Here is my code snippet:
function findRecipe(product, qty) {
return new Promise((resolve, reject) => {
Recipe.findOne({
product: product
}, (err, recipe) => {
if (err) {
reject(err)
} else {
for (let i = 0; i < recipe.items.length; i++) {
Item.findOne({
name: recipe.items[i].name
}, (err, item) => {
if (err) {
reject(err)
} else {
var lessAmt = recipe.quantities[i] * qty;
item.stock -= lessAmt;
item.save((err, item) => {
if (err) {
console.log(err)
} else {
resolve(item)
}
})
}
})
}
}
})
});
}
for (let i = 0; i < bill.product.length; i++) {
//Calling function for updates for each item
findRecipe(bill.product[i], bill.qty[i])
}
It looks like you are nearly there, Just Wrap the loop in a function and make it async.
async function updateAllRecipe(){
for(let i=0;i<bill.product.length;i++){
//Calling function for updates for each item
await findRecipe(bill.product[i],bill.qty[i])
}
}
But seriously though, I think you can leverage of parallelism here using the Promise.All. Is it really necessary to wait for the recipe to finish before queing the next findRecipe method? If not use the promise.all for it to perform faster
Async Await is easy to implement once you know the basic concept of asynchronous nature of Nodejs, I have used for...of loop here which also works asynchronously.
//Async function to perform database updates
async function findRecipe(product, qty) {
try {
let recipe = await Recipe.findOne({ product: product });
let i = 0;
for (itemObj of recipe.items) {
let item = await Item.findOne({ name: itemObj.name });
var lessAmt = recipe.quantities[i] * qty;
item.stock -= lessAmt;
let updatedItem = await item.save();
i++;
}
return true;
}
catch (err) {
return err;
}
}
async function someasyncFunction() {
for (let i = 0; i < bill.product.length; i++) {
//Calling function for updates for each item
await findRecipe(bill.product[i], bill.qty[i])
}
}
Use Promise.all and start process parallel. It'll increase the performance of API.
async function findRecipe(product, qty) {
try {
let recipe = await Recipe.findOne({
product: product
});
const items = await Promise.all(recipe.items.map(itemObj => Item.findOne({
name: itemObj.name
})));
const items = await Promise.all(items.map((item, i) => {
var lessAmt = recipe.quantities[i] * qty;
item.stock -= lessAmt;
return item.save();
}));
return true;
} catch (err) {
return err;
}
}
async function someasyncFunction() {
await Prmise.all(bill.product.map((product, i) => findRecipe(product, bill.qty[i])));
}
Novice question; I need to create a conditional loop which makes a call to the database. The value returned from the promise determines if I should break out of the loop as follows:
let id, caged;
do {
caged = false;
id = models.Cage.generateId();
caged = models.Cage.findOne({ where: { dispatchLabelId: id } });
} while( caged );
Can anybody please advise how I can structure my code?
You can do that using promises and recursive function calls:
let getCage = () => {
return models.Cage
.generateId()
.then(id => models.Cage.findOne({ where: { dispatchLabelId: id } }))
.then(caged => {
if (caged) {
return caged;
}
return getCage();
});
}
In the case if you can use node.js v7.6 or higher you can avoid using recursive function calls and implement this using async/await:
let getCage = async () => {
try {
do {
let id = await models.Cage.generateId();
let caged = await models.Cage.findOne({ where: { dispatchLabelId: id } });
if (caged) {
return caged;
}
} while (true);
} catch (err) {
console.log(err);
}
}