The following code is a serial port event inside a winforms form ( so obviously running on it's own thread ). The slow line of code is run elsewhere ( in Nodejs ) and takes about 10 seconds. In this code the same line takes 45 seconds - sometimes 60 seconds.
private async void SerialPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
SerialPort sp = (SerialPort)sender;
string indata = sp.ReadLine().Replace("?","").Replace(" ","");
if(indata != null)
{
var weightMatch = Regex.Match(indata, #"(\d+\.\d+)kg$");
var weight = weightMatch.Groups[1].Value;
var message = barcode + " Weight:" + weight;
this.Invoke(new Action(() => { scanData.Add(new ScanDataItem(scanData.Count + 1, message, "")); }));
this.Invoke(new Action(() => { mainList.SelectedIndex = mainList.Items.Count - 1; }));
string jsonData = "";
// ***************************************************************
// The following line is about 30 seconds slower than it should be - inner function calls a web service
// ***************************************************************
var jsonString = await TrayWeighScan.doOrder(barcode, Decimal.Parse(weight));
try
{
var jsonDoc = JsonDocument.Parse(jsonString);
jsonData = JsonSerializer.Serialize(jsonDoc, new JsonSerializerOptions { WriteIndented = true });
}
catch (Exception)
{
jsonData = jsonString;
}
this.Invoke(new Action(() => { scanData.Last().responseData = jsonData; }));
this.Invoke(new Action(() => { responseTextBox.Text = jsonData; }));
this.Invoke(new Action(() => { statusTextBox.Text = "Ready to Scan"; }));
this.Invoke(new Action(() => { busy = false; }));
}
}
public class TrayWeighScan
{
//private static readonly HttpClient httpClient = new();
public static async Task<string> doOrder(string orderNumber, decimal weight )
{
HttpClient httpClient = new();
var url = "http://r2hserver/logistics/weighscan?orderNumber="+orderNumber+"&weight="+weight.ToString("F2");
HttpResponseMessage response = await httpClient.GetAsync(url);
string responseText = await response.Content.ReadAsStringAsync();
return responseText;
}
}
If I run the "slow" line directly from a form button, 11 seconds.
private async void btnUrlTest_Click(object sender, EventArgs e)
{
string orderNumber = "ORD100302338";
decimal weight = 1.606m;
//following line - 11 seconds.
var a = await TrayWeighScan.doOrder(orderNumber,weight);
var b = "";
}
Hans Passant as pointed out that most likely the cause isn't in the code structure at all, and most likely he's right. To test the hypothesis, descend into trayWeighScan.doOrder and change the async HttpClient async call into a sync call. If the problem goes away, my answer is good. If the problem remains, look elsewhere, say the network or the client machine's TCP stack or AV or somesuch.
I've not seen this exact thing before, but I've learned that launching async operations in a WinForms app doesn't do the expected when launched in the naive way. The following should work much better
Task.Run(async () => {
var jsonString = await trayWeighScan.doOrder(barcode, Decimal.Parse(weight);
try
{
var jsonDoc = JsonDocument.Parse(jsonString);
jsonData = JsonSerializer.Serialize(jsonDoc, new JsonSerializerOptions { WriteIndented = true });
}
catch (Exception)
{
jsonData = jsonString;
}
this.Invoke(new Action(() => { scanData.Last().responseData = jsonData; }));
this.Invoke(new Action(() => { responseTextBox.Text = jsonData; }));
this.Invoke(new Action(() => { statusTextBox.Text = "Ready to Scan"; }));
this.Invoke(new Action(() => { busy = false; }));
});
Wrapping winforms-originating async in Task.Run seems to completely fix the problem.
A lot of stuff talks about .ConfigureAwait() being the solution; but it's overly complex. The simplest and best solution is to only spawn async/await from threadpool threads, which means starting an async/await chain from Task.Run.
Related
I am trying to develop a MS Teams bot that sends content to students module(unit) wise. I have created 3 classes:
methods.js = Contains all the methods for sending texts, attachments etc.
teamBot.js = Captures a specific keyword from the users and based on that executes a function.
test.js = Connects the bot with Airtable and sends the content accordingly
I am facing Cannot perform 'get' on a proxy that has been revoked error. I figured it might be because of the context. I am passing context as a parameter, which I feel might not be the correct way, how can I achieve the result, and retain the context between files.
teamsBot.js
const test = require("./test");
class TeamsBot extends TeamsActivityHandler {
constructor() {
super();
// record the likeCount
this.likeCountObj = { likeCount: 0 };
this.onMessage(async (context, next) => {
console.log("Running with Message Activity.");
let txt = context.activity.text;
// const removedMentionText = TurnContext.removeRecipientMention(context.activity);
// if (removedMentionText) {
// // Remove the line break
// txt = removedMentionText.toLowerCase().replace(/\n|\r/g, "").trim();
// }
// Trigger command by IM text
switch (txt) {
case "Begin": {
await test.sendModuleContent(context)
}
// By calling next() you ensure that the next BotHandler is run.
await next();
});
// Listen to MembersAdded event, view https://learn.microsoft.com/en-us/microsoftteams/platform/resources/bot-v3/bots-notifications for more events
this.onMembersAdded(async (context, next) => {
const membersAdded = context.activity.membersAdded;
for (let cnt = 0; cnt < membersAdded.length; cnt++) {
if (membersAdded[cnt].id) {
const card = cardTools.AdaptiveCards.declareWithoutData(rawWelcomeCard).render();
await context.sendActivity({ attachments: [CardFactory.adaptiveCard(card)] });
break;
}
}
await next();
});
}
test.js
const ms = require('./methods')
async function sendModuleContent(context) {
data = module_text //fetched from Airtable
await ms.sendText(context, data)
}
methods.js
const {TeamsActivityHandler, ActivityHandler, MessageFactory } = require('botbuilder');
async function sendText(context, text){
console.log("Sending text")
await context.sendActivity(text);
}
Refer this: TypeError: Cannot perform 'get' on a proxy that has been revoked
make the following changes to test.js
const {
TurnContext
} = require("botbuilder");
var conversationReferences = {};
var adapter;
async function sendModuleContent(context) {
data = module_text //fetched from Airtable
const currentUser = context.activity.from.id;
conversationReferences[currentUser] = TurnContext.getConversationReference(context.activity);
adapter = context.adapter;
await adapter.continueConversation(conversationReferences[currentUser], async turnContext => {
await turnContext.sendActivity(data);
});
}
I am using the code below to setup MassTransit to use ServiceBus
private static ServiceProvider SetupServiceCollection()
{
var connectionString = ConfigurationManager.AppSettings["AzureServiceBusConnectionString"];
var services = new ServiceCollection()
.AddMassTransit(x =>
{
x.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host(connectionString);
cfg.ConfigureEndpoints(context);
cfg.Message<MyMessage>(x =>
{
x.SetEntityName("my-topic");
});
});
});
return services.BuildServiceProvider();
}
I use the following code to send a message
var message = new MyMessage()
{
MessageIdentifier = Guid.NewGuid().ToString(),
};
await _busControl.Publish(message);
I want my message to be sent to my-topic only
However, MassTransit is creating topic, the names seem to be getting generated using the type names. How do I totally stop this?
I am setting up the receiver as below
public static void SetupMassTransit(this ServiceCollection services, string connectionString)
{
services.AddMassTransit(x =>
{
x.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host(connectionString);
cfg.ConfigureEndpoints(context);
x.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host(connectionString);
cfg.SubscriptionEndpoint<MyMessage>("low", e =>
{
e.Consumer<MyMessageConsumer>(context);
e.PrefetchCount = 100;
e.MaxConcurrentCalls = 100;
e.LockDuration = TimeSpan.FromMinutes(5);
e.MaxAutoRenewDuration = TimeSpan.FromMinutes(30);
e.UseMessageRetry(r => r.Intervals(100, 200, 500, 800, 1000));
e.UseInMemoryOutbox();
e.ConfigureConsumeTopology = false;
});
});
});
}
I can see that the message is being sent correctly, as its shown inside the subscription in Service Bus Explorer. However, the receiver is not picking it up? There are no errors or anything to go on? Really frustrating
Paul
You're calling ConfigureEndpoints, which will by default create receive endpoints for the consumers, sagas, etc. that have been added. However, your code sample doesn't show any .AddConsumer methods. If you don't have any consumers, don't call ConfigureEndpoints.
For your receiver, you should use:
public static void SetupMassTransit(this ServiceCollection services, string connectionString)
{
services.AddMassTransit(x =>
{
x.AddConsumer<MyMessageConsumer>();
x.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host(connectionString);
cfg.SubscriptionEndpoint("your-topic-name", "your-subscription-name", e =>
{
e.PrefetchCount = 100;
e.MaxConcurrentCalls = 100;
e.LockDuration = TimeSpan.FromMinutes(5);
e.MaxAutoRenewDuration = TimeSpan.FromMinutes(30);
e.UseMessageRetry(r => r.Intervals(100, 200, 500, 800, 1000));
e.UseInMemoryOutbox();
e.ConfigureConsumer<MyMessageConsumer>(context);
});
});
});
}
For your producer, you can simply use:
private static ServiceProvider SetupServiceCollection()
{
var connectionString = ConfigurationManager.AppSettings["AzureServiceBusConnectionString"];
var services = new ServiceCollection()
.AddMassTransit(x =>
{
x.UsingAzureServiceBus((context, cfg) =>
{
cfg.Host(connectionString);
});
});
return services.BuildServiceProvider();
}
Then, to publish, using your IServiceProvider that was created above:
var bus = serviceProvider.GetRequiredService<IBus>();
var endpoint = await bus.GetSendEndpoint(new Uri("topic:your-topic-name"));
await endpoint.Send(new MyMessage());
That should do the absolute minimum required that you need.
I am trying to send a lot of https requests and processing that causes my code to crash. I know I can increase my memory but that won't scale. Uncommenting the code below causes OOM crash at some point. The solution should probably be to flush the buffer or something, but I am learning nodejs so not sure what to do.
var https = require('https');
// url anonymized for example
var urlArray = ["https://example.com/blah", ....] // 5000 urls here
var options = {
headers: { "x-api-key": "mykey" }
};
for (let dest of urlArray) {
https.request(dest, options, (res) => {
if (res.statusCode != 200) {
console.log(res.statusCode+" "+res.statusMessage+" at "+dest)
}
})
// uncommenting below causes a crash during runtime
// .on("error", (err) =>
// console.log(err.stack))
.end();
}
NodeJs being non-blocking, is not waiting for the ith http.request to finish before it moves to the i+1th. So the request keeps on accumulating in the memory, and as the memory is not big enough, it crashes. So what we can do here is execute the requests in batches and wait for that batch to finish before starting with the next batch. With this, at any instant, there will be at most n requests present in the memory (n is the batch size).
The code will look something like this:
const request = require('request-promise');
const urlArray = ["https://example.com/blah", ....];
async function batchProcess (urlArray){
const batchSize = 100;
// ^^^^^^^^^^
let i=0;
while(i<urlArray.length) {
let batch = [];
for(let j=0; j<batchSize && i<urlArray.length; j++, i++){
batch.push(request({
uri: urlArray[i],
headers: {
'User-Agent': 'Request-Promise',
"x-api-key": "mykey"
},
json: true // Automatically parses the JSON string in the response
}));
}
let batchResult = await Promise.all(batch);
console.log(batchResult);
}
}
batchProcess(urlArray);
One way would be to turn them into an async iterable where you can run them after another and process them as they return (Apologies for the TypeScript, just pass them through playground to transpile if you don't know TS):
import fetch from "node-fetch";
class MyParallelCalls implements AsyncIterable<any> {
constructor(private readonly urls: string[]) {}
[Symbol.asyncIterator](): AsyncIterator<any> {
return this.iterator();
}
async *iterator(): AsyncGenerator<any> {
for (const url of this.urls) {
yield (await fetch(url, {headers: { "x-api-key": "mykey" }})).json();
}
}
}
async function processAll() {
const calls = new MyParallelCalls(urls);
for await (const call of calls) {
// deal with them as the happen: e.g. pipe them into a process or a destination
console.log(call);
}
}
processAll();
If you want I can modify the above to batch your calls too. It's easy just add an option to the constructor for batch size and you can set how many calls you want to do in your batch and use promise.all for doing the yield.
It will look something like below (Refactored a little so its more generic):
import fetch from "node-fetch";
interface MyParallelCallsOptions {
urls: string[];
batchSize: number;
requestInit: RequestInit;
}
class MyParallelCalls<T> implements AsyncIterable<T[]> {
private batchSize = this.options.batchSize;
private currentIndex = 0;
constructor(private readonly options: MyParallelCallsOptions) {}
[Symbol.asyncIterator](): AsyncIterator<T[]> {
return this.iterator();
}
private getBatch(): string[] {
const batch = this.options.urls.slice(this.currentIndex, this.currentIndex + this.batchSize);
this.setNextBatch();
return batch;
}
private setNextBatch(): void {
this.currentIndex = this.currentIndex + this.batchSize;
}
private isLastBatch(): boolean {
return this.currentIndex === this.options.urls.length;
}
async *iterator(): AsyncGenerator<T[]> {
while (!this.isLastBatch()) {
const batch = this.getBatch();
const requests = batch.map(async (url) => (await fetch(url, this.options.requestInit)).json());
yield Promise.all(requests);
}
}
}
async function processAll() {
const batches = new MyParallelCalls<any>({
urls, // string array of your urls
batchSize: 5,
requestInit: { headers: { "x-api-key": "mykey" } }
});
for await (const batch of batches) {
console.log(batch);
}
}
processAll();
I have an API in NestJs which is not sending data on the first hit. However, on hitting it again it sends the desired data. I am guessing the API returns before the internal processing is done.
How to stop this. Is sleep a good option for this?
Or is there any other way to do this?
#Post("load")
#UseGuards(AuthGuard("jwt"))
async load(#Req() body: any)
{
const organizationId = body.user.organizationId;
const userId = body.user.userId;
if ("brandIds" in body.body)
{
await this.userService.onBoardUser(userId);
}
var settings = await this.settingsService.fetchLayout(organizationId, "home");
settings.forEach(async (element) =>
{
var parsedElement = JSON.parse(JSON.stringify(element));
var innerContent = await this.fetchContent(parsedElement.method, organizationId, userId);
var template = parsedElement.content[0];
let formattedItem = {};
innerContent.forEach((item) =>
{
try
{
formattedItem = template;
Object.keys(template).forEach((key) =>
{
if (template[key]!= "" && key != "type")
{
formattedItem[key] = eval(template[key]);
}
});
parsedElement.content.push(formattedItem);
formattedItem = null;
}
catch(err)
{
}
});
this.response.data.push(parsedElement);
innerContent = null;
template = null;
formattedItem = null;
parsedElement = null;
});
return(this.response);
}
looks like your main problem here is that your using async/await inside foreach which isnt working.
Use it like this:
for (const setting of settings) {
... your async code here.
}
I've been trying to convert the functions in this https://blog.jeremylikness.com/build-a-serverless-link-shortener-with-analytics-faster-than-finishing-your-latte-8c094bb1df2c to a WebAPI equivalent. This is my webapi call:
[HttpPost]
public async Task<IActionResult> PostAsync([FromBody] ShortRequest shortRequest)
{
_logger.LogInformation($"ShrinkUrl api called with req: {shortRequest}");
if(!Request.IsHttps && !Request.Host.Host.Contains("localhost"))
return StatusCode(StatusCodes.Status400BadRequest);
if (string.IsNullOrEmpty(shortRequest.Input))
return StatusCode(StatusCodes.Status404NotFound);
try
{
var result = new List<ShortResponse>();
var analytics = new Analytics();
// determine whether or not to process analytics tags
bool tagMediums = analytics.Validate(shortRequest);
var campaign = string.IsNullOrWhiteSpace(shortRequest.Campaign) ? DefaultCampaign : shortRequest.Campaign;
var url = shortRequest.Input.Trim();
var utm = analytics.TagUtm(shortRequest);
var wt = analytics.TagWt(shortRequest);
_logger.LogInformation($"URL: {url} Tag UTM? {utm} Tag WebTrends? {wt}");
// get host for building short URL
var host = Request.Scheme + "://" + Request.Host;
await _tableOut.CreateIfNotExistsAsync();
if (_keyTable == null)
{
_logger.LogInformation($"Keytable is null, creating initial partition key of 1");
_keyTable = new NextId
{
PartitionKey = "1",
RowKey = "KEY",
Id = 1024
};
var keyAdd = TableOperation.Insert(_keyTable);
await _tableOut.ExecuteAsync(keyAdd);
}
// strategy for getting a new code
string getCode() => Utility.Encode(_keyTable.Id++);
// strategy for logging
void logFn(string msg) => _logger.LogInformation(msg);
// strategy to save the key
async Task saveKeyAsync()
{
var operation = TableOperation.Replace(_keyTable);
await _tableOut.ExecuteAsync(operation);
}
// strategy to insert the new short url entry
async Task saveEntryAsync(TableEntity entry)
{
var operation = TableOperation.Insert(entry);
await _tableOut.ExecuteAsync(operation);
}
// strategy to create a new URL and track the dependencies
async Task saveWithTelemetryAsync(TableEntity entry)
{
await TrackDependencyAsync(
"AzureTableStorageInsert",
"Insert",
async () => await saveEntryAsync(entry),
() => true);
await TrackDependencyAsync(
"AzureTableStorageUpdate",
"Update",
async () => await saveKeyAsync(),
() => true);
}
if (tagMediums)
{
// this will result in multiple entries depending on the number of
// mediums passed in
result.AddRange(await analytics.BuildAsync(
shortRequest,
Source,
host,
getCode,
saveWithTelemetryAsync,
logFn,
HttpUtility.ParseQueryString));
}
else
{
// no tagging, just pass-through the URL
result.Add(await Utility.SaveUrlAsync(
url,
null,
host,
getCode,
logFn,
saveWithTelemetryAsync));
}
_logger.LogInformation($"Done.");
//return req.CreateResponse(HttpStatusCode.OK, result);
}
catch (Exception ex)
{
_logger.LogError("An unexpected error was encountered.", ex);
//return req.CreateErrorResponse(HttpStatusCode.BadRequest, ex);
}
return null;
}
And this is the function params:
[FunctionName("ShortenUrl")]
public static async Task<HttpResponseMessage>([HttpTrigger(AuthorizationLevel.Function, "post")],HttpRequestMessage req,
[Table(Utility.TABLE, "1", Utility.KEY, Take = 1)]NextId keyTable,
[Table(Utility.TABLE)]CloudTable, TraceWriter log)
The azure function takes care of ensuring that the keyTable contains the next Id in the counter but I cannot figure out how to do the same in the webapi call.
Any ideas?