Azure Function App in PowerShell generating host threshold exceeded errors - azure

We just started getting this error yesterday but haven't changed anything in our app. Any ideas? If we restart the function app, it will run for a short time and then start giving us this error again. The function app is in PowerShell.
Host Error: Microsoft.Azure.WebJobs.Script: Host thresholds exceeded: [Connections]

This was recently added in the runtime to track running out of available connections on the VM
https://github.com/Azure/azure-webjobs-sdk-script/pull/2063
It should point to you running out of available connections. It could be wrong or correct, but I can't tell without looking at your functions themselves.
If you would like to discuss it, then you probably should use the repo above

I'm not sure if it is the above change that caused this but it is a coding issue on my side that is now being handled correctly by Azure Functions. I created this small repo and after I commented out the close, I received the error. My real code is more complex but clearly somewhere I'm not closing it out.
$Ports = #(21,22,23,53,69,71,80,98,110,139,111,389,443,445,1080,1433,2001,2049,3001,3128,5222,6667,6868,7777,7878,8080,1521,3306,3389,5801,5900,5555,5901)
for($i = 1; $i -le $ports.Count;$i++) {
$port = $Ports[($i-1)]
$client = New-Object System.Net.Sockets.TcpClient
$beginConnect = $client.BeginConnect("123.123.123.123",$port,$null,$null)
#$client.Close();
}

Related

Azure Functions - PowerShell - "The pwsh executable cannot be found at ..."

When my PowerShell Azure Function runs using the Test/Run feature in the portal, I get this error in the connected console output.
The pwsh executable cannot be found at "C:\Program Files (x86)\SiteExtensions\Functions\3.3.1\workers\powershell\7\runtimes\win\lib\netcoreapp3.1\pwsh.exe"
Note that 'Start-Job' is not supported by design in scenarios where PowerShell is being hosted in other applications. Instead, usage of the 'ThreadJob' module is recommended in such scenarios
My script looks something like below.
Note the invocation of the web request does indeed fail with an HTTP 500, triggering, I assume the catch block and the if.
try {
Invoke-WebRequest ...
}
catch {
$exc = $_;
}
if ($null -ne $exc) {
Writing-Warning "This failed when something blah.";
throw $exc;
}
This is the gyst. The real script actually makes a few web requests, any of which could fail. I want to ensure they all get executed, so I catch and then store the exception, and then only at the end the script can throw and fail, and my hope is at least one of the problems makes it out into logging or somewhere in the portal or something.
The actual message looks like this. It smells like an Azure problem to me.
I just ran it again and it fixed itself. Thanks for wasting my time, Azure.

Why is my webjob terminating without throwing an exception?

My azure webjob appears to be terminating without throwing an exception and I'm lost.
My web job is run on-demand (or scheduled) and has a dependency on my web site DLL (and MVC app). It calls into it to do most of the work, which includes working with an entity frameworks database and making REST calls to several other sites. Most of the work is done asynchronously. Most of the code used to do this work is also called from other parts of the site without problem, and it goes without saying that the web job works flawlessly when run locally.
The web job terminates and doesn't seem to throw an exception when it does and it doesn't seem to be possible to debug a web that's not of the continuously run variety (?). Therefor, my debugging has mostly been of the Console.WriteLine variety. Because of that and the asynchronisity, I haven't been able to nail down exactly where it's crashing - I thought it was while accessing the database, but after mucking with it, the database access started working.. ugh. My next best guess it that it dies during an await or other async plumbing. It does, however, crash within two try/catch blocks that have finallys that log results to redis and azure storage. None of that happens. I can not figure out, or imagine, how this process is crashing without hitting any exception handlers.. ?
Anyone had this problem with an azure webjob? Any idea what I should be looking for or any tips for debugging this?
Thanks!
I figured it out! One of the many things happening asynchronously was the creation of a certificate. I traced it down to this:
signedCert = new X509Certificate2(cert, "notasecret", X509KeyStorageFlags.Exportable);
This code works fine when called from my azure website or my tests, but kills the webjob process completely without throwing an exception! For example, the WriteLine in the exception handler below never gets called:
X509Certificate2 signedCert;
try
{
signedCert = new X509Certificate2(cert, "notasecret", X509KeyStorageFlags.Exportable);
}
catch (Exception ex)
{
// We never get here! Argh!
Console.WriteLine("Exception converting cert: " + ex);
throw;
}
Extremely time consuming and frustrating. Unlike the diagnosis, the fix is simple:
signedCert = new X509Certificate2(
cert,
"notasecret",
X509KeyStorageFlags.Exportable |
X509KeyStorageFlags.MachineKeySet |
X509KeyStorageFlags.PersistKeySet);

Continuous Execution of commands in Linux using Ganymed SSH

I am using Ganymed SSH to execute commands in my Linux Machine from Java application. I have to execute 100 commands one by one.
For this in my code under for loop I am opening and closing the session for each iteration. But it is throwing some exception after 10 or 15 iterations saying that Unable to establish the connection.
I believe this is happening because the sessions are not creating frequently.
Please check the below code and guide me some good solution for achieve this.
for(String user : usersList)
{
session = connection.openSession();
session.execCommand("[ -f /home/"+user+"/.file_name] && echo \"Found\" || echo \"Not Found\"");
session.close();
}
Thanks & Regards,
Gupta Katakam
If you try to do this, you will end up in the situation that you are in. SSH has many configuration parameter but that you need to specify in the end machine. I am not sure if you have access to the end machine or not, but in case you do not have access to the target machine, please consider adding some delay on you code after doing a disconnect and re-connect. I was also facing the same problem although I was using a different library. Let me know if this things works or not, and then we can take it forward from there.

Windows Azure Mobiles Services scheduler execution timeout

I am using a Mobile Services in Windows Azure. I use the new Scheduler available for a Mobile Service. The Scheduler I called SendOut.
I am running a pretty simple script that will insert a message to a queue. The entire script:
function SendOut() {
var azure = require('azure');
var queueService = azure.createQueueService("mailsoutscheduler", "[The key to the storage]");
queueService.createQueueIfNotExists("mailsout", function(error){ });
queueService.createMessage("mailsout", "SendOut", function(error){});
}
It works fine when I try to run the script once. It it scheduled to run every 5 minutes. And it usually goes fine. However sometimes I receive this error:
An unhandled exception occurred. Error: One of your scripts caused the
service to become unresponsive and the service was restarted. This is
commonly caused by a script executing an infinite loop or a long,
blocking operation. The service was restarted after the script
continuously executed for longer than 1000 milliseconds.
at EventEmitter. (C:\DWASFiles\Sites\VogSendOut\VirtualDirectory0\site\wwwroot\runtime\server.js:84:17)
at EventEmitter.emit (events.js:88:20)
I cannot figure out why I get this error - or how to solve it.
Could it be because it's running in the FREE Mobile Service Tier?
I don't think it's due to the FREE mobile subscription.
try to add a
try{}
catch{} block
and use console.log() to log if an error occured. It could help you to resolve you problem.

SharePoint 2013 :- Web Application taking forever to create

I have a SharePoint 2013 installation on a Window 8 machine.
I am trying to create a web application and it is taking forever. The creation process never stops. I checked in application event logs and found this error:
*Machine 'SHAREPOINT2013C (SharePoint - 43000(_LM_W3SVC_1458308317_ROOT))' failed ping validation and has been unavailable since '1/22/2013 3:56:48 AM'.*
Searched the web but could not find anything that works for me.
Can anyone suggest a way to resolve the issue? Thanks a lot in advance.
Below are my findings:
In order to recognize routing targets IIS has to be able to process SPPING HTTP method
To test run this code in Powershell:
$url = "http://your-Routing-Target-Server-Name"
$myReq = [System.Net.HttpWebRequest]::Create($url)
$myReq.Method = "SPPING";
$response = $myReq.GetResponse();
$response.StatusCode
If you get the following error message:
Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (405) Method Not Allowed."
that means that web front end is not set up to process SPPING HTTP method
To resolve the issue run the following commands on each routing target server:
Import-Module WebAdministration
add-WebConfiguration /system.webserver/handlers "IIS:\" -value #{
name = "SPPINGVerbHandler"
verb = "SPPING"
path = "*"
modules = "ProtocolSupportModule"
requireAccess = "None"
}
This will add a handler for SPPING verb to IIS configuration.
Run the test script again to make sure this works.
So this has to do with the Request Management Service that runs on the WFE servers on SharePoint 2013. The Request Management Service is of no value since you only have one server. If you disable this service on your single server farm these messages will go away and your Web Application creation performance will greatly increase.
Mark Ringo
I recently faced this issue, I created new Web Application and it was showing a popup of "It shouldn't take long", then after some time it showed a Connection failure page. I browsed to the virtual directory folder for the new web application and found that the folder was totally empty.
Then what I did to solve this problem:
1. Open IIS
2. Go to Applicatin Pools
3. Select Central Admin application pool and right click and select "Advance Settings".
4. There was a property named "Shutdown Time Limit", it was set to "90" by default. I changed it to 400 and clicked OK.
It restarted the applicaition pool automatically. Then again I created new web application from central admin and it worked for me.
I've found that these events correlate to when the specified application pools are recycled (mine are at a specific time in the morning). It's unfortunate that they're logged in the event viewer and can't really clean it up.

Resources