How to investigate Dapr service invocation 500 error - azure

When I use service invocation in my Dapr applications and when they are deployed to Azure Container Apps I get:
Dapr.Client.InvocationException: An exception occurred while invoking method: 'health-check' on app-id: 'customers'
---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 500 (Internal Server Error).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Dapr.Client.DaprClientGrpc.InvokeMethodAsync[TResponse](HttpRequestMessage request, CancellationToken cancellationToken)
Service invocation is working fine locally. How do I figure out where my problem lies?
I would think enabling the debug log level in Dapr might give some insight 🤷‍♂️.
[HttpGet, Route("health-check")]
public ActionResult HealthCheck()
{
logger.LogInformation("health check called");
return Ok(new HealthCheckResponse
{
Message = "I'm up 👍 - orders service"
});
}
[HttpGet, Route("health-check-service-invocation")]
public async Task<ActionResult> HealthCheckServiceInvocation()
{
try
{
logger.LogInformation("health check service invocation of the catalog service called");
return Ok(await daprClient.InvokeMethodAsync<HealthCheckResponse>(HttpMethod.Get, "customers", "health-check"));
}
catch (Exception ex)
{
logger.LogError(ex, "unable to talk to the customers service");
return StatusCode(StatusCodes.Status500InternalServerError);
}
}

Service to service Invocation can be affected by ssl
When Running Locally are you running with mtls Enabled ?
# enabling
dapr init --enable-mtls=true
dapr init -k --enable-mtls=true
# Disable
dapr init --enable-mtls=false
dapr init -k --enable-mtls=false
You could try locally with mtls,
when running your app locally, the command should include --app-ssl
dapr run ... --app-ssl ...

Related

Docker container cannot connect to azure via MSI

I have an azure function which has this code:
run = Run.get_context()
ws = Workspace.from_config('config.json',auth=MsiAuthentication()) if not hasattr(run, 'experiment') else run.experiment.workspace
This works when I test the function locally using func start
but when I build docker image for the azure function and run the docker, it gives following error:
Message: Failed to connect to MSI. Please make sure MSI is configured correctly.
Get Token request returned http error: 400, reason: Bad Request
InnerException None
ErrorResponse
{
"error": {
"message": "Failed to connect to MSI. Please make sure MSI is configured correctly.\nGet Token request returned http error: 400, reason: Bad Request"
}
}
The identity configuration for azure function has the required roles:
Or, is there any other authentication method that I should use?

Monitor AKS container/pod status from pod running in different cluster/context

I have applications running in different cluster, like below (say as example)
cluster1 - for scratch work
cluster2 - as staging env (pods running application )
cluster3 - as testing env (pods running application )
cluster4 - monitor app
All the AKS cluster are connected in the network, a monitoring app is running on cluster4, which has custom monitoring dashboard. I am trying to add the container/pod status of app running on cluster2 and cluster3 to that dashboard.
Is there a way to fetch the container/pod status from the app running in cluster4 either using java/REST API/shell.
I came across Kubernetes java client InclusterClientExample.
Is there any better approach for this situation to fetch the container/pod status of the application running on different cluster.
Another options was to enable Log Analytics API.
create a workspace and use REST API to fire query. below is at a high level of how to do it.
If AAD is setup, get the TENENT ID info.
Register App, create a Service principal. (provides application (client) id)
Within the SP, add a secret key. (note this down, since it won't be viewable later)
Add API Permission, select Log Analytics
Grant Delegate permission (Data.Read) and Application Permissiong (Data.Read) which was in my case.
In Log Analytics workspace, Access Control (IAM), add the Service principal and provide Reader access. ( workspace-name, resourceGroup-name, workspace-id, subscription-id to be noted)
Per the link.
First we need to fetch the access_token using the SP and Tenet id. Few of the screenshot in this link is old.
Using curl command:
Fetch token
curl -vX POST -d 'grant_type=client_credentials&client_id=[SP application(client) id]&client_secret=[Client secret created in SP]&resource=https://management.azure.com/' https://login.microsoftonline.com/[TENENT_ID]/oauth2/token
Fetch log info
curl -vX post -H "Authorization: Bearer [TOKEN-FROM-ABOVE]" -H "Content-Type: application/json" -H "Prefer: response-v1=true"-d #samplequery.json https://api/loganalytics.io/v1/subscriptions/[subscription-id-of-workspace]/resourceGroups/[Resource-group-name-of-workspace]/providers/Microsoft.OperationalInsights/workspaces/<workspace-name>/api/query?api-version=2020-08-01
Sample query file
{
"query": "Perf | where CounterName == 'Available MBytes' | summarize avg(CounterValue) by bin(TimeGenerated, 1h)",
"timespan": "PT12H"
}
}
output would be the result of the query.
With many other options, tried out camel-kubernetes component. This uses kubernetes-client managed by fabric8.
If using minikube, set and start the application.
use minikube dashboard or kubectl proxy --port=8080 so the cluster can be accessed from host machine.
add the camel-core and camel-kubernetes dependencies to the project
package com.learning.camel.examples.prog3;
import org.apache.camel.CamelContext;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.impl.DefaultCamelContext;
public class ListPodsInK8s {
public static void main(String[] args) throws Exception {
CamelContext context = new DefaultCamelContext();
try {
context.addRoutes(new RouterToAccessK8s());
context.start();
ProducerTemplate template = context.createProducerTemplate();
template.start();
template.sendBody("direct:input1","example");
Thread.sleep(3000); // sleep 3 seconds
}finally {
context.stop();
}
}
}
code that will route to fetch the data
package com.learning.camel.examples.prog3;
import java.util.List;
import org.apache.camel.builder.RouteBuilder;
import io.fabric8.kubernetes.api.model.Pod;
public class RouterToAccessK8s extends RouteBuilder{
public String host = "http://127.0.0.1:8080"; // use az command to get server url
public String authToken = ""; // fetch the token using az command in case AKS
public String certPath = "C:\\Users\\tim\\.minikube\\profiles\\minikube\\client.crt"; //minikube
public String certKey = "C:\\Users\\tim\\.minikube\\profiles\\minikube\\client.key"; // minikube
#Override
public void configure() throws Exception {
from("direct:input1")
.toF("kubernetes-pods://%s?clientCertFile=%s&clientKeyFile=%s&namespace=default&operation=listPods", host,certPath,certKey)
.log("pod size: ${body.size()}")
.process(exchange -> {
List<Pod> pods = exchange.getIn().getBody(List.class);
System.out.println("NameSpace | PodName | Status");
pods.stream()
.forEach(pod -> {System.out.println(pod.getMetadata().getNamespace()+ " | "+ pod.getMetadata().getName()+" | "+pod.getStatus().getPhase());});
})
.end();
}
}
output:
NameSpace | PodName | Status
default | ngnix | Succeeded
kube-system | coredns-74ff55c5b-26drq | Running
kube-system | etcd-minikube | Running
kube-system | kube-apiserver-minikube | Running
kube-system | kube-controller-manager-minikube | Running
kube-system | kube-proxy-b97ss | Running
kube-system | kube-scheduler-minikube | Running
kube-system | storage-provisioner | Running
kubernetes-dashboard | dashboard-metrics-scraper-c95fcf479-x4bgq | Running
For Azure, az aks get-crediential --file - can be used to fetch the master server url (host) and access token info.

How do I resolve Access Denied error when trying to deploy Echobot in Azure?

I had a operating bot that I tried to push an update to and got a failure response. I tried building and deploying in Kudu with no luck either. Just as a sanity check I also made a brand new echobot on Azure and tried to run the build and deploy commands in Kudu Console.
EDIT: Meant to mention I've seen a few other mentions of similar issues including:
Error - Access is denied - deployment to Azure App Services
https://github.com/projectkudu/kudu/issues/3177
https://medium.com/rare-crew/hot-issue-on-azure-and-deployment-of-apps-by-kudu-scripts-dotnet-sdk-v3-1-301-92d6e336756a
MSBUILD : error MSB1025:Unhandled exception. An internal failure occurred while running MSBuild.
System.ComponentModel.Win32Exception (5): Access is denied.
at System.Diagnostics.Process.set_PriorityClassCore(ProcessPriorityClass value)
at System.Diagnostics.Process.set_PriorityClass(ProcessPriorityClass value)
at Microsoft.Build.CommandLine.MSBuildApp.Execute(String[] commandLine)
System.ComponentModel.Win32Exception (5): Access is denied.
at System.Diagnostics.Process.set_PriorityClassCore(ProcessPriorityClass value)
at System.Diagnostics.Process.set_PriorityClass(ProcessPriorityClass value)
at Microsoft.Build.CommandLine.MSBuildApp.Execute(String[] commandLine)
at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
Failed exitCode=-532462766, command=dotnet restore "EchoBot.sln"
An error has occurred during web site deployment.
We arrived at an answer in this thread: Microsoft Help Link
For the default Echobot project generated by Azure you need a global.json file in "D:\home\site\wwwroot" with the following code. You can get to this folder by using the Kudu debug console.
{
"sdk": {
"version": "3.1.202"
}
}
The issue is recently introduced by latest dotnet sdk (2.1.515 and 3.1.301) versions. This impacts the projects with custom deployment script still using dotnet restore and publish to build.Could you please try the below workaround in deploy.cmd to fix it.
SET MSBUILD_PATH=%ProgramFiles(x86)%\MSBuild-16.4\MSBuild\Current\Bin\MSBuild.exe
call :ExecuteCmd "%MSBUILD_PATH%" -t:Restore "%DEPLOYMENT_SOURCE%\my-solution.sln"
call :ExecuteCmd "%MSBUILD_PATH%" -t:Publish "%DEPLOYMENT_SOURCE%\vstar-next\my-proj.csproj" -p:OutputPath="%DEPLOYMENT_TEMP%" -p:Configuration=Dev

After upgrading the Azure SDK version to 1.7.0, authentication code failed java.lang.NoClassDefFoundError:

After upgrading the Azure SDK version to 1.7.0, authentication code failed with "java.lang.NoClassDefFoundError: com/microsoft/azure/management/batchai/implementation/BatchAIManager"
My authentication code is
/**
* Default constructor.
*/
public Azure() {
try {
credentials = new ApplicationTokenCredentials(Config.AzureclientId_admin, Config.AzuretenantId_admin,
Config.Azuresecret_admin, AzureEnvironment.AZURE);
azure = com.microsoft.azure.management.Azure.authenticate(credentials)
.withSubscription(Config.AzuresubscriptionId_admin);
} catch (Exception ex) {
Log.Message("Unable to create the Azure object", LogLevel.ERROR);
ex.printStackTrace();
}
}
This looked to be maven issue. The dependency was downloaded and seen in the .m2 directory but was not loaded in Eclipse Maven dependency. I deleted azure directory from the .m2 and compiled it again which resolved the issue.

Deploying a ASP.Net Core web application via IIS

I am having trouble deploying a ASP.Net Core 2.0 web application via IIS. My application builds and runs through VS but I cannot get the IIS instance to work.
I get the following error message:
HTTP Error 502.5 - Process Failure
Common causes of this issue:
- The application process failed to start
- The application process started but then stopped
- The application process started but failed to listen on the configured port
None of the troubleshooting steps have worked. I do not think the problem is with the application itself as when I run the following through command line:
"C:\Program Files\dotnet\dotnet.exe" C:\inetpub\wwwroot\MyApp\MyApp.dll
The application runs and I can view it at http://localhost:5000/
I have been through all the steps at https://learn.microsoft.com/en-gb/aspnet/core/host-and-deploy/iis/index?tabs=aspnetcore1x#common-errors and still have not made any progress.
This is what my WebHostBuilder method looks like:
var host = new WebHostBuilder()
.UseKestrel()
.UseContentRoot(Directory.GetCurrentDirectory())
.UseIISIntegration()
.UseStartup<Startup>()
.UseApplicationInsights()
.Build();
host.Run();
and I use the following settings in my Startup.cs:
services.Configure<IISOptions>(options =>
{
options.AutomaticAuthentication = true;
});
That error can be for a number of reasons.
On you IIS server have you installed the .Net Core Runtime?
Is your application pool configured correctly?
Are your site bindings correct?
Check the install runtimes on your computer are the same you are trying to run on the project

Resources