Telegraf daemon not receiving metrics from app deployed with Dokku - node.js

I've already monitored NodeJS apps in the past with Telegraf+InfluxDB+Grafana, but it's the first time I try to monitor a Dokku app, without success so far.
I have the following setup
app server monitoring serveur
- telegraf daemon (listening on udp 8125) ------> InfluxDB + Grafana
- dokku
- myapp : sending metrics on localhost:udp:8125
I have the issue that metrics sent from myapp on udp://localhost:8125 are not received by telegraf.
The connectivity from telegraf to influxdb and grafana is ok, because I ran the following command on the app server:
echo "foobar:1|c" | nc -u -w0 127.0.0.1 8125
And I could visualize inside Grafana the foobar counter increment.
So I'm guessing the issue is due to myapp or how I configured Dokku.
In myapp (a node JS app), I'm using the hot-shots package to send metrics (also tried with lynx without success).
This is the code I use to instanciate the statsd client.
var StatsD = require('hot-shots');
var client = new StatsD();
// Attaching an error handler to client's socket
client.socket.on('error', function(error) {
console.error('StatsD client error in socket: ', error);
});
I increment a counter on app startup to signal a restart. This is done with the following code:
metrics.increment('server_restart', function(err, bytes) {
if (err) {
console.log(err);
} else {
console.log('StatsD sent : ', bytes);
}
});
Inside the app server, I ran dokku logs myapp after performing a deployment.
I get the following output
$ dokku logs myapp
2017-05-06T14:51:19.977938845Z app[web.1]:
2017-05-06T14:51:19.977996938Z app[web.1]: > myapp#1.0.0 start /app
2017-05-06T14:51:19.978002950Z app[web.1]: > node .
2017-05-06T14:51:19.978005766Z app[web.1]:
2017-05-06T14:51:21.849086537Z app[web.1]: StatsD sent : 18
$
So it seems that the metrics payload is properly sent on the right port, but it never reaches telegraf.
Is there any configuration to do in Dokku to authorize external UDP requests to localhost ?
What can I do to debug this further ?

As the author requested in another thread the solution seems to be:
By default, can a docker container call host's localhost UDP?
Opening ports is only needed when you want to Listen for the requests not sending. By default Docker provides the necessary network namespace for your container to communicate to the host or outside world.
So, you could do it in two ways:
use --net host in your docker run and send requests to localhost:8125 in this case you containerized app is effectively sharing the host's network stack. So localhost points to the daemon that's already running in your host.
talk to the container network gateway (which is usually 172.17.0.1) or your host's hostname from your container. Then your are able to send packets to your daemon in your host.

Related

Azure container App fails to start with Julia image

I am trying to run Julia code in azure container app, but it fails for some reason.
docker file is pretty simple:
FROM julia:1.7-bullseye
COPY . /
EXPOSE 8080
RUN julia --project='/' -e 'using Pkg; Pkg.build()'
ENTRYPOINT julia --project='/' main.jl "0.0.0.0" "8080"
"0.0.0.0" and "8080" there is the host and port on which HTTP is served:
HTTP.serve(GENERAL_ROUTER, host, port_restful_api)
The app supposed to run a simple HTTP server that is served over port 8080, GET to / is the health check that returns simple "Up"
When I run that same image locally with docker run -p 8080:8080 my-test:latest it works fine and hitting localhost:8080 gives me my expected response
But when I have it in container app - revision fails to be provisioned (and no extra errors provided). In the ingress I added availability to accept traffic from anywhere and target port is 8080...
What I am missing?
UPDATE
At the same time in log analytics I can see my output to console that should indicate that the app is supposedly running fine, e.g.
[ Info: API handler service: Running on thread 1
In Julia those are done with
#info "API Server: Running on thread $(Threads.threadid())"
But in logs those are in stderr stream. Could it be that the ACA treats any output to stderr as startup error?
UPDATE 2:
The very same code works without any issues or changes on Azure Container Instance and in Azure Kubernetes Service...
So for now I stopped investigating and decided to use ACI.
You should verify if any process inside the julia image needs root privileges as ACA do not support running privileged containers based on the docs:
Azure Container Apps can't run privileged containers. If your program
attempts to run a process that requires root access, the application
inside the container experiences a runtime error.
Did you specify 8080 as your target port when configuring the HTTP ingress of your Container App? https://learn.microsoft.com/en-us/azure/container-apps/ingress?tabs=bash
The platform will try to ping your app at the specified port due to the default health probes. If it receives no response, it will restart your app: https://learn.microsoft.com/en-us/azure/container-apps/health-probes?tabs=arm-template

Dapr: Failed to proxy request: proxy not initialized. daprd startup may be incomplete

I,m performing some tests with Dapr and Service Invocation using gRPC and have run in to an issue. I'm runnig Dapr in standalone mode and starts two applications like this:
dapr run --app-id App1 --app-port 10078 --dapr-grpc-port 11078 .\App1.exe
dapr run --app-id App2 --app-port 10114 --dapr-grpc-port 11114 .\App2.exe
When App2 tries to make a gRPC call to localhost:11114 to reach App1 i get an error
'Grpc.Core.RpcException: Status(StatusCode="Unknown", Detail="failed to proxy request: proxy not initialized. daprd startup may be incomplete.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"#1650461046.340000000","description":"Error received from peer ipv4:10.16.100.25:11114","file":"..\..\..\src\core\lib\surface\call.cc","file_line":904,"grpc_message":"failed to proxy request: proxy not initialized. daprd startup may be incomplete.","grpc_status":2}")
App1 hosts a gRPC service on port 10078 and App2 host a gRPC service on port 10114. Both applications show up in the dashboard and the ports look correct.
When the app starts the log says "API gRPC server is running on port 11114" for App2 which seams correct. How can i diagnose this further?

Azure web app for container - failed during startup - didn't respond to HTTP pings

I'm running into an annoying issue where my containerized app runs fine locally, but fails when deployed to azure, despite the ports being exposed properly.
The details:
azure web app for container with Linux OS on a Basic B1 machine.
nodejs v12 LTS with express server listening on HTTPS port 443. Includes an HTTP 404 (Not Found) error handler.
database db Mongodb Atlas on M0 Sandbox (free tier). Local and Azure IPs whitelisted.
docker container using Dockerfile. no Kubernetes, no yml config file.
docker image hosted on private Docker Hub repo, pulled by azure on build
Running locally on Windows 10 x64:
without docker:
node app.js
runs fine via https://localhost:443
with docker:
docker build -t myapp:latest . then
docker run -p 49160:443 -d myapp:latest
runs fine via https://localhost:49160
In both cases I get the expected output from my nodejs app:
Running in production on x64
Express HTTPS server listening on port 443
mongo :: connected to database mydb at mydb-tot7b.azure.mongodb.net
(obvously, myapp and mydb are not the actual names. Used here for convenience.)
Running in azure:
Under Configuration -> Application settings:
PORT=443
WEBSITES_PORT=443
DOCKER_ENABLE_CI=true
Pipeline set up pulling docker image myrepo/myapp:latest from private Docker Hub repo on build. Image is pulled correctly.
Fails via https://myapp.azurewebsites.net/ with error 503 Service Temporarily Unavailable.
Azure log:
2020-04-10 18:02:21.040 INFO - Pulling image: myrepo/myapp:latest
2020-04-10 18:02:22.128 INFO - latest Pulling from myrepo/myapp
2020-04-10 18:02:22.129 INFO - Digest: sha256:982[...]
2020-04-10 18:02:22.129 INFO - Status: Image is up to date for myrepo/myapp:latest
2020-04-10 18:02:22.131 INFO - Pull Image successful, Time taken: 0 Minutes and 1 Seconds
2020-04-10 18:02:22.143 INFO - Starting container for site
2020-04-10 18:02:22.144 INFO - docker run -d -p 7909:443 --name myapp_0_138f197c -e PORT=80 -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITES_PORT=443 -e WEBSITE_SITE_NAME=myapp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=myapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=6fc3[...] myrepo/myapp:latest
2020-04-10 18:02:22.144 INFO - Logging is not enabled for this container.Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-04-10 18:02:23.473 INFO - Initiating warmup request to container myapp_0_138f197c for site myapp
2020-04-10 18:02:45.304 INFO - Waiting for response to warmup request for container myapp_0_138f197c. Elapsed time = 21.8308149 sec
[...]
2020-04-10 18:06:08.252 INFO - Waiting for response to warmup request for container myapp_0_138f197c. Elapsed time = 224.7791547 sec
2020-04-10T18:02:24.893958779Z Running in production on x64
2020-04-10T18:02:26.275376640Z Express HTTPS server listening on port 80
2020-04-10T18:02:26.601261213Z mongo :: connected to database mydb at mydb-tot7b.azure.mongodb.net
2020-04-10 18:06:14.346 ERROR - Container myapp_0_138f197c for site myapp did not start within expected time limit. Elapsed time = 230.8732099 sec
2020-04-10 18:06:14.365 ERROR - Container myapp_0_138f197c didn't respond to HTTP pings on port: 443, failing site start. See container logs for debugging.
2020-04-10 18:06:14.427 INFO - Stoping site myapp because it failed during startup.
Why the containerized app seems to start correctly, as seen in the azure log, but the container fails to respond to HTTP ping on 443, which is exposed both in dockerfile and in azure with PORT and WEBSITES_PORT?
Do I need to make my HTTPS server explicitly respond to HTTP PING and if so, how do I do it?
Dockerfile:
FROM node:12
WORKDIR /usr/src/myapp
COPY package*.json ./
ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}
RUN npm install
COPY . .
EXPOSE 443
CMD [ "node", "app.js" ]
This issue might be considered a duplicate, but I provide a lot more relevant information, compared to most of the issues below. Also, I spent significant time in troubleshooting and none of these relevant resources seemed to help:
Docker never runs on Azure - Waiting for response to warmup request for container
Azure Web App on Linux: "Error: Container didn't respond to HTTP pings on port: 8080" - when using: "start": "pm2 start server.js"
How do you expose port 3000 using an Azure Web App Container?
https://superuser.com/questions/1479521/i-cannot-create-a-node-js-web-app-to-be-deployed-on-azurer
https://learn.microsoft.com/en-us/archive/blogs/waws/things-you-should-know-web-apps-and-linux#troubleshootingr
https://learn.microsoft.com/en-us/azure/app-service/containers/app-service-linux-intro#troubleshootingr
https://learn.microsoft.com/en-us/azure/app-service/containers/configure-language-nodejsr
https://github.com/MicrosoftDocs/azure-docs/issues/46401r
https://github.com/MicrosoftDocs/azure-docs/issues/34451r
https://omgdebugging.com/2017/12/22/azure-web-app-for-container-failing-site-start/r
https://devops.stackexchange.com/questions/4543/how-to-debug-a-docker-on-web-app-for-containers-in-azure-which-does-not-startr
https://learn.microsoft.com/en-us/azure/app-service/containers/app-service-linux-faq#custom-containersr
Running a docker container in Azure web app: didn't respond to HTTP pings on port
Docker Container fails to start in an Azure App Service
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-troubleshootingr
https://serverfault.com/questions/1003418/azure-docker-app-error-site-did-not-start-within-expected-time-limit-and-cor
https://medium.com/#sergiibielskyi/azure-event-hub-listener-in-the-container-dd6f8d053814r
When enforcing HTTPS at the App Service level, TLS termination will happen and traffic will be routed to your container port so there's no need to enforce TLS from your container.
If your container listens on another port (ex: 8081), simply set the WEBSITES_PORT application setting to that port number. App Service will listen on port 80 and forward traffic to your container port.
My own app had been doing app.UseHttpsRedirection() in Startup.Configure. That was the cause of this error for me. As soon as I stopped doing that, the problem went away.
I think I am also facing similar issue and in my case I can't changed the source code for application to run on http.
We have a very basic requirement, wherein there is an API build on ASP.NET Core running on specific port (say 11000), and served only on HTTPS, i.e http traffic is disabled, and the API is rendered on custom domain, (on local machine it is achieved through host file entries).
Working on POC to deploy this API on Docker container on Azure App service. Running this setup on local system and it works perfectly fine.
Steps followed.
Compose build
Create Image
Tag this Image
Push this image to Azure Container Register
Set the correct values in Deployment Center
Add the Application Settings for WEBSITES_PORT and set its value to the port, 11000.
While starting the container in Azure it is throwing error as
28/06/2022 18:40:04.370 INFO - Site: XXXXXXXXXXXXXX-webapp - [
some-alpha-numeric-identifier] - Waiting for container to be ready.
Container Id: some-alpha-numeric-identifier
And the I know the reason for the error, while starting the container is connecting to http://127.0.0.1:11000 and the base application only listens on https, I need to know how to overcome this error, can we for the app service to connect to the base docker image only over https.

Does Azure Web App for containers support ssh access when running a multi-container app?

I'm running an Azure Web app (containers) with custom container images. I've followed the steps to enable ssh into a container image and it works great when I'm only running a single container. But when I run the app as a multi-container app (with docker-compose file) with more than one container image I get the error below. For additional context this is a small python web app that using nginx and redis hence the need for more than one container. Only one of my custom images has ssh enabled and running and exposing port 2222.
Is this even possible? If not then I'm not sure how feasible it is to run a web app multi-container if I have no way to access a container for support purposes.
az webapp remote-connection create -g GROUPNAME -n APPNAME -p 2222 --verbose
Configured default 'GROUPNAME' for arg resource_group_name
remote-connection is deprecated and moving to cli-core, use `webapp create-remote-connection`
Port 2222 is open
Creating a socket on port: 2222
Setting socket options
Binding to socket on local address and port
Finished initialization
Status response message: FAILURE:2222:Unable to connect to WebApp
WARNING - Remote debugging may not be setup properly. Reponse content: FAILURE:2222:Unable to connect to WebApp
SSH is available { username: root, password: Docker! }
Start your favorite client and connect to port 2222
I also tried the create-remote-connection command but got similar results.
az webapp create-remote-connection -n APPNAME -g GROUPNAME --verbose &
Error I receive is:
Auto-selecting port: 52661
Finished initialization
Status response message: FAILURE:2222:Unable to connect to WebApp
WARNING - Remote debugging may not be setup properly. Reponse content: FAILURE:2222:Unable to connect to WebApp
Connection is not ready yet, please wait
.
Status response message: FAILURE:2222:Unable to connect to WebApp
WARNING - Remote debugging may not be setup properly. Reponse content: FAILURE:2222:Unable to connect to WebApp
Looks like its not supported :(
see...
How to SSH in to different containers in Multi Container Azure App Service
and...
Support SSH to specific container in multi-container setup

After complete installation IPFS not working on Browser

i have installed IPFS on server and server on browser its not working
root#vultr:~/live_horce_chain# cd go-ipfs
root#vultr:~/live_horce_chain/go-ipfs# ipfs daemon
Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/202.182.100.59/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
when i enter url 202.182.100.59:5001 or localhost:5001 or 127.0.0.1:5001 or 127.0.0.8080 or 202.182.100.59:8080 but not working. what was going wrong??
it should show IPFS panel
The URL you are looking for is 202.182.100.59:5001/webui or localhost:5001/webui.

Resources