I am using fluentd to collect data from cloudwatch (EKS metrics and objects) and sending it to splunk.
Observed 403 error in pod that collects the metrics from cloudwatch by fluentd and sending it to splunk.
Metrics Pod log:
[fluentd-monitor-agent] listening monitoring http server on http://0.0.0.0:24220/api/plugins
[fluentd-monitor-agent] start webrick HTTP server listening
[error] : #0 unexpected error error_class=Net::HTTPServerException error=“403 \ “Forbidden\””
Related
Kubernetes Version: 1.21
Spark Version: 3.0.0
I am using a container in a Kubernetes pod (client pod) to invoke Spark Submit which then starts a Driver pod. The client pod which did the Spark Submit starts to watch the Driver pod via LoggingPodStatusWatcherImpl. After approximately 1 hour, the client pod experiences 401 error
22/11/03 13:05:44 WARN WatchConnectionManager: Exec Failure: HTTP 401, Status: 401 - Unauthorized
java.net.ProtocolException: Expected HTTP 101 response but was '401 Unauthorized'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
22/11/03 13:05:46 INFO LoggingPodStatusWatcherImpl: Application status for spark-blahblahblah (phase: Running)
I think Spark on Kubernetes usually looks in /var/run/secrets/kubernetes.io/serviceaccount/token so I would get the warning below when starting the client pod.
22/11/03 13:13:13 WARN Config: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
However, since I provide another oauth token file via the conf below in Spark Submit command the client pod was able to connect to the Kubernetes API and start the Driver pod.
--conf spark.kubernetes.authenticate.submission.oauthTokenFile=/mytokendir/token
The token is provided to the client pod via projected volume (new to Kubernete versions 1.20+), the token expiration duration can be specified in the yaml manifest as shown below:
See this doc for reference on how this is implemented:
https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-bound-service-account-tokens
spec:
serviceAccountName: my-serviceaccount
volumes:
- name: token-vol
projected:
sources:
- serviceAccountToken:
expirationSeconds: 7200
path: token
containers:
-name: my-container
image: some-image
volumeMounts:
-name: token-vol
mountPath: /mytokendir
I then exec into the client pod to get the JWT token in /mytokendir and decoded it.
It showed valid for 2 hours; however, coming back to the original question, my client pod is still getting 401 error after 1 hour.
Sometimes I would get this error:
22/11/03 14:10:57 INFO LoggingPodStatusWatcherImpl: Application my-application with submission ID my-namespace:my-driver finished
22/11/03 14:10:57 INFO ShutdownHookManager: Shutdown hook called
22/11/03 14:10:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-blahblah
The connection to the server localhost:8080 was refused - did you specify the right host or port?
(index):116 Uncaught DOMException: Failed to execute 'send' on 'WebSocket': Still in CONNECTING state.
at HTMLFormElement. (http://apppanel.fitnfurious.in/chatm/m/:116:8)enter image description here
a few days ago i moved my frontend with NextJS to Vercel and i started experiencing this following intermittent error during build and production server. So i searched all over the internet but i cannot find the solution.
I am receiving the following error when my App tries to make HTTPS request:
Compiled successfully.
09:47:13.026 Automatically optimizing pages...
09:47:15.212 Error at BlogService.getRecentPosts, details: Error: request to https://**masked** failed, reason: write EPROTO 140256911292224:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1550:SSL alert number 80
09:47:16.629 Error at getStaticProps on Homepage, details: Error: write EPROTO 140657767638848:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1550:SSL alert number 80
09:47:16.654 Error at BlogService.getRecentPosts, details: Error: request to https:/**masked** failed, reason: write EPROTO 139955108906816:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1550:SSL alert number 80
The backend destination is using a Nginx with Reverse Proxy with LetsEncrypt Certificate
So anybody can help me? I never experienced this issue
UPDATE
I changed my backend to AWS Api Gateway but the issue remains.
I already changed my frontend to just make Https requests using TLS 1.2, and the error remains intermittent
Anybody help me?
I am simply trying to setup an ODBC driver to Databricks Cluster.
According to the MS documentation
https://learn.microsoft.com/en-us/azure/databricks/kb/bi/jdbc-odbc-troubleshooting
If you get an TTransport exception using the curl command, you successfully reached and authenticated.
When I run...
curl https://adb-77180857967XXXXX.6.azuredatabricks.net:443/sql/protocolv1/o/7718085796704186/0910-172424-pizza885 -H "Authorization: Bearer XXXXX"
It does produce the error which indicates success...
Error 500 Server Error
HTTP ERROR 500
<p>Problem accessing /cliservice. Reason:
<pre> Server Error</pre></p><h3>Caused by:</h3><pre>javax.servlet.ServletException: org.apache.thrift.transport.TTransportException
When I test the connection from the ODBC driver I get the following error:
FAILED!
[Simba][ThriftExtension] (14) Unexpected response from server during a HTTP connection: Could not resolve host for client socket..
I have deployed a Bitnami Parse stack to a Google Cloud VM. This stack is basically Apache running as a proxy, Then a MEAN stack handling parse server and dashboard etc.
Everything appears to be working fine when accessing the dashboard and sending individual CURL requests to test it's all available.
However, there appears to be a fatal flaw in that. When Parse returns one of it's errors e.g.
{"code":101,"error":"Object not found."}
Express sees it as an uncaught exception and dies. Now I realise that it's intentional behaviour (There's enough information out there as to why you don't want to persist after an uncaught exception)
The Bitnami stack uses Forever to run the Parse Server, but as you can see from the logs I will include below, you still run into issues with refused connections.
2016-11-25T11:41:16+00:00 DEBUG (7): {"code":101,"error":"Object not found."}
2016-11-25T11:41:16+00:00 DEBUG (7): Failed to connect to X.X.X.X: Connection refused
2016-11-25T11:41:17+00:00 DEBUG (7): Failed to connect to X.X.X.X: Connection refused
2016-11-25T11:41:17+00:00 DEBUG (7): Failed to connect to X.X.X.X: Connection refused
2016-11-25T11:41:17+00:00 DEBUG (7): Failed to connect to X.X.X.X: Connection refused
Meanwhile in the Forever logs:
Error generating response. ParseError { code: 101, message: 'Object not found.' } code=101, message=Object not found.
[object Object]
/opt/bitnami/apps/parse/htdocs/node_modules/parse-server/lib/ParseServer.js:425
throw err;
^
Error: EACCES: permission denied, open '/opt/bitnami/apps/parse/htdocs/logs/parse-server.info.2016-11-25'
at Error (native)
error: Forever detected script exited with code: 7
error: Script restart attempt #1
parse-server running on port 1337
parse-dashboard running on port 4040
So you can see that immediately following a Parse error being returned, The service is unavailable until Forever can restart the Parse Server.
This issue appears that it would affect all users as well, so if one user receives an exception, all users lose connection for a brief period.
So the question is, how should this be properly handled? Is Parse + Express not the way to go about it? Or is there some method of configuring Node or Express to return the API error to the requester, without Express dying (And without having to fall into the trap of using uncaughtException to keep Node alive)
Any help would be much appreciated, thank you
The logs from Forever said it's EACCES error, that means ParseServer couldn't access the log file and not able to write the info log in it.
Error: EACCES: permission denied, open '/opt/bitnami/apps/parse/htdocs/logs/parse-server.info.2016-11-25'
at Error (native)
error: Forever detected script exited with code: 7
error: Script restart attempt #1
parse-server running on port 1337
parse-dashboard running on port 4040
I think you have to change the /opt/bitnami/apps/parse/htdocs/logs with proper permission so that ParseServer can write log files in it.