I got the following error:
controller.go:228] unable to sync kubernetes service: Post "https://[::1]:6443/api/v1/namespaces": dial tcp [::1]:6443: connect: cannot assign requested address
I have the following warnings in my cluster kube (3x3 master/workers on prem (kvm)) with 3 etcd on masters.
kubectl get events --field-selector type!=Normal -n kube-system
LAST SEEN TYPE REASON OBJECT MESSAGE
3m25s Warning Unhealthy pod/kube-apiserver-kube-master-1 Readiness probe failed: HTTP probe failed with statuscode: 500
3m24s Warning Unhealthy pod/kube-apiserver-kube-master-2 Readiness probe failed: HTTP probe failed with statuscode: 500
3m25s Warning Unhealthy pod/kube-apiserver-kube-master-2 Liveness probe failed: HTTP probe failed with statuscode: 500
3m27s Warning Unhealthy pod/kube-apiserver-kube-master-3 Readiness probe failed: HTTP probe failed with statuscode: 500
17m Warning Unhealthy pod/kube-apiserver-kube-master-3 Liveness probe failed: HTTP probe failed with statuscode: 500
This error not affect my cluster or my servicies in any form. It's appear from the begining. How do I solve? :D
Somewhere you are assigning [::1] address to endpoints..
The endpoint IPs must not be: loopback (127.0.0.0/8 for IPv4, ::1/128
for IPv6), or link-local (169.254.0.0/16 and 224.0.0.0/24 for IPv4,
fe80::/64 for IPv6).
[::1] Means 127.0.0.1 in ipv6 address.
I had the same Error.
My CoWorker deactivated IPv6 (to try something..) and Kubernetes tried to use IPv6.
After rebooting my Master, IPv6 came back and it worked again.
I searched for a bit and found this article: https://kubernetes.io/blog/2021/12/08/dual-stack-networking-ga/ which basically says you can set ipFamilyPolicy to one of three options:
SingleStack
PreferDualStack
RequireDualStack
Related
I don't have an access to the etcd part of the project's source code, however I do have access to the /var/log/syslog.
The goal is to setup up 3-node cluster.
(1)The very first etcd error that comes up is:
rafthttp: failed to dial 76e7ffhh20007a98 on stream MsgApp v2 (dial tcp 10.0.0.134:2380: i/o timeout)
Before continuing, I would say that I can ping all three nodes from each of the nodes. As well as I have tried to open the 2380 TCP ports and still no success - same error.
(2)So, before that error I had following messages from the etcd, which in my opinion confirm that cluster is setup correctly:
etcdserver/membership: added member 76e7ffhh20007a98 [https://server2:2380]
etcdserver/membership: added member 222e88db3803e816 [https://server1:2380]
etcdserver/membership: added member 999115e00e17123d [https://server3:2380]
In /etc/hosts file these DNS names are resolved as:
server2 10.0.0.135
server1 10.0.0.134
server3 10.0.0.136
(3)The initial setup, however (on each nodes looks like this):
embed: listening for peers on https://127.0.0.1:2380
embed: listening for client requests on 127.0.0.1:2379
So, to sum up, each node have got this initial setup log (3) and then adds members (2) then once these steps are done it fails with (1). As I know the etcd cluster creation is following this pattern: https://etcd.io/docs/v3.5/tutorials/how-to-setup-cluster/
Without knowing the source code is really hard to debug, however maybe some ideas on the error and what could cause it?
UPD: etcdctl cluster-health output (ETCDCTL_ENDPOINT is exported):
cluster may be unhealthy: failed to list members Error: client: etcd
cluster is unavailable or misconfigured; error #0: client: endpoint
http://127.0.0.1:2379 exceeded header timeout ; error #1: dial tcp
127.0.0.1:4001: connect: connection refused
error #0: client: endpoint http://127.0.0.1:2379 exceeded header
timeout error #1: dial tcp 127.0.0.1:4001: connect: connection refused
Am trying to bring Jfrog up, in local tomcat is running and artifactory service also looking fine. But in UI jfrog is not coming up.
Getting 502 Bad Gateway error. I have shared the console log details below.
Below is the console log
[TRACE] [Service registry ping] operation attempt #94 failed. retrying in 1s. current error: error while trying to connect to local router at address 'http://localhost:8046/access/api/v1/system/ping': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused
[TRACE] [Service registry ping] running retry attempt #95
[INFO ] Cluster join: Retry 95: Service registry ping failed, will retry. Error: error while trying to connect to local router at address 'http://localhost:8046/access/api/v1/system/ping': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused
[TRACE] [Service registry ping] operation attempt #95 failed. retrying in 1s. current error: error while trying to connect to local router at address 'http://localhost:8046/access/api/v1/system/ping': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused
2022-09-10T06:14:20.271Z [jffe ] [INFO ] [ ] [ ] [main ] - pinging artifactory, attempt number 90
2022-09-10T06:14:20.274Z [jffe ] [INFO ] [ ] [ ] [main ] - pinging artifactory attempt number 90 failed with code : ECONNREFUSED
[TRACE] [Service registry ping] running retry attempt #96
[DEBUG] Cluster join: Retry 96: Service registry ping failed, will retry. Error: error while trying to connect to local router at address 'http://localhost:8046/access/api/v1/system/ping': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused
[TRACE] [Service registry ping] operation attempt #96 failed. retrying in 1s. current error: error while trying to connect to local router at address 'http://localhost:8046/access/api/v1/system/ping': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused
2022-09-10T06:14:21.188Z [jfrou] [INFO ] [2b4bfed554e45cf6] [join_executor.go:169 ] [main ] [] - Cluster join: Retry 100: Service registry ping failed, will retry. Error: could not parse error from service registry, status code: 404, raw body: <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1></body></html>
[TRACE] [Service registry ping] running retry attempt #97
[DEBUG] Cluster join: Retry 97: Service registry ping failed, will retry. Error: error while trying to connect to local router at address 'http://localhost:8046/access/api/v1/system/ping': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused
[TRACE] [Service registry ping] operation attempt #97 failed. retrying in 1s. current error: error while trying to connect to local router at address 'http://localhost:8046/access/api/v1/system/ping': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused
2022-09-10T06:14:22.016Z [jfmd ] [INFO ] [ ] [accessclient.go:60 ] [main ] - Cluster join: Retry 100: Service registry ping failed, will retry. Error: Error while trying to connect to local router at address 'http://localhost:8046/access': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused [access_client]
[TRACE] [Service registry ping] running retry attempt #98
[DEBUG] Cluster join: Retry 98: Service registry ping failed, will retry. Error: error while trying to connect to local router at address 'http://localhost:8046/access/api/v1/system/ping': Get "http://localhost:8046/access/api/v1/system/ping": dial tcp 127.0.0.1:8046: connect: connection refused
and this is the error am getting in UI.
502 Bad Gateway error
Is it a newly installed Artifactory instance? If yes, we need to verify whether the required ports are in place (open at the firewall level). If the ports are already available, disable the IPv6 address from the VM where Artifactory is installed and restart the Artifactory. There are chances of this error occurrence if the application is trying to pick up the IPv6 address for initialisation instead of Ipv4.
I need to regularly and randomly test with Linux/C++/libcurl the responses of several servers that are available through a single DNS name, such as
$ host example.com
n1.example.com 1.2.3.4
n2.example.com 1.2.3.5
n3.example.com 1.2.3.6
The list changes. When I try https://example.com libcurl always uses the same IP for the span of the TTL, and I cannot switch to the next host. There is CURLOPT_DNS_CACHE_TIMEOUT setopt, but setting it to zero does not help - even if I fully recreate easycurl object I still get the same IP. Therefore, this does not help: curl - How to set up TTL for dns cache & How to clear the curl cache
I can of course manually resolve DNS names and iterate, but are there any options? Polling randomly is okay. I see curl uses c-ares. Is there a way to clean up the cache there and will it help?
I cannot do exactly what I need with curl without doing a resolve by myself, but there are findings for the others to share with:
First of all, as a well-written TCP client, curl will try the hosts from the DNS list from top to bottom until a successful connection is made. Since then it will use that host even if it returns some higher level error (such as SSL error or HTTP 500). This is good for all major cases.
Curl command line of newer curl versions has --retry and --retry-all-errors - but there are no such things in libcurl, unfortunately. The feature is being enhanced right now, and there is no release yet as of 2021-07-14 that will enumerate all DNS hosts until there is one that returns HTTP 200. Instead, the released curl versions (I tried 7.76 and 7.77) will always do retries with the same host. But the nightly build (2021-07-14) does enumerate all DNS hosts. Here is how it behaves for two retries and three inexisting hosts (note, the retries will happen if any host returns HTTP 5xx):
$ ./src/curl http://nohost.sureno --trace - --retry 2 --retry-all-errors
== Info: Trying 192.168.1.112:80...
== Info: connect to 192.168.1.112 port 80 failed: No route to host
== Info: Trying 192.168.1.113:80...
== Info: connect to 192.168.1.113 port 80 failed: No route to host
== Info: Trying 192.168.1.114:80...
== Info: connect to 192.168.1.114 port 80 failed: No route to host
== Info: Failed to connect to nohost.sureno port 80 after 9210 ms: No route to host
== Info: Closing connection 0
curl: (7) Failed to connect to nohost.sureno port 80 after 9210 ms: No route to host
Warning: Problem (retrying all errors). Will retry in 1 seconds. 2 retries
Warning: left.
== Info: Hostname nohost.sureno was found in DNS cache
== Info: Trying 192.168.1.112:80...
== Info: connect to 192.168.1.112 port 80 failed: No route to host
== Info: Trying 192.168.1.113:80...
== Info: connect to 192.168.1.113 port 80 failed: No route to host
== Info: Trying 192.168.1.114:80...
== Info: connect to 192.168.1.114 port 80 failed: No route to host
== Info: Failed to connect to nohost.sureno port 80 after 9206 ms: No route to host
== Info: Closing connection 1
curl: (7) Failed to connect to nohost.sureno port 80 after 9206 ms: No route to host
Warning: Problem (retrying all errors). Will retry in 2 seconds. 1 retries
This behavior can be very helpful for the users of libcurl, but unfortunately, these retry flags presently have no mapping to curl_easy_setopt. And as a result, if you give --libcurl to the command line you will not see any retry-related code
Following steps described here to setup logspout:
https://hyperledger-fabric.readthedocs.io/en/release-2.2/deploy_chaincode.html
Running this produces below errors:
./monitordocker.sh net_test
Starting monitoring on all containers on the network net_test
xxxx
docker: Error response from daemon: network net_test not found.
curl: (7) Failed to connect to 127.0.0.1 port 8000: Connection refused
xxx#xxxx:/home/fabric/fabric-samples/test-network#
xxx#xxxx:/home/fabric/fabric-samples/test-network# ./monitordocker.sh
Starting monitoring on all containers on the network basicnetwork_basic
xxxx
docker: Error response from daemon: network basicnetwork_basic not found.
curl: (7) Failed to connect to 127.0.0.1 port 8000: Connection refused
xxx#xxxx:/home/fabric/fabric-samples/test-network#
xxxx#xxxx:/home/fabric/fabric-samples/test-network# ./monitordocker.sh net_basic
Starting monitoring on all containers on the network net_basic
xxxx
docker: Error response from daemon: network net_basic not found.
curl: (7) Failed to connect to 127.0.0.1 port 8000: Connection refused
Few questions:
there is no process running in default port 8000. So connection refused error is expected. Do we need to use any other port ?
what is the name of the network to be given when running monitordocker.sh ?
Any other troubleshooting info is appreciated.
Ok, found the issue. The network name is fabric_test. So I issued command like
./monitordocker.sh fabric_test
This resolved the problem.
So I have a single node cassandra running on an AWS machine which also has the OpsCenter installed. I'm trying to manage it with OpsCenter GUI from a windows machine (which is in the same private network as the cassandra node)however I keep getting the following error
"No HTTP communication to the agent"
Opscenter logs show the following information -
2017-02-19 18:08:17,622 [Test_Cluster] INFO: Node 172.18.51.175 changed its mode to normal (MainThread)
2017-02-19 18:08:17,773 [Test_Cluster] INFO: Using 1.2.3.4 as the RPC address for node 172.18.51.175 (MainThread)
2017-02-19 18:09:12,046 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: User timeout caused connection failure. (MainThread)
2017-02-19 18:10:12,045 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: User timeout caused connection failure. (MainThread)
2017-02-19 18:11:12,046 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: IPv4Address(TCP, '1.2.3.4', 61621) (MainThread)
2017-02-19 18:12:12,045 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: IPv4Address(TCP, '1.2.3.4', 61621) (MainThread)
2017-02-19 18:13:12,433 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: IPv4Address(TCP, '1.2.3.4', 61621) (MainThread)
2017-02-19 18:14:12,045 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: IPv4Address(TCP, '1.2.3.4', 61621) (MainThread)
2017-02-19 18:15:12,045 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: User timeout caused connection failure. (MainThread)
2017-02-19 18:16:12,044 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: IPv4Address(TCP, '1.2.3.4', 61621) (MainThread)
2017-02-19 18:17:12,044 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: IPv4Address(TCP, '1.2.3.4', 61621) (MainThread)
2017-02-19 18:18:12,045 [Test_Cluster] WARN: These nodes reported this message, Nodes: ['172.18.51.175'] Message: HTTP request http://1.2.3.4:61621/connection-status? failed: IPv4Address(TCP, '1.2.3.4', 61621) (MainThread)
So I guess my cassandra.yaml file needs some change ?
Currently I have set listen_address as private IP of my node
my rpc_address is 0.0.0.0
and my broadcast_rpc_address is set as 1.2.3.4
Which is how the datastax doc recommended.
I tried setting the rpc_address and broadcast_rpc_address to the node's private IP and it failed in that scenario as well.
netstat --listen shows the below line for the port 61621 and 61620
tcp6 0 0 [::]:61620 [::]:* LISTEN
tcp6 0 0 [::]:61621 [::]:* LISTEN
I'm not sure what I'm doing wrong or how to set these parameters in cassandra.yaml for it to work with Opscenter.
Note : I seem to be having issues only with OpsCenter with the above config. Cassandra services start up fine and my web application is connecting to the cluster using the datastax driver. Any one have comments on what might be going wrong ?
Thanks
my rpc_address is 0.0.0.0
and my broadcast_rpc_address is set as 1.2.3.4
That is your mistake, change the rpc_address to the local IP --> 172.18.51.175 [if this is the nodes IP]
Check in cassandra.yaml file that the listen_address is also set to --> 172.18.51.175