etcd2 cluster not communicating after HTTPS migration - coreos

I am following the Coreos guide for Enabling HTTPS in an existing etcd cluster. Link to Documentation
I have two questions:
1) After reconfiguring the Peer URL to HTTPS using the following command as stated in the documentation:
etcdctl member list | awk -F'[: =]' '{print "etcdctl member update "$1" https:"$7":"$8}'
The output is the same as the documentation, but what is the output I am suppose to after running:
etcdctl member list
I am suppose to see the peerUrls updated to HTTPs?
2) I went ahead and configured documentation and change the etcd client URLs. After doing that the cluster completely stops communicating:
etcd2[5063]: 5ebdc721c084a4b1 is starting a new election at term 20548
etcd2[5063]: 5ebdc721c084a4b1 became candidate at term 20549
etcd2[5063]: 5ebdc721c084a4b1 received vote from 5ebdc721c084a4b1 at term 20549
etcd2[5063]: 5ebdc721c084a4b1 [logterm: 20478, index: 6405417] sent vote request to d5df37b45e3cb90f at term 20549
etcd2[5063]: 5ebdc721c084a4b1 [logterm: 20478, index: 6405417] sent vote request to f3aee5692d89a2a3 at term 20549
etcd2[5063]: 5ebdc721c084a4b1 [logterm: 20478, index: 6405417] sent vote request to fb362473ced21e89 at term 20549
etcd2[5063]: the connection to peer d5df37b45e3cb90f is unhealthy
etcd2[5063]: the connection to peer f3aee5692d89a2a3 is unhealthy
etcd2[5063]: the connection to peer fb362473ced21e89 is unhealthy
And when I running the following command to debug:
sudo etcdctl --ca-file /etc/ssl/etcd/ca.pem --cert-file /etc/ssl/etcd/server.pem --key-file /etc/ssl/etcd/server-key.pem member list
I get the following output
Failed to get leader: client: etcd cluster is unavailable or misconfigured
And when I run curl as in the documentation its gives me the correct input
curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/server.pem --key /etc/ssl/etcd/server-key.pem https://172.16.0.2:2379/v2/stats/self
{"name":"coreos0","id":"5ebdc721c084a4b1","state":"StateFollower","startTime":"2017-03-21T11:33:13.964177689+03:00","leaderInfo":{"leader":"fb362473ced21e89","uptime":"13m37.308602575s","startTime":"2017-03-21T11:33:14.480109854+03:00"},"recvAppendRequestCnt":33,"sendAppendRequestCnt":0}
My Configuration
/run/systemd/system/etcd2.service.d/20-cloudinit.conf
[Service]
Environment="ETCD_ADVERTISE_CLIENT_URLS=http://172.16.0.2:2379"
Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.16.0.2:2380"
Environment="ETCD_INITIAL_CLUSTER=coreos1=http://172.16.0.4:2380,coreos2=http://172.16.0.5:2380,coreos0=http://172.16.0.2:2380"
Environment="ETCD_INITIAL_CLUSTER_STATE=new"
Environment="ETCD_INITIAL_CLUSTER_TOKEN=cluster1"
Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379"
Environment="ETCD_LISTEN_PEER_URLS=http://172.16.0.2:2380"
Environment="ETCD_NAME=coreos0"
/etc/systemd/system/etcd2.service.d/25-insecure_localhost.conf
[Service]
Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://127.0.0.1:4001"
/etc/systemd/system/etcd2.service.d/30-certs.conf
[Service]
Environment="ETCD_CERT_FILE=/etc/ssl/etcd/server.pem"
Environment="ETCD_KEY_FILE=/etc/ssl/etcd/server-key.pem"
Environment="ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
Environment="ETCD_CLIENT_CERT_AUTH=true"
Environment="ETCD_PEER_CERT_FILE=/etc/ssl/etcd/server.pem"
Environment="ETCD_PEER_KEY_FILE=/etc/ssl/etcd/server-key.pem"
Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ca.pem"
Environment="ETCD_PEER_CLIENT_CERT_AUTH=true"
/etc/systemd/system/etcd2.service.d/40-tls.conf
[Service]
Environment="ETCD_ADVERTISE_CLIENT_URLS=https://172.16.0.2:2379"
Environment="ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379,http://127.0.0.1:4001"
Environment="ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
Thank you in advance

There was a bug in cfssl that cause the peer certificates to be generated wrong. After regeneration was created with echo, the issue resvolbed itself.

Related

Debugging TLS handshake failure

I'm trying to access my peer through the fabric-network nodejs sdk.
However, I encounter an error during the gateway.connect in the sdk and the logs I find in the peer container is not helpful.
All I have, even with the grpc=debug logging mode is :
peer0.catie-test | 2020-09-21 13:27:07.731 UTC [core.comm] ServerHandshake -> ERRO 087 TLS handshake failed with error remote error: tls: handshake failure server=PeerServer remoteaddress=172.17.0.1:49918
peer0.catie-test | 2020-09-21 13:27:07.731 UTC [grpc] handleRawConn -> DEBU 088 grpc: Server.Serve failed to complete security handshake from "172.17.0.1:49918": remote error: tls: handshake failure
Any way to have more helpful logs ? I would like to know, for example, which key are used for the TLS handshake check.
Edit with more infos: Configuration files and TLS verification
My peer is configured with TLS with the env variables :
CORE_PEER_TLS_ENABLED=true
CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/crypto/peer/tls-msp/keystore/key.pem
CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/crypto/peer/tls-msp/signcerts/cert.pem
CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/crypto/peer/tls-msp/tlscacerts/tlsca.catie-test-cert.pem
I have the correct tlscacert of my peer on the client side, because the output from the peer and from client side are the same:
cat /etc/hyperledger/crypto/peer/tls-msp/tlscacerts/tlsca.catie-test-cert.pem # From the peer, output ZTd/o8LLw== at the end
cat /tmp/fabric-start-catie-test/building/artifacts/peer0.catie-test-crypto/tls-msp/tlscacerts/tlsca.catie-test-cert.pem # From the client, output ZTd/o8LLw== at the end
Path to the peer tlscacert is filled in the client side connection-profile.json :
"peers": {
"peer0.catie-test": {
"tlsCACerts": {
"path": "/tmp/fabric-start-catie-test/building/artifacts/peer0.catie-test-crypto/tls-msp/tlscacerts/tlsca.catie-test-cert.pem"
},
"grpcOptions":{
"ssl-target-name-override": "172.17.0.7",
"grpc.keepalive_time_ms": 10000
},
"url": "grpcs://172.17.0.4:7051",
"eventUrl": "grpcs://172.17.0.4:7053"
}
}
And I also checked that the tlsCAcert is the one that generated my peer cert :
openssl verify -CAfile $CORE_PEER_TLS_ROOTCERT_FILE $CORE_PEER_TLS_CERT_FILE # Output : /etc/hyperledger/crypto/peer/tls-msp/signcerts/cert.pem: OK
Edit 2 : Grpc option, peer name instead of IP and client logs
Also tried adding the grpcOptions to the peer section of the connection-profile.json (see the updated paragraph above) but it didn't change anything.
Also tried to add peer name to my /etc/hosts to reach my peer via its name instead of its IP. It makes a warning disappear but don't solve my problem and I prefer to work with IPs in my scripts.
Here are the logs of the nodejs sdk client in case it helps to diagnose the problem, but it only says that the Endorser must be connected and I think it is, because it reaches my peer as I have this TLS error in my peer's logs.
(node:59350) [DEP0123] DeprecationWarning: Setting the TLS ServerName to an IP address is not permitted by RFC 6066. This will be ignored in a future version.
2020-09-23T06:42:20.704Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer0.catie-test, url:grpcs://172.17.0.7:7051, connected:false, connectAttempted:true
2020-09-23T06:42:20.705Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.catie-test url:grpcs://172.17.0.7:7051 timeout:3000
2020-09-23T06:42:20.708Z - error: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer0.catie-test due to Error: Failed to connect before the deadline on Endorser- name: peer0.catie-test, url:grpcs://172.17.0.7:7051, connected:false, connectAttempted:true
at checkState (/home/rqueraud/CATIE/Myrmica/myrmica-start/node_modules/#grpc/grpc-js/build/src/client.js:69:26)
at Timeout._onTimeout (/home/rqueraud/CATIE/Myrmica/myrmica-start/node_modules/#grpc/grpc-js/build/src/channel.js:292:17)
at listOnTimeout (internal/timers.js:549:17)
at processTimers (internal/timers.js:492:7) {
connectFailed: true
}
(node:59350) UnhandledPromiseRejectionWarning: Error: Endorser must be connected
at Channel.addEndorser (/home/rqueraud/CATIE/Myrmica/myrmica-start/node_modules/fabric-common/lib/Channel.js:259:10)
at buildChannel (/home/rqueraud/CATIE/Myrmica/myrmica-start/node_modules/fabric-network/lib/impl/ccp/networkconfig.js:50:21)
at Object.loadFromConfig (/home/rqueraud/CATIE/Myrmica/myrmica-start/node_modules/fabric-network/lib/impl/ccp/networkconfig.js:34:19)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async Gateway.connect (/home/rqueraud/CATIE/Myrmica/myrmica-start/node_modules/fabric-network/lib/gateway.js:279:13)
at async queryChaincode (/home/rqueraud/CATIE/Myrmica/myrmica-start/test/chaincode-sdk/index.js:41:5)
at async /home/rqueraud/CATIE/Myrmica/myrmica-start/test/chaincode-sdk/index.js:57:5
Edit 3 : Docker IPs ? Trying with EC2 instances.
As #Urko mentionned, my nodes are in fact docker container running docker-in-docker (dind) images. Inside these containers are some other containers running the hyperledger peers, cli, ... images.
I access them from the host which is also where I run the fabric sdk nodejs client. I cannot access them via their container name, I think it is only possible in a docker-compose configuration, isn't it ? I already tried (see Edit 2 above) to add their name to my /etc/hosts to reach them via a name instead of an IP but it didn't change anything.
However, as my network startup is scripted, I deployed it using docker-machine in AWS instead of the dind docker containers this time, so these are real instances reachable on the internet. But I still encounter the same errors, here is the log from the peer where you can see this is coming from a public IP :
2020-09-24 08:32:57.653 UTC [core.comm] ServerHandshake -> ERRO 0d7 TLS handshake failed with error remote error: tls: handshake failure server=PeerServer remoteaddress=31.36.26.4:35462
It seems that the connection with your Peer have been defined to be secured by the TLS protocol. So, you may configure you Peer configuration to know wich certificates are you using at the TLS.
As when you connect to any server using this protocol, the communication among the parties is encripted using the certificate of the server (in this case, the Peer will be the server). So, you need to configure your client to trust on the server by the Root CA that haven been used to issue the Peers TLS certificates.
The client is where you use the SDK, so, you should configure it to trust on the Peer TLS certificate. When you configure the connection to the Blockchain nodes (peers and orderers), you would define their direction, as well as their TLS certificate. This one is an example that you can find at the following link. There, you have to define the value of the tlsCACerts param:
orderers:
orderer.example.com:
url: grpcs://localhost:7050
grpcOptions:
ssl-target-name-override: orderer.example.com
grpc-max-send-message-length: 4194304
tlsCACerts:
path: test/fixtures/channel/crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tlscacerts/example.com-cert.pem
peers:
peer0.org1.example.com:
url: grpcs://localhost:7051
grpcOptions:
ssl-target-name-override: peer0.org1.example.com
grpc.keepalive_time_ms: 600000
tlsCACerts:
path: test/fixtures/channel/crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tlscacerts/org1.example.com-cert.pem
----- Edited ----
Also, you have to check the value of the ssl-target-name-override param. It should be the same to you nodes name, as you can see in the example file
----- Edited ----
Why are you using those IPs?! I understand that those IPs are internal from the Docker network, so you should not use them. Could you try using your containers names instead of the docker networks IPs?
----- Edited ----
Could you verify your ca-server configuration file and check that the tls is set to true?
You try a gRPC call in peer Server where peer server is secured with it's TLS system. So if you fail to provide the valid TLS certificate, server tls handshake will be failed and you will not get success to establish the connection.
Please check that your network config file is properly develop, also check that you are using the same TLS certificate that is used to run the peer server and your TLS certificate path is correct.

I'm trying to instantiate a chaincode wriiten in node.js on the hyperledger fabric but I'm getting error

I installed the chaincode on the network but when I'm trying to instantiate,
I'm getting :
"Error: error getting broadcast client: orderer client failed to connect to 127.0.0.1:7050: failed to create new connection: context deadline exceeded"
Also, I tried to add TLS but then I get the following error:
Error: error getting broadcast client: failed to load config for OrdererClient: unable to load orderer.tls.rootcert.file:
And, the orderer logs are:
My orderer TLS is enabled now:
I'll be glad if someone can help or guide me atleast, I'm new to HyperledgerFabric.Thank you.
Adapt to your case:
peer chaincode instantiate -n mychannel -v 1.0 -C petshop -c '{"Args":[""]}' -o 127.0.0.1:7050 --tls --cafile /path/to/orderer/tls/ca.pem
Check the "orderer" logs precisely at the time you are instantiating the chaincode (first without TLS and then with TLS). You can see that time from your command-line when you are running that command.
Update your question with that section of orderer logs.
Also, if I need to guess from your current orderer nodes, you might not have enabled TLS for the orderer container.

Hyperledger Fabric: Do we need to pass TLS cert/key files while joining channel?

I have a multi-org network fabric network up and running from different hosts.
The docker containers for the peers have TLS enabled. The build configuration of the peer:
- CORE_PEER_TLS_ENABLED=true
- CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt
- CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key
- CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt
While creating and joining the channel I followed byfn docs and did not supply the TLS cert/file of the peer while joining the channel. All the peers were able to join the channel.
However, when I tried to fetch the newest block using peer channel fetch newest -o orderer.example.com:7050 -c examplechannel, I got the error:
Serve failed to complete security handshake from "ip:43402": tls: first record does not look like a TLS handshake
Further, I referred this doc on TLS and this doc
on passing TLS certs of the peer with the above fetch command:
peer channel fetch newest -o orderer.example.com:7050 -c examplechannel --tls --certfile $CORE_PEER_TLS_CERT_FILE --keyfile $CORE_PEER_TLS_KEY_FILE --cafile $CORE_PEER_TLS_ROOTCERT_FILE
This gave a new error:
grpc: Server.Serve failed to complete security handshake from "ip:43496": remote error: tls: bad certificate
Debugging TLS issues doc states that this happens when the server does not trust the client certificate. So in my case, I infer that the orderer is not trusting the certificate that the peer is passing.
So
Does it mean that I was supposed to pass TLS_CERT, TLS_KEY and TLS_ROOT cert while proposing a channel join from this peer?
If #1 is true, can I pass these certificates without bringing down the network and rejoining the channel with certs?
While creating the channel, it uses orderer's ca-cert. Also, when submitting anchor peer transaction, it uses orderer's ca cert? So am I supposed to use Orderer's ca cert for peer fetch command as well?
If #3 is true, then in a production environment, do we need to build up the peer containers such that it contains orderer's ca-cert?
There are multiple concepts baked into your questions. It's important to understand that there is a difference between using the peer to run a peer node--peer node start-- and using the peer as a CLI (e.g. peer channel fetch).
When the peer is running as a server, there's no need to pass in crypto material for the channels as the peer actually extracts the required TLS certificate information from the config block passed in the peer channel join ... command.
When the peer is running in CLI mode, you do need to provide the the TLS certificate information to connect to the various endpoints. When communicating with peers, this information is extracted from the peer config (either in core.yaml or from the corresponding CORE_ environment variables). When communicating with the orderer, there are specific command line flags for setting the TLS material.

Kubernetes on Azure : connectex

Followed steps from the link to create a K8s cluster using the Azure Portal. Tried using kubectl on a remote machine to check if it's working. Got this error.
Unable to connect to the server: dial tcp 13.90.35.157:443: connectex:
A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond.
I can SSH to the K8s master. Tried kubectl get nodes from the master and got similar error.
It is really hard to say from such a description what went wrong, but as this is a new cluster ( and I'm saying this because sometimes k8s cluster gets deployed but doesn't really work, so ), I would suggest deleting it and creating a new one and\or creating it using the Azure Cli\Azure Cloud Shell.
Basically its as simple as:
az acs create -n acs-cluster -g acsrg1 -d applink789 --generate-ssh-keys
if you have the resource group created, if not you can create it with:
az group create -n acsrg1 -l "westus"
According to your description, it seems you have not configured the Service Principal correctly. I use wrong service principal to deploy K8S in Azure, get the same error:
C:\Users>kubectl get nodes
Unable to connect to the server: dial tcp 13.90.27.73:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
You may need to check to ensure the credentials were provided accurately, and that the configured Service Principal has read and write permissions to the target Subscription.
If your Service Principal is misconfigured, none of the kubernetes components will come up in a healthy manner. We can check to see if this the problem:
root#k8s-master-6FEE48E1-0:~# journalctl -u kubelet | grep --text autorest
If you see output that looks like the following, it means you have not configured the service Principal correctly.
root#k8s-master-6FEE48E1-0:~# journalctl -u kubelet | grep --text autorest
Jun 01 01:58:47 k8s-master-6FEE48E1-0 docker[5522]: E0601 01:58:47.447321 6028 kubelet.go:1186] Cannot get Node info: failed to get external ID from cloud provider: autorest#WithErrorUnlessStatusCode: POST https://login.microsoftonline.com/1fcf418e-66ed-4c99-9449-d8e18bf8737a/oauth2/token?api-version=1.0 failed with 400 Bad Request: StatusCode=400
Jun 01 01:58:47 k8s-master-6FEE48E1-0 docker[5522]: E0601 01:58:47.627128 6028 kubelet_node_status.go:70] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: autorest#WithErrorUnlessStatusCode: POST https://login.microsoftonline.com/1fcf418e-66ed-4c99-9449-d8e18bf8737a/oauth2/token?api-version=1.0 failed with 400 Bad Request: StatusCode=400
Jun 01 01:58:47 k8s-master-6FEE48E1-0 docker[5522]: E0601 01:58:47.885092 6028 kubelet_node_status.go:70] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: autorest#WithErrorUnlessStatusCode: POST https://login.microsoftonline.com/1fcf418e-66ed-4c99-9449-d8e18bf8737a/oauth2/token?api-version=1.0 failed with 400 Bad Request: StatusCode=400
More information about how to create /configure a service principal for ACS-Engin Kubernetes cluster, please refer to this link.

What is etcd looking for in 127.0.0.1:4001?

I'm trying to set up a test cluster using etcd 2.3.7 installed from CentOS RPM on CentOS 7.1. On the Loader 1 I executed:
etcdctl member add loader2 http://10.11.51.231:2380
And received response which confirmed the operation completed successfully.
Similarly:
etcdctl member add loader3 http://10.11.51.231:2380
with all default settings, and here's what I see:
Loader 1 10.11.51.166
systemctl status etcd -ln1
etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled)
Active: active (running) since Sun 2017-02-19 14:33:18 IST; 28min ago
Main PID: 19009 (etcd)
CGroup: /system.slice/etcd.service
└─19009 /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://localhost:2379
Feb 19 15:02:03 loader3 etcd[19009]: cannot get the version of member a4803061db803edc (Get http://10.11.51.166:2380/version: dial tcp 10.11.51.166:2380: getsockopt: connection refused)
Tried to see cluster health:
etcdctl --debug cluster-health
Cluster-Endpoints: http://127.0.0.1:4001, http://127.0.0.1:2379
cURL Command: curl -X GET http://127.0.0.1:4001/v2/members
cURL Command: curl -X GET http://127.0.0.1:2379/v2/members
member ce2a822cea30bfca is unhealthy: got unhealthy result from http://localhost:2379
member da05b63349d818dc is unreachable: no available published client urls
cluster is unhealthy
Note how this ignores the two nodes added previously, but sends requests to random port on localhost...
Loader 2 10.11.51.174
At first this machine started OK, but after I saw there was something wrong with Loader 1, I tried adding Loader 1 as a member from this machine, and now I see the same picture on this machine too. I.e. it tries to query this 4001 port, where nobody responds. On all machines:
netstat -tupln | grep etcd
tcp 0 0 127.0.0.1:7001 0.0.0.0:* LISTEN 4507/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 4507/etcd
tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN 4507/etcd
Nobody listens on 4001...
Loader 3 10.11.51.231
On this loader I didn't try to add new members. So it looks like this:
etcdctl --debug cluster-health
Cluster-Endpoints: http://127.0.0.1:4001, http://127.0.0.1:2379
cURL Command: curl -X GET http://127.0.0.1:4001/v2/members
cURL Command: curl -X GET http://127.0.0.1:2379/v2/members
member ce2a822cea30bfca is healthy: got healthy result from http://localhost:2379
cluster is healthy
In other words it still sends requests to random port, but this time it isn't bothered by the fact that nobody replied...
Below is the contents of the configuration files:
cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
And:
cat /etc/etcd/etcd.conf
# [member]
ETCD_NAME=default
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
#ETCD_SNAPSHOT_COUNT="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_LISTEN_PEER_URLS="http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#
#[cluster]
#ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380"
# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
#ETCD_INITIAL_CLUSTER="default=http://localhost:2380"
#ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_STRICT_RECONFIG_CHECK="false"
#
#[proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[security]
#ETCD_CERT_FILE=""
#ETCD_KEY_FILE=""
#ETCD_CLIENT_CERT_AUTH="false"
#ETCD_TRUSTED_CA_FILE=""
#ETCD_PEER_CERT_FILE=""
#ETCD_PEER_KEY_FILE=""
#ETCD_PEER_CLIENT_CERT_AUTH="false"
#ETCD_PEER_TRUSTED_CA_FILE=""
#
#[logging]
#ETCD_DEBUG="false"
# examples for -log-package-levels etcdserver=WARNING,security=DEBUG
#ETCD_LOG_PACKAGE_LEVELS=""
#
#[profiling]
#ETCD_ENABLE_PPROF="false"
So... what is going on? The error messages given by etcd are the typical mindless nonsense produced by Go built-ins. The HTTP server that etcd uses is again, the Go built-in junk, that produces non-standard and absolutely worthless replies. So I cannot understand what was (if at all) misconfigured / missing.

Resources