I'm attempting to start concourse worker with --garden-dns-proxy-enable flag in hope of getting past some git authentication errors that I read could be as a result of DNS issues
Command I am running is below
sudo ./concourse worker \
--work-dir worker \
--tsa-host <ip_address> \
--tsa-public-key host_key.pub \
--tsa-worker-private-key worker_key \
--garden-dns-proxy-enable \
However, I get the following error as address is already in use
Exit trace for group:
garden exited with error: Exit trace for group:
dns-proxy exited with error: listen udp :53: bind: address already in use
garden-runc exited with nil
baggageclaim exited with nil
beacon exited with nil
On further investigation by running $ sudo lsof -i :53 I discovered that port is already in used by dnsmasq
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dnsmasq 3239 nobody 4u IPv4 30745 0t0 UDP WCI:domain
dnsmasq 3239 nobody 5u IPv4 30746 0t0 TCP WCI:domain (LISTEN)
Worker is running on Ubuntu 16.04 and concourse binary is v3.4.0
I appreciate your help in resolving this.
Related
Node Exporter is always running on my local machine on localhost:9100 even if I don't execute it with terminal following this error message:
FATA[0000] listen tcp :9100: bind: address already in use source="node_exporter.go:172"
By which I can understand that this port number is already being used by another application but the thing is I don't have anything hosted there.
This is what netstat | grep 9100 gives:
tcp 0 0 localhost:60232 localhost:9100 ESTABLISHED
tcp6 0 0 localhost:9100 localhost:60232 ESTABLISHED
All I had to do was to "kill" the 9100 port in which Node Exporter was running by using fuser -k 9100/tcp as this was shown on How to kill a process running on particular port in Linux?.
my version of apache
Server version: Apache/2.4.6 (CentOS)
Server built: Apr 20 2018 18:10:38
when I run the command lsof -i :443 it returns nothing
but if I try to run apache (directly by running httpd I got the error, I verified with ps aux that there was no previous httpd/apache process already running)
(98)Address already in use: AH00072: make_sock: could not bind to address [::]:443
However if i try to run a netcat process on 443 nc 0.0.0.0 -l 443 , it does open and I can send data
I'm a bit lost on what could be the problem ?
Found it
Listen 443 was present two times among the different configuration files of apache
it's a pity apache does not have a more explicit error/warning message (i.e "option defined two times" etc.)
It seems another process is using port 443 on your server.
netstat -anp | grep 443
output will be
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
disable port 443 and start
systemctl start httpd.service
Surely I know that same question is already posted here. However, when I searched it, the status is different from mine and I cannot understand the answers. Therefore I post my problem here. Sorry for duplicating issues.
My homepage suddenly doesn't work and I found out that it failed to start httpd service. Following image is the result when I command 'sudo service httpd start'
Starting httpd: (98)Address already in use: AH00072: make_sock: could
not bind to address [::]:80 (98)Address already in use: AH00072:
make_sock: could not bind to address 0.0.0.0:80 no listening sockets
available, shutting down AH00015: Unable to open logs
[FAILED]
restart doesn't work also.
$ sudo service httpd restart
Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: AH00072: make_sock: could not bind
to address [::]:80 (98)Address already in use: AH00072: make_sock:
could not bind to address 0.0.0.0:80 no listening sockets available,
shutting down AH00015: Unable to open logs
[FAILED]
What should I do to restart httpd service and revive my homepage?
Error 98 usually occurs when some webserver is using the port, here 80, or
The clean release port/address was not done.
If port is being used by other webserver, shutdown the server. You can find out which service is using port 80 by
netstat -pan |grep 80
and then shutdown the service.
If the port was not released upon unclean shutdown of server, then
sudo service networking restart
to release address/port combination from bind. This usually fixes error 98 for me.
I have the same problem. So i looked to netstat:
sudo netstat -tulpn | grep :80
and received:
tcp6 0 0 :::80 :::* LISTEN 7836/docker-proxy
after killing process:
sudo kill 7836
Files defined inside conf.d would have Listen port as 80 along with repetitive declaration of Listen port in httpd.conf which can cause this issue.
Seems port 80 is used by some other process, it can be checked by "netstat -anp|grep :80" Or assign a new available port to the Listen directive in httpd.conf and restart httpd.
Your httpd server is already started. Try restarting the service instead of starting it again:
sudo service httpd restart
I'm trying to set up a test cluster using etcd 2.3.7 installed from CentOS RPM on CentOS 7.1. On the Loader 1 I executed:
etcdctl member add loader2 http://10.11.51.231:2380
And received response which confirmed the operation completed successfully.
Similarly:
etcdctl member add loader3 http://10.11.51.231:2380
with all default settings, and here's what I see:
Loader 1 10.11.51.166
systemctl status etcd -ln1
etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled)
Active: active (running) since Sun 2017-02-19 14:33:18 IST; 28min ago
Main PID: 19009 (etcd)
CGroup: /system.slice/etcd.service
└─19009 /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://localhost:2379
Feb 19 15:02:03 loader3 etcd[19009]: cannot get the version of member a4803061db803edc (Get http://10.11.51.166:2380/version: dial tcp 10.11.51.166:2380: getsockopt: connection refused)
Tried to see cluster health:
etcdctl --debug cluster-health
Cluster-Endpoints: http://127.0.0.1:4001, http://127.0.0.1:2379
cURL Command: curl -X GET http://127.0.0.1:4001/v2/members
cURL Command: curl -X GET http://127.0.0.1:2379/v2/members
member ce2a822cea30bfca is unhealthy: got unhealthy result from http://localhost:2379
member da05b63349d818dc is unreachable: no available published client urls
cluster is unhealthy
Note how this ignores the two nodes added previously, but sends requests to random port on localhost...
Loader 2 10.11.51.174
At first this machine started OK, but after I saw there was something wrong with Loader 1, I tried adding Loader 1 as a member from this machine, and now I see the same picture on this machine too. I.e. it tries to query this 4001 port, where nobody responds. On all machines:
netstat -tupln | grep etcd
tcp 0 0 127.0.0.1:7001 0.0.0.0:* LISTEN 4507/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 4507/etcd
tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN 4507/etcd
Nobody listens on 4001...
Loader 3 10.11.51.231
On this loader I didn't try to add new members. So it looks like this:
etcdctl --debug cluster-health
Cluster-Endpoints: http://127.0.0.1:4001, http://127.0.0.1:2379
cURL Command: curl -X GET http://127.0.0.1:4001/v2/members
cURL Command: curl -X GET http://127.0.0.1:2379/v2/members
member ce2a822cea30bfca is healthy: got healthy result from http://localhost:2379
cluster is healthy
In other words it still sends requests to random port, but this time it isn't bothered by the fact that nobody replied...
Below is the contents of the configuration files:
cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
And:
cat /etc/etcd/etcd.conf
# [member]
ETCD_NAME=default
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_WAL_DIR=""
#ETCD_SNAPSHOT_COUNT="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
#ETCD_LISTEN_PEER_URLS="http://localhost:2380"
ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#
#[cluster]
#ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380"
# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
#ETCD_INITIAL_CLUSTER="default=http://localhost:2380"
#ETCD_INITIAL_CLUSTER_STATE="new"
#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_STRICT_RECONFIG_CHECK="false"
#
#[proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
#
#[security]
#ETCD_CERT_FILE=""
#ETCD_KEY_FILE=""
#ETCD_CLIENT_CERT_AUTH="false"
#ETCD_TRUSTED_CA_FILE=""
#ETCD_PEER_CERT_FILE=""
#ETCD_PEER_KEY_FILE=""
#ETCD_PEER_CLIENT_CERT_AUTH="false"
#ETCD_PEER_TRUSTED_CA_FILE=""
#
#[logging]
#ETCD_DEBUG="false"
# examples for -log-package-levels etcdserver=WARNING,security=DEBUG
#ETCD_LOG_PACKAGE_LEVELS=""
#
#[profiling]
#ETCD_ENABLE_PPROF="false"
So... what is going on? The error messages given by etcd are the typical mindless nonsense produced by Go built-ins. The HTTP server that etcd uses is again, the Go built-in junk, that produces non-standard and absolutely worthless replies. So I cannot understand what was (if at all) misconfigured / missing.
My logstash version is:
# /opt/logstash/bin/logstash --version
logstash 2.2.4
it is configured to receive input from port 5044 according to the filebeat file:
/etc/logstash/conf.d/02-beats-input.conf
input {
beats {
port => 5044
ssl => false
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
I have set ssl to false as I am not using it
but when I start the logstash service normally with systemctl it start and checking the status confirms it is running
systemctl status logstash
● logstash.service - LSB: Starts Logstash as a daemon.
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (exited) since Mon 2016-07-18 19:14:51 BST; 15h ago
Docs: man:systemd-sysv-generator(8)
Process: 19965 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 19970 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
...
logstash started
The problem is that logstash does not seem to be receiving input on port 5044. hosts sending filebeats encounter:
single.go:126: INFO Connecting error publishing events (retrying): dial tcp 192.72.0.92:5044: getsockopt: connection refused
when I check the port
# netstat -an | grep 5044
I get nothing. So even though logstash is running, I can't tell what port it is bound to and listening on.
Also the firewall is stopped temporarily to investigate this.
The strange thing is that is I run logstash is debug mode like so:
# ./logstash --debug -f /etc/logstash/conf.d/02-beats-input.conf
I can see
# netstat -an | grep 5044
tcp6 0 0 :::5044 :::* LISTEN
tcp6 0 0 192.72.0.92:5044 192.168.36.70:53720 ESTABLISHED
tcp6 0 0 192.72.0.92:5044 192.72.0.90:45980 ESTABLISHED
tcp6 0 0 192.72.0.92:5044 192.72.0.90:45975 ESTABLISHED
tcp6 0 0 192.72.0.92:5044 192.72.0.90:45976 ESTABLISHED
or
# lsof -i :5044
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 15136 root 7u IPv6 7191510 0t0 TCP *:lxi-evntsvc (LISTEN)
java 15136 root 33u IPv6 7192379 0t0 TCP hostname:lxi-evntsvc->192.72.0.90:45975 (ESTABLISHED)
and the host sending filebeats can connect
output.go:87: DBG output worker: publish 7 events
2016/07/19 10:02:08.017890 client.go:146: DBG Try to publish 7 events to logstash with window size 10
2016/07/19 10:02:08.038579 client.go:124: DBG 7 events out of 7 events sent to logstash. Continue sending ...
2016/07/19 10:02:08.038615 single.go:135: DBG send completed
Please help point out what I may be doing wrong with this configuration. Thanks
Based on the hing provided by #LiGhTx117
I think
The startup script used by logstash in:
/etc/init.d/logstash
has the following variables among others:
LS_USER=logstash
LS_GROUP=logstash
LS_HOME=/var/lib/logstash
LS_LOG_DIR=/var/log/logstash
LS_LOG_FILE="${LS_LOG_DIR}/$name.log"
LS_CONF_DIR=/etc/logstash/conf.d
The ownership and permission on these seem to be the issue.
I ensured that the directories where recursively accessible to the
user logstash as well as the group logstash
and
Then I also ensured that the log_file: logstash.log was writeable by
the user/group logstash
restarted logstash