Inexplicable node.js http throwing connect ECONNREFUSED (IPv6?) - node.js

I am running node.js as follows:
> http = require('http')
> http.get('http://myhost.local:8080',
function (res) { console.log("RES" + res) }
).on('error', function (e) { console.log("Error:", e) })
> uri = require('url').parse("http://myhost.local:8080")
{ protocol: 'http:',
slashes: true,
auth: null,
host: 'myhost.local:8080',
port: '8080',
hostname: 'myhost.local',
hash: null,
search: null,
query: null,
pathname: '/',
path: '/',
href: 'http://myhost.local:8080/' }
> http.get(uri,
function (res) { console.log("RES" + res) }
).on('error', function (e) { console.log("Error:", e) })
An error is thrown for both the implicit and explicitly parsed URI and I get the following output for both:
Error: { [Error: connect ECONNREFUSED]
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect' }
The host myhost.local is an alias for localhost in /etc/hosts, being:
127.0.0.1 localhost myhost.local myhost
255.255.255.255 broadcasthost
::1 localhost myhost.local myhost
fe80::1%lo0 localhost myhost.local myhost
EDIT: I tried virtually every permutation for the hosts file, including the most obvious:
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost myhost.local myhost
fe80::1%lo0 localhost
EDIT I should also mention that I have tried this on more than one Mac now.
Although it seems this is a rather common error, I have seen no useful explanations or workarounds. Here are some notable related facts:
Running $ wget http://myhost.local:8080 works as expected, so it isn't a firewall problem.
Running $ telnet myhost.local 8080 and then manually GET'ing the url works fine, so it's not a weird HTTP problem.
I have no trouble using node.js to connect to other hosts e.g. http://www.google.com
I expect the useful system information would include:
$ node -v
v0.9.11
$ uname -a
Darwin myhost.local 12.2.1 Darwin Kernel Version 12.2.1:
Thu Oct 18 12:13:47 PDT 2012; root:xnu-2050.20.9~1/RELEASE_X86_64 x86_64
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.8.2
BuildVersion: 12C3104
$ sudo netstat -nalt | grep LISTEN | grep 8080
tcp6 0 0 ::1.8080 *.* LISTEN
Does anyone have any idea what is going on here, and what a fix might be?

I'm going to post this here in case somebody else has the problem.
Bert Belder, Node.js mailing list:
On your system "myhost.local" resolves to three different addresses
(127.0.0.1, ::1, and fe80::1). Node prefers ipv4 over ipv6 so it'll
try to connect to 127.0.0.1. Nothing is listening on 127.0.0.1:8080 so
the connect() syscall fails with ECONNREFUSED. Node doesn't retry with
any of the other resolved IPs - it just reports the error to you. A
simple solution would be to replace 'localhost' by the intended
destination ip address, '::1'.
Whether this behavior is right is somewhat open for debate, but this
is what causes it.
Bert

This stemmed from an issue with Node (though there are ways to work around it), as per the discussion on nodejs/Google Groups, as #alessioalex alluded in his answer. A useful comment per Bert Belder:
there should be a getaddrinfo wrapper that returns more that just the first result
For example,
> require('dns').lookup('myhost.local', console.log)
{ oncomplete: [Function: onanswer] }
> null '127.0.0.1' 4
This is the first of multiple getaddrinfo results passed to Node. It seems that nodejs only uses the first item of the getaddrinfo call. Bert and Ben Noordhuis agreed in the Groups discussion that there should be a way to return more than just the first result with the getaddrinfo wrapper.
Contrast python, which returns all results from getaddrinfo:
>>> import socket
>>> socket.getaddrinfo("myhost.local", 8080)
[(30, 2, 17, '', ('::1', 8080, 0, 0)),
(30, 1, 6, '', ('::1', 8080, 0, 0)),
(2, 2, 17, '', ('127.0.0.1', 8080)),
(2, 1, 6, '', ('127.0.0.1', 8080)),
(30, 2, 17, '', ('fe80::1%lo0', 8080, 0, 1)),
(30, 1, 6, '', ('fe80::1%lo0', 8080, 0, 1))]

does this work?
var http = require('http');
var options = {
host: 'myhost.local',
port: 8080,
path: '/'
};
http.get(options, function (res) {
console.log("RES" + res)
}).on('error', function (e) {
console.log("Error:", e)
});

Related

Node Integration Issue on Drupal8

After some efforts the HTTP server started and status report showed that
node.js server was successfully reached. For testing my Drupal site I hit
a random url and waited for it to reflect on my dblog at the same time (as
demonstrated in the video). It failed. The backend showed the error - The
channel "watchdog_dblog" doesn't exist.
Here, the port used was 8888 (as per the video). It was changed to 8080
and then this error did not show up, but the drupal site still did not
auto-refresh.
The nodejs.config.js file currently:
settings = {
scheme: 'http',
port: 8080,
host: 'localhost',
resource: '/socket.io',
serviceKey: 'mytest1',
backend: {
port: 80,
host: 'drupal8',
scheme: 'http',
basePath: '',
messagePath: '/nodejs/message'
},
debug: true,
sslKeyPath: '',
sslCertPath: '',
sslCAPath: '',
baseAuthPath: '/nodejs/',
publishUrl: 'publish',
kickUserUrl: 'user/kick/:uid',
logoutUserUrl: 'user/logout/:authtoken',
addUserToChannelUrl: 'user/channel/add/:channel/:uid',
removeUserFromChannelUrl: 'user/channel/remove/:channel/:uid',
addChannelUrl: 'channel/add/:channel',
removeChannelUrl: 'channel/remove/:channel',
setUserPresenceListUrl: 'user/presence-list/:uid/:uidList',
addAuthTokenToChannelUrl: 'authtoken/channel/add/:channel/:uid',
removeAuthTokenFromChannelUrl: 'authtoken/channel/remove/:channel/:uid',
toggleDebugUrl: 'debug/toggle',
contentTokenUrl: 'content/token',
publishMessageToContentChannelUrl: 'content/token/message',
extensions: [],
clientsCanWriteToChannels: true,
clientsCanWriteToClients: true,
transports: ['websocket', 'flashsocket', 'htmlfile', 'xhr-polling', 'jsonp-polling'],
jsMinification: true,
jsEtag: true,
logLevel: 1
};
I did not see socket.io while looking at the page source and inspect
element network. After that the node.js config module was enabled but the
service key field was non-editable. I then created a file nodejs.config.js
in modules/nodejs and copied the configuration there. Socket.io showed up
with the path localhost:8080//socket.io/socket.io.js. Even if I change the
port to 8888, the socket path/port remains the same.
This is the error now cleanupSocket: Cleaning up after socket id
C0I_r38wIbcT1LbtAAAD, uid undefined

Not able to connect to Elasticsearch from docker container (node.js client)

I have set up an elasticsearch/kibana docker configuration and I want to connect to elasticsearch from inside of a docker container using the #elastic/elasticsearch client for node. However, the connection is "timing out".
The project is taken with inspiration from Patrick Triest : https://blog.patricktriest.com/text-search-docker-elasticsearch/
However, I have made some modification in order to connect kibana, use a newer ES image and the new elasticsearch node client.
I am using the following docker-compose file:
version: "3"
services:
api:
container_name: mp-backend
build: .
ports:
- "3000:3000"
- "9229:9229"
environment:
- NODE_ENV=local
- ES_HOST=elasticsearch
- PORT=3000
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
container_name: elasticsearch
environment:
- node.name=elasticsearch
- cluster.name=es-docker-cluster
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- "http.cors.allow-origin=*"
- "http.cors.enabled=true"
- "http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization"
- "http.cors.allow-credentials=true"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- data01:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- elastic
kibana:
image: docker.elastic.co/kibana/kibana:7.5.1
ports:
- "5601:5601"
links:
- elasticsearch
networks:
- elastic
depends_on:
- elasticsearch
volumes:
data01:
driver: local
networks:
elastic:
driver: bridge
When building/ bringing the container up, I able to get a response from ES: curl -XGET "localhost:9200", "you know, for search"... And kibana is running and able to connect to the index.
I have the following file located in the backend container (connection.js):
const { Client } = require("#elastic/elasticsearch");
const client = new Client({ node: "http://localhost:9200" });
/*Check the elasticsearch connection */
async function health() {
let connected = false;
while (!connected) {
console.log("Connecting to Elasticsearch");
try {
const health = await client.cluster.health({});
connected = true;
console.log(health.body);
return health;
} catch (err) {
console.log("ES Connection Failed", err);
}
}
}
health();
If I run it outside of the container then I get the expected response:
node server/connection.js
Connecting to Elasticsearch
{
cluster_name: 'es-docker-cluster',
status: 'yellow',
timed_out: false,
number_of_nodes: 1,
number_of_data_nodes: 1,
active_primary_shards: 7,
active_shards: 7,
relocating_shards: 0,
initializing_shards: 0,
unassigned_shards: 3,
delayed_unassigned_shards: 0,
number_of_pending_tasks: 0,
number_of_in_flight_fetch: 0,
task_max_waiting_in_queue_millis: 0,
active_shards_percent_as_number: 70
}
However, if I run it inside of the container:
docker exec mp-backend "node" "server/connection.js"
Then I get the following response:
Connecting to Elasticsearch
ES Connection Failed ConnectionError: connect ECONNREFUSED 127.0.0.1:9200
at onResponse (/usr/src/app/node_modules/#elastic/elasticsearch/lib/Transport.js:214:13)
at ClientRequest.<anonymous> (/usr/src/app/node_modules/#elastic/elasticsearch/lib/Connection.js:98:9)
at ClientRequest.emit (events.js:223:5)
at Socket.socketErrorListener (_http_client.js:415:9)
at Socket.emit (events.js:223:5)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:81:21) {
name: 'ConnectionError',
meta: {
body: null,
statusCode: null,
headers: null,
warnings: null,
meta: {
context: null,
request: [Object],
name: 'elasticsearch-js',
connection: [Object],
attempts: 3,
aborted: false
}
}
}
So, I tried changing the client connection to (I read somewhere that this might help):
const client = new Client({ node: "http://172.24.0.1:9200" });
Then I am just "stuck" waiting for a response. Only one console.log of "Connecting to Elasticsearch"
I am using the following version:
"#elastic/elasticsearch": "7.5.1"
As you probably see, I do not have a full grasp of what is happening here... I have also tried to add:
links:
- elasticsearch
networks:
- elastic
To the api service, without any luck.
Does anyone know what I am doing wrong here? Thank you in advance :)
EDIT:
I did a "docker network inspect" on the network with *_elastic. There I see the following:
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.22.0.0/16",
"Gateway": "172.22.0.1"
}
]
},
Changing the client to connect to the "GateWay" Ip:
const client = new Client({ node: "http://172.22.0.1:9200" });
Then it works! I am still wondering why as this was just "trial and error" Is there any way to obtain this Ip without having to inspect the network?
In Docker, localhost (or the corresponding IPv4 address 127.0.0.1, or the corresponding IPv6 address ::1) generally means "this container"; you can't use that host name to access services running in another container.
In a Compose-based setup, the names of the services: blocks (api, elasticsearch, kibana) are usable as host names. The caveat is that all of the services have to be on the same Docker-internal network. Compose creates one for you and attaches containers to it by default. (In your example api is on the default network but the other two containers are on a separate elastic network.) Networking in Compose in the Docker documentation has some more details.
So to make this work, you need to tell your client code to honor the environment variable you're setting that points at Elasticsearch
const esHost = process.env.ES_HOST || 'localhost';
const esUrl = 'http://' + esHost + ':9200';
const client = new Client({ node: esUrl });
In your docker-compose.yml file delete all of the networks: blocks to use the provided default network. (While you're there, links: is unnecessary and Compose provides reasonable container_name: for you; api can reasonably depends_on: [elasticsearch].)
Since we've provided a fallback for $ES_HOST, if you're working in a host development environment, it will default to using localhost; outside of Docker where it means "the current host" it will reach the published port of the Elasticsearch container.

Connecting to Cayley serving over localhost

I've followed the 'Getting Started' guide in Cayley's documentation and installed Cayley on my remote server:
Getting Started: https://github.com/google/cayley
Server OS: CentOS 7.2.1511
I've added cayley to my $PATH:
echo $PATH :
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/csse/cayley/src/github.com/google/cayley
Here is my config file at /etc/cayley.cfg
{
"database": "leveldb",
"db_options": {
"cache_size_mb": 2,
"write_buffer_mb": 20
},
"db_path": "~/cayley/src/github.com/google/cayley/data/testdata.nq",
"listen_host": "127.0.0.1",
"listen_port": "64210",
"read_only": false,
"replication_options": {
"ignore_missing": false,
"ignore_duplicate": false
},
"timeout": 30
}
I serve cayley over http by simply doing:
cayley http
and the terminal outputs:
Cayley now listening on 127.0.0.1:64210
On my main machine (Mac OSX 10.10.5 Yosemite), I've used npm to install the cayley package and written a test:
##testconnection.js
var cayley = require('cayley');
var client = cayley("137.112.104.107");
var g = client.graph;
g.V().All(function(err, result) {
if(err) {
console.log('error');
} else {
console.log('result');
}
});
However, it fails when I run it: node testconnection.js
error: Error: Invalid URI "137.112.104.107/api/v1/query/gremlin"
I'd like to connect to Cayley and modify the database from my test. I've found a great powerpoint full of Cayley information:
https://docs.google.com/presentation/d/1tCbsYym1kXWWDcnRU9ymj6xP0Nvgq-Qhy9WDmqWcM-o/edit#slide=id.g3776708f1_0319
As well as pertinent Cayley docs:
Overview Doc
Configuration Doc
HTTP API Doc
And a post on stackoverflow:
Cayley db user and password protection over HTTP connections
But I'm struggling to come up with a way to connect Cayley (on my remote machine) with my local machine. I'd like to connect with npm if possible, but am open to other options. Where am I going wrong?
Edit #1
I've appended the "http://" to my ip, so now it reads http://137.112.104.107. At that point, I solved another issue by performing
cayley init --config=/etc/cayley.cfg
as mentioned by the author here
I've also removed the listen_post and listen_port from my config file (each individually first, then both), yet have still have the same socket hang up error. Here's a printout of client from the test script:
Client {
host: 'http://137.112.104.107',
request:
{ [Function]
get: [Function],
head: [Function],
post: [Function],
put: [Function],
patch: [Function],
del: [Function],
cookie: [Function],
jar: [Function],
defaults: [Function] },
graph: Gremlin { client: [Circular], query: [Function] },
g: Gremlin { client: [Circular], query: [Function] },
write: [Function: bound ],
delete: [Function: bound ],
writeFile: [Function: bound ]
}
Your Cayley server is listening on 127.0.0.1 / localhost and therefor not reachable from another machine. To be able to reach it from a virtual machine or another computer on your network it needs to bind to an interface that is reachable.
If you configure host: 0.0.0.0 and check what is your network IP (I assume: 137.112.104.107) and connect it, it should work or you need to open it or forward the port on your firewall (depending on your network).

How to clean up masscan output (-oG)

I have a problem with the output produced by the masscan utility with the -oG options ("grep-able" output); for instance, it outputs this:
# Masscan 1.0.3 scan initiated Wed Jun 4 01:35:02 2014
# Ports scanned: TCP(3;21-23,) UDP(0;) SCTP(0;) PROTOCOLS(0;)
Host: 192.168.100.19 () Ports: 2222/open/tcp////
Host: 192.168.100.13 () Ports: 2222/open/tcp////
Host: 192.168.100.16 () Ports: 443/open/tcp////
Host: 192.168.100.8 () Ports: 21/open/tcp////
Host: 192.168.100.5 () Ports: 22/open/tcp////
Host: 192.168.100.5 () Ports: 443/open/tcp////
Host: 192.168.100.16 () Ports: 80/open/tcp////
Host: 192.168.100.19 () Ports: 22/open/tcp////
Host: 192.168.100.7 () Ports: 80/open/tcp////
Host: 192.168.100.8 () Ports: 80/open/tcp////
Host: 192.168.100.12 () Ports: 2222/open/tcp////
Host: 192.168.100.13 () Ports: 22/open/tcp////
# Masscan done at Wed Jun 4 01:35:16 2014
The above is neither very readable nor easy to understand.
How can I use Linux command-line utilities, e.g. sed, awk, or grep, to output something as follows, using the file above?
Host: 192.168.100.5
Ports: 22, 443
Host: 192.168.100.7
Ports: 80
Host: 192.168.100.8
Ports: 21, 80
Host: 192.168.100.12
Ports: 2222
Host: 192.168.100.13
Ports: 2222, 22
......
As you can see, the output is much more readable in this layout:
sorted by IP address, with all associated ports listed below, consolidated across multiple input lines with the same IP address.
Try this:
awk -F' +|/' '
!/\s*#/ { # ignore comment lines
# Add the port on the current line to the associative array
# element for the IP address on the current line.
ips[$2] = ips[$2] (ips[$2] == "" ? $5 : ", " $5)
}
END {
# Enumerate all IPs and the ports for each.
# Since the IPs will be listed in no specific order, the output
# is piped as a _single_ line to "sort" in order to sort by IP address,
# and then expanded into 2 lines via "tr".
for (ip in ips) {
printf "Host: %s#Ports: %s#\n", ip, ips[ip] | \
"sort -t. -n -k 1.6,1 -k 2,2 -k 3,3 -k 4,4 | tr # \"\n\""
}
}
' file
This solution properly sorts the output by IP address and separates the ports with commas.
By contrast, for a given IP address, the port numbers are listed in the order they were encountered in the input (as in the sample output data in the question).

Regarding the error occurred in my test case in nodeload.js script

I have designed an app using elastic search. And when Iam trying to write the test case using node load.js. I have got a problem that when I increase the number users I was getting the warning that "WARN: Error during HTTP request: Error: ECONNREFUSED, Could not contact DNS servers" and Iam unable rectify the problem so please help me in solving this error.
nl.run({
name: "test",
host: 'localhost',
port: 9200,
//path: '/my_river/page/_search?q=sweden',
numUsers: 2000, //Increased my num of user**
timeLimit: 180,
targetRps: 500,
stats: [
'result-codes',
{ name: 'latency', percentiles: [0.9, 0.99] },
'concurrency',
'rps',
'uniques',
{ name: 'http-errors', successCodes: [200,404], log: 'http-errors.log' }
],

Resources