I'm attempting to use the packages 'request promise' and 'cheerio' on NodeJS to scrape links from a website, via an Amazon EC2 instance.
I'm able to run my script on my own desktop (outside the Amazon EC2 instance) and scrape the links without any problems.
However, when I run the same script in my Amazon EC2 instance, I get the following error:
Unhandled rejection RequestError: Error: socket hang up
at new RequestError (/home/ec2-user/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/home/ec2-user/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/home/ec2-user/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/home/ec2-user/node_modules/request/request.js:186:22)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at Request.onRequestError (/home/ec2-user/node_modules/request/request.js:845:8)
at emitOne (events.js:96:13)
at ClientRequest.emit (events.js:188:7)
at TLSSocket.socketOnEnd (_http_client.js:346:9)
Does this have something to do with my AWS security group settings? Currently, outbound traffic is configured to 'All Traffic' with destination 0.0.0.0/0, while inbound traffic is only set to the following configuration: SSH, TCP, Port 20, my own IP address. Should I be adding another rule for inbound traffic?
Alternatively, could this have something to do with my instance size? I'm currently using a micro instance that is free under AWS.
Or finally, could it be the case that the website has blocked access from AWS instances?
Related
I am testing a nodejs app on GCP functions and a MongoDB instance in GCP cloud run(I know this is not a good idea) but again this is a test. I am able to get the mongo image running on cloud run and I am given a service endpoint as https://mongodb.foo.bar.run.app and I have the container port in this cloud run service as 27017.
When I try and edit the mongoose.connect() method to include this service uri I am not able to connect with the app. I have the mongodb cloud run instance open to receive traffic from all traffic(Again not a good idea), but I still can't get the node app to connect.
I have setup the connection in node as mongoose.connect('mongodb://mongodb.foo.bar.run.app:27017/test');
but I get this error
Error: connect ETIMEDOUT <IP ADDRESS GOES HERE>:27017
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16)
at TCPConnectWrap.callbackTrampoline (internal/async_hooks.js:129:14)
Emitted 'error' event on NativeConnection instance at:
at NativeConnection.Connection.error (/Users/foo/nodejs/bar/node_modules/mongoose/lib/connection.js:443:8)
at /Users/foo/nodejs/bar/node_modules/mongoose/lib/connection.js:472:15
at /Users/foo/nodejs/bar/node_modules/mongoose/lib/drivers/node-mongodb-native/connection.js:59:21
at /Users/foo/nodejs/bar/node_modules/mongodb/lib/db.js:232:14
at Server.<anonymous> (/Users/foo/nodejs/bar/node_modules/mongodb/lib/server.js:240:9)
at Object.onceWrapper (events.js:421:26)
at Server.emit (events.js:314:20)
at Pool.<anonymous> (/Users/foo/nodejs/bar/node_modules/mongodb-core/lib/topologies/server.js:308:68)
at Pool.emit (events.js:314:20)
at Connection.<anonymous> (/Users/foo/nodejs/bar/node_modules/mongodb-core/lib/connection/pool.js:115:12)
at Object.onceWrapper (events.js:421:26)
at Connection.emit (events.js:314:20)
at Socket.<anonymous> (/Users/foo/nodejs/bar/node_modules/mongodb-core/lib/connection/connection.js:144:49)
at Object.onceWrapper (events.js:421:26)
at Socket.emit (events.js:314:20)
at emitErrorNT (internal/streams/destroy.js:100:8) {
name: 'MongoError'
}
I tried to curl the service uri as
curl https://mongodb.foo.bar.run.app
and I get a 200 response in the mongo logs GET 200 466B but if I
curl mongodb.foo.bar.run.app
I get a 302 response code GET 302 0B.
I looked at the mongoose documentation but I couldn't figure what I was doing wrong. I know the documentation states user and password on the uri string, but when I run it locally on my docker engine I did not use a user and password and it worked.
As mentioned in the comments, it's not possible to run MongoDB on Cloud Run since as mentioned here:
Cloud Run can only receive HTTP requests or Pub/Sub push events.
Connecting to MongoDB requires a TCP protocol rather than HTTP, that's why every call to it would fail even if the port is reachable.
Cloud Run is in this sense more similar to AWS Lambda than it's to AWS Fargate as it's even billed on a per request basis rather than on resources consumed.
I'm trying to setup a Node.js Websocket server over at my webhost (Namecheap). However, I am getting this error:
events.js:292
throw er; // Unhandled 'error' event
^
Error: Unexpected server response: 301
at ClientRequest.<anonymous> (D:\****\node_modules\ws\lib\websocket.js:576:7)
at ClientRequest.emit (events.js:315:20)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:596:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:119:17)
at TLSSocket.socketOnData (_http_client.js:469:22)
at TLSSocket.emit (events.js:315:20)
at addChunk (_stream_readable.js:295:12)
at readableAddChunk (_stream_readable.js:271:9)
at TLSSocket.Readable.push (_stream_readable.js:212:10)
at TLSWrap.onStreamRead (internal/stream_base_commons.js:186:23)
Emitted 'error' event on WebSocket instance at:
at abortHandshake (D:\****\node_modules\ws\lib\websocket.js:694:15)
at ClientRequest.<anonymous> (D:\****\node_modules\ws\lib\websocket.js:576:7)
[... lines matching original stack trace ...]
at TLSSocket.Readable.push (_stream_readable.js:212:10)
The code for the client and server came from here:
https://github.com/websockets/ws#Simple server
https://github.com/websockets/ws#sending-and-receiving-text-data
with ws://www.host.com/path changed to ws://<mydomainname>/<dir1>/<dir2>
Both client and server code runs properly when run locally. Http related code runs properly when used on my webhost. I am trying to get the client code to run locally and connect via websockets to my server hosted at Namecheap.
I suspect something related to the server is preventing connection. I'd ask support but I doubt if they can resolve this. Maybe there's a locked feature or something? Is there something else I could have missed?
Websockets cannot work on Namecheap's hosted servers. Their incoming ports are blocked and cannot be opened for security reasons per their customer's support.
If anyone is looking for a solution just in case, try out a Heroku free account.
node enrollAdmin.js
error: [FabricCAClientService.js]: Failed to enroll admin, error:%o message=Calling enrollment endpoint failed with error [Error: con
nect ECONNREFUSED 127.0.0.1:7054], stack=Error: Calling enrollment endpoint fail
ed with error [Error: connect ECONNREFUSED 127.0.0.1:7054]
at ClientRequest.request.on (C:\Users\Nibir Hira\go\src\github.com\hyperledg
er\fabric-samples\fabcar\javascript\node_modules\fabric-ca-client\lib\FabricCACl
ient.js:487:12)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at Socket.socketErrorListener (_http_client.js:387:9)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
Failed to enroll admin user "admin": Error: Calling enrollment endpoint failed w
ith error [Error: connect ECONNREFUSED 127.0.0.1:7054]
Where should I do corrections? The file registeradmin.js is not even working
Here is a solution for Hyperledger Fabric v2.x. The problem is port no: 7054 is missing in test network.
In Linux (try if it works in virtual box), one way to solve is fresh restart from fabcar folder (not test network folder) then run again.
cd $HOME/fabric-samples/fabcar
./startFabric.sh javascript (by default it's goLang)
A fresh restart of the network will occur as "startFabric.sh" contains network up, down, chaincode installing, all in one.
Now try again:
node enrollAdmin.js
Worked for me.
Extra: Now you can run node registerUser.js and node invoke.js if you were about to use invoke.js .
Your node application cannot connect to the Fabric CA server running in the container - so either it is not running, has failed or possibly you have some firewall or other connectivity problem.
You can test connectivity by using wget or curl or a browser to hit this URL - http://localhost:7054/api/v1/cainfo and until you can hit this URL your enrollAdmin program will fail.
I don't use windows, but docker ps on Linux would tell me if the ca container was running, and docker logs ca.example.com would give me the log of the ca container (assuming you are running the fabcar sample on the basic sample network.)
I had the exact same problem. This worked for me:
Open VirtualBox Manager and select the “default” VM
Click “Settings”, “Network”, “Advanced”, “Port Forwarding”.
Create a new rule by clicking the “+” sign on the right and entering the following: Name -> “ca”, Host Port -> 7054, Guest Port -> 7054.
You can leave the Host IP and Guest IP unspecified.
Source:
https://developer.ibm.com/opentech/2017/11/29/running-hyperledger-fabric-windows-revised/
I have a MongoDB Cluster in MongoDB Atlas and I use this MONGO_URI that they provided to connect to it. The format is something like this:
mongodb://<USERNAME>:<PASSWORD>#cluster0-shard-00-00-nnn.gcp.mongodb.net:27017,cluster0-shard-00-01-nnn.gcp.mongodb.net:27017,cluster0-shard-00-02-nnn.gcp.mongodb.net:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true
When I run my node.js program locally, I can read and write the database. However, when I deploy to google cloud, I'm having connecting to it. This is what I see in Google Console:
Mongoose connection "error" event fired with:
{ MongoError: connection 5 to cluster0-shard-00-02-32tq4.mongodb.net:27017 closed
at Function.MongoError.create (/srv/node_modules/mongoose/node_modules/mongodb-core/lib/error.js:29:11)
at TLSSocket.<anonymous> (/srv/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:214:22)
at Object.onceWrapper (events.js:315:30)
at emitOne (events.js:121:20)
at TLSSocket.emit (events.js:211:7)
at _handle.close (net.js:557:12)
at Socket.done (_tls_wrap.js:356:7)
at Object.onceWrapper (events.js:315:30)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at TCP._handle.close [as _onclose] (net.js:557:12)
I've tried to add allow-mongodb to firewall with protocols and ports as tcp:27017, direction ingress, IP ranges 0.0.0.0/0 and it's still not working. Can someone tell me what I'm missing here?
P.S I'm kinda suspecting that I have to add my mongodb cluster's IP to firewall as well though I'm not sure
I am having this problem while connecting the elasticsearch connections.
var elasticsearch = require('elasticsearch');
var client = new elasticsearch.Client({ host: 'localhost:9200',
log: 'trace'});
Elasticsearch ERROR: 2016-07-19T19:09:26Z
Error: Request error, retrying -- connect ECONNREFUSED 127.0.0.1:9200
at Log.error (/root/git_build/FirstMoveChess/node_modules/elasticsearch/src/lib/log.js:225:56)
at checkRespForFailure (/root/git_build/FirstMoveChess/node_modules/elasticsearch/src/lib/transport.js:195:18)
at HttpConnector. (/root/git_build/FirstMoveChess/node_modules/elasticsearch/src/lib/connectors/http.js:154:7)
at ClientRequest.bound (/root/git_build/FirstMoveChess/node_modules/lodash-node/modern/internals/baseBind.js:56:17)
at emitOne (events.js:96:13)
at ClientRequest.emit (events.js:188:7)
at Socket.socketErrorListener (_http_client.js:308:9)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at emitErrorNT (net.js:1272:8)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)
Elasticsearch TRACE: 2016-07-19T19:09:27Z
-> HEAD http://localhost:9200/
I understand this question is quite old but I wanted to share how you can solve this problem.
If you are locally using elasticsearch
First thing you must do is running elasticsearch on your machine.
Error: Request error, retrying -- connect ECONNREFUSED 127.0.0.1:9200
Because above message indicates that you're not running elasticsearch locally.
So, visit the link and follow the insturction.
Docker environment
It gets much trickier here.
First, follow the instruction here.
And in case you're using node.js elasticsearch client, you have to specify elasticsearch host as 172.24.0.1.
If you use container_name or private IP of container in docker-compose.yml, it won't work.
In case of Docker Container Environment after changing from http://localhost:9200 to http://ipaddress:9200 in docker-compose.yml
please change the following live in docker-compose.yml that is related to CORS
Change this
** http.cors.allow-origin=/https?://localhost(:[0-9]+)?/ **
into this
*- http.cors.allow-origin= **