I have a kafka-node ("kafka-node": "^2.6.1",) client, which successfully produces messages when i run node app in localhost and connect with remote_address:2181. But the same code produces below error when deploy to a test server.
BrokerNotAvailableError: Broker not available
0|www | at new BrokerNotAvailableError (/var/www/backend/node_modules/kafka-node/lib/errors/BrokerNotAvailableError.js:11:9)
0|www | at Client.loadMetadataForTopics (/var/www/backend/node_modules/kafka-node/lib/client.js:371:15)
0|www | at Client.send (/var/www/backend/node_modules/kafka-node/lib/client.js:542:10)
0|www | at /var/www/backend/node_modules/kafka-node/lib/client.js:240:10
0|www | at /var/www/backend/node_modules/kafka-node/node_modules/async/dist/async.js:473:16
0|www | at iteratorCallback (/var/www/backend/node_modules/kafka-node/node_modules/async/dist/async.js:1064:13)
0|www | at /var/www/backend/node_modules/kafka-node/node_modules/async/dist/async.js:969:16
0|www | at buildRequest (/var/www/backend/node_modules/kafka-node/lib/client.js:256:24)
0|www | at /var/www/backend/node_modules/kafka-node/node_modules/async/dist/async.js:3110:16
0|www | at eachOfArrayLike (/var/www/backend/node_modules/kafka-node/node_modules/async/dist/async.js:1069:9)
0|www | at eachOf (/var/www/backend/node_modules/kafka-node/node_modules/async/dist/async.js:1117:5)
0|www | at Object.eachLimit (/var/www/backend/node_modules/kafka-node/node_modules/async/dist/async.js:3172:5)
0|www | at Client.sendProduceRequest (/var/www/backend/node_modules/kafka-node/lib/client.js:238:9)
0|www | at HighLevelProducer.BaseProducer.send (/var/www/backend/node_modules/kafka-node/lib/baseProducer.js:120:10)
0|www | at /var/www/backend/clients/3rdparty/kafka.js:104:16
Possible causes could be the remoteServer blocks incoming requests at port 2181, but i am able to telnet at remote_address:2181. Here is my code:
const clientConfig = {
connectionString: process.env.KAFKA_HOST,
clientId: process.env.KAFKA_CLIENTID || 'nodejs-kafka-client'
}
client = new kafka.Client(clientConfig.connectionString,
clientConfig.clientId)
producer = new kafka.HighLevelProducer(client, {})
bindListeners()
I read about a similar issue issue on kafka-node repo. The guy who raised the issue, seemed to resolve it using some zookeeper config. I use the default config of zookeeper. I am not sure if that's one of the reason, as i am able to connect with my localhost.
I added kafka IP in /etc/hosts
After that I was able to solve this issue
For all of you guys, trying out kafka-node.
Try using kafka.KafkaClient() instead of kafka.client(). It uses, Kafka-broker directly, instead of connecting via zookeeper.
Although, i am sure zookeeper configuration can also be updated to support kafka.client() way. But i am not sure, what is the actual configuration.
Related
I'm trying to setup node.js and Elasticsearch together in a docker container,
I've been following this post to the letter,until to the section of launching the application
when I type
docker-compose up
I keep getting this node.js error that is saying models module is not found
tqd-node | { Error: Cannot find module '../models'
this is the first few lines of the error message
tqd-elasticsearch is up-to-date
Starting tqd-node ... done
Attaching to tqd-elasticsearch, tqd-node
tqd-elasticsearch | WARNING: no logs are available with the 'none' log driver
tqd-node |
tqd-node | > nodejselastic#1.0.0 start /usr/src/app
tqd-node | > pm2-runtime start ./src/main.js --name node_app
tqd-node |
tqd-node | 2019-12-14T23:09:56: PM2 log: Launching in no daemon mode
tqd-node | 2019-12-14T23:09:56: PM2 log: App [node_app:0] starting in -fork mode-
tqd-node | 2019-12-14T23:09:56: PM2 log: App [node_app:0] online
tqd-node | { Error: Cannot find module '../models'
tqd-node | at Function.Module._resolveFilename (internal/modules/cjs/loader.js:582:15)
tqd-node | at Module.Hook._require.Module.require (/usr/local/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:51:29)
so I downloaded the full code from this GitHub repository(the-quotes-database)
but I still got the same error
I did try to change the path in controller file here
const model = require("../models");
I thought maybe it needs the full c:// path, but that didn't change the error message.
when I do hover over models in VSC editor it shows that it is the correct path.
so my question is : is this a bug in the repo or is the issue with my windows docker configuration?
I successfully managed to create simple python containers with docker,
so I'm not sure what is the root cause of this error and how to fix it.
update: I will answer here in case anyone finds a similar issue.
I installed docker and downloaded the files and repeated the steps on my Ubuntu laptop and it worked!
so the downloaded files were not prepared to work on Windows, it is not a bug in the repo if you have Linux.
I've been trying to dockerize a Sails 1.0.2 app to be able to run it locally with docker-compose, but it doesn't seem to be able to connect to PostgreSQL server by any means.
Also, I think I might be missing some concepts on how DB creation works with Sails. I've tried either model "safe" behaviour (for manual migrations), "alter" and "drop".
When running safe mode, I tried using sails-migrations db:create, or sails-migrations migrate in a separate task within the container to check if any of them would be necessary for setting up the database, even though I still have only a scaffold generated User.js model.
None of them seem to work. On the other way, the Docker database itself looks like it was created. Any help would be appreciated, thanks in advance.
Versions I'm using:
Sails: 1.0.2
PostgreSQL: 9.6
sails-migrations: 2.1.0
sails-postgresql: 1.0.0
File List:
docker-compose.yml
version: '2'
volumes:
postgres_data_dev: {}
postgres_backup_dev: {}
services:
postgres:
build: ./compose/postgres
volumes:
- postgres_data_dev:/var/lib/postgresql/data
- postgres_backup_dev:/backups
environment:
- POSTGRES_USER=test_project
sails:
build:
context: .
dockerfile: ./compose/sails/Dockerfile
command: /start-dev.sh
depends_on:
- postgres
environment:
- POSTGRES_USER=test_project
- USE_DOCKER=yes
volumes:
- .:/app
ports:
- "8000:8000"
links:
- postgres
compose/sails/Dockerfile
FROM node:latest
RUN npm install -g sails grunt npm-check-updates
COPY ./package.json /package.json
RUN npm install
RUN npm install --save sails-postgresql
COPY ./compose/sails/entrypoint.sh /entrypoint.sh
RUN sed -i 's/\r//' /entrypoint.sh
RUN chmod +x /entrypoint.sh
COPY ./compose/sails/start-dev.sh /start-dev.sh
RUN sed -i 's/\r//' /start-dev.sh
RUN chmod +x /start-dev.sh
WORKDIR /app
ENTRYPOINT ["/entrypoint.sh"]
compose/sails/entrypoint.sh
#!/bin/bash
set -e
cmd="$#"
export REDIS_URL=redis://redis:6379
# the official postgres image uses 'postgres' as default user if not set explictly.
if [ -z "$POSTGRES_USER" ]; then
export POSTGRES_USER=postgres
fi
export DATABASE_URL=postgres://$POSTGRES_USER:$POSTGRES_PASSWORD#postgres:5432/$POSTGRES_USER
function postgres_ready(){
node << END
var postgres = require("pg");
var client = new postgres.Client({
user: '$POSTGRES_USER',
password: '$POSTGRES_PASSWORD',
database: '$POSTGRES_USER',
port: 5432,
host: 'postgres',
});
client.connect(function(err) {
if (err) {
console.log('connection failed');
console.error(err);
process.exit(1);
}
console.log('connection successful');
process.exit(0);
});
END
}
until postgres_ready; do
>&2 echo "Postgres is unavailable - sleeping"
sleep 1
done
>&2 echo "Postgres is up - continuing..."
exec $cmd
compose/sails/start-dev.sh
sails lift --port 8000 --verbose
compose/postgres/Dockerfile
FROM postgres:9.6
# add backup scripts
ADD backup.sh /usr/local/bin/backup
ADD restore.sh /usr/local/bin/restore
ADD list-backups.sh /usr/local/bin/list-backups
# make them executable
RUN chmod +x /usr/local/bin/restore
RUN chmod +x /usr/local/bin/list-backups
RUN chmod +x /usr/local/bin/backup
config/datastores.js
default: {
adapter: "sails-postgresql",
user: "test_project",
host: "localhost",
},
Notes and error outputs
Note: I've also tried datastores url: 'postgresql://test_project#localhost:5432' and 'postgresql://test_project#localhost:5432/postgres' instead of separate declarations for each attribute on the connection string.
Although the database created by Docker seems to be fine, I keep getting either ECONNREFUSED (probably whenever I play around with docker DB or DB connection's parameters to try and make it work), or the following error (when it connects to the DB successfully):
error: Could not tear down the ORM hook.
Error details: Error: Invalid data store identity.
No data store exist with that identity.
The error's full output can be found down below (when using model alter strategy):
sails_1 | Postgres is unavailable - sleeping
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | LOG: database system was shut down at 2018-09-05 13:45:04 UTC
postgres_1 | LOG: MultiXact member wraparound protections are now enabled
postgres_1 | LOG: database system is ready to accept connections
postgres_1 | LOG: autovacuum launcher started
sails_1 | connection successful
sails_1 | Postgres is up - continuing...
sails_1 |
sails_1 | info: Starting app...
sails_1 |
sails_1 | verbo: Using locally-installed Sails.
sails_1 | verbo: • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
sails_1 | verbo: • Loading Sails with "verbose" logging enabled... •
sails_1 | verbo: • (For even more details, try "silly".) •
sails_1 | verbo: • •
sails_1 | verbo: • http://sailsjs.com/config/log •
sails_1 | verbo: • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
sails_1 | verbo: moduleloader hook loaded successfully. (0ms)
sails_1 | verbo: userconfig hook loaded successfully. (104ms)
sails_1 | verbo: Exposing global variables... (you can customize/disable this by modifying the properties in `sails.config.globals`. Set it to `false` to disable all globals.)
sails_1 | verbo: userhooks hook loaded successfully. (8669ms)
sails_1 | verbo: logger hook loaded successfully. (4ms)
sails_1 | verbo: request hook loaded successfully. (0ms)
sails_1 | verbo: views hook loaded successfully. (46ms)
sails_1 | verbo: responses hook loaded successfully. (49ms)
sails_1 | verbo: helpers hook loaded successfully. (11ms)
sails_1 | verbo: policies hook loaded successfully. (9ms)
sails_1 | verbo: services hook loaded successfully. (1ms)
sails_1 | verbo: security hook loaded successfully. (4ms)
sails_1 | verbo: i18n hook loaded successfully. (19ms)
sails_1 | verbo: session hook loaded successfully. (149ms)
sails_1 | verbo: http hook loaded successfully. (1341ms)
sails_1 | info: Initializing hook... (`api/hooks/custom`)
sails_1 | verbo: Some optional settings have not been configured yet:
sails_1 | ---------------------------------------------------------------------
sails_1 | No `sails.config.custom.stripeSecret` was configured.
sails_1 | No `sails.config.custom.stripePublishableKey` was configured.
sails_1 | No `sails.config.custom.mailgunSecret` was configured.
sails_1 | No `sails.config.custom.mailgunDomain` was configured.
sails_1 |
sails_1 | Until this is addressed, this app's billing and email features
sails_1 | will be disabled and/or hidden in the UI.
sails_1 |
sails_1 | [?] If you're unsure or need advice, come by https://sailsjs.com/support
sails_1 | ---------------------------------------------------------------------
sails_1 | verbo: custom hook loaded successfully. (1ms)
sails_1 | info: Initializing `apianalytics` hook... (requests to monitored routes will be logged!)
sails_1 | verbo: apianalytics hook loaded successfully. (0ms)
sails_1 | verbo: grunt hook loaded successfully. (11ms)
sails_1 | verbo: organics hook loaded successfully. (135ms)
sails_1 | verbo: sockets hook loaded successfully. (205ms)
sails_1 | verbo: Loading adapter (`sails-postgresql`) from this app's `node_modules/` directory...
sails_1 | info: ·• Auto-migrating... (alter)
sails_1 | info: Hold tight, this could take a moment.
sails_1 | error: A hook (`orm`) failed to load!
sails_1 | verbo: Lowering sails...
sails_1 | error: Could not tear down the ORM hook. Error details: Error: Invalid data store identity. No data store exist with that identity.
sails_1 | at Object.teardown (/app/node_modules/sails-postgresql/helpers/teardown.js:60:26)
sails_1 | at wrapper (/app/node_modules/#sailshq/lodash/lib/index.js:3275:19)
sails_1 | at Deferred.parley.retry [as _handleExec] (/app/node_modules/machine/lib/private/help-build-machine.js:1076:19)
sails_1 | at Deferred.exec (/app/node_modules/parley/lib/private/Deferred.js:286:10)
sails_1 | at Deferred.switch (/app/node_modules/machine/lib/private/help-build-machine.js:1469:16)
sails_1 | at teardownDatastore (/app/node_modules/sails-postgresql/lib/adapter.js:96:18)
sails_1 | at /app/node_modules/async/dist/async.js:3047:20
sails_1 | at replenish (/app/node_modules/async/dist/async.js:884:21)
sails_1 | at /app/node_modules/async/dist/async.js:888:13
sails_1 | at eachLimit$1 (/app/node_modules/async/dist/async.js:3136:26)
sails_1 | at Object.<anonymous> (/app/node_modules/async/dist/async.js:920:20)
sails_1 | at Object.teardown (/app/node_modules/sails-postgresql/lib/adapter.js:91:13)
sails_1 | at /app/node_modules/waterline/lib/waterline.js:758:27
sails_1 | at /app/node_modules/async/dist/async.js:3047:20
sails_1 | at eachOfArrayLike (/app/node_modules/async/dist/async.js:1002:13)
sails_1 | at eachOf (/app/node_modules/async/dist/async.js:1052:9)
sails_1 | at Object.eachLimit (/app/node_modules/async/dist/async.js:3111:7)
sails_1 | at Object.teardown (/app/node_modules/waterline/lib/waterline.js:742:11)
sails_1 | at Hook.teardown (/app/node_modules/sails-hook-orm/index.js:246:30)
sails_1 | at Sails.wrapper (/app/node_modules/#sailshq/lodash/lib/index.js:3275:19)
sails_1 | at Object.onceWrapper (events.js:273:13)
sails_1 | at Sails.emit (events.js:182:13)
sails_1 | at Sails.EventEmitter.emit (domain.js:442:20)
sails_1 | at Sails.emitter.emit (/app/node_modules/sails/lib/app/private/after.js:56:26)
sails_1 | at /app/node_modules/sails/lib/app/lower.js:67:11
sails_1 | at beforeShutdown (/app/node_modules/sails/lib/app/lower.js:45:12)
sails_1 | at Sails.lower (/app/node_modules/sails/lib/app/lower.js:49:3)
sails_1 | at Sails.wrapper [as lower] (/app/node_modules/#sailshq/lodash/lib/index.js:3275:19)
sails_1 | verbo: (The error above was logged like this because `sails.hooks.orm.teardown()` encountered an error in a code path where it was invoked without providing a callback.)
sails_1 | error:
sails_1 | error: Exception: `registerDataStore` failed ("badConfiguration"). The configuration was invalid. (Also got an additional error -- see `.raw`).
sails_1 | at Object.registerDatastore (/app/node_modules/sails-postgresql/lib/adapter.js:56:17)
sails_1 | at /app/node_modules/waterline/lib/waterline.js:714:27
sails_1 | at /app/node_modules/async/dist/async.js:3047:20
sails_1 | at eachOfArrayLike (/app/node_modules/async/dist/async.js:1002:13)
sails_1 | at eachOf (/app/node_modules/async/dist/async.js:1052:9)
sails_1 | at Object.eachLimit (/app/node_modules/async/dist/async.js:3111:7)
sails_1 | at Object.initialize (/app/node_modules/waterline/lib/waterline.js:650:11)
sails_1 | at buildOntologyAndRunAutoMigrations (/app/node_modules/sails-hook-orm/lib/build-ontology-and-run-auto-migrations.js:55:7)
sails_1 | at async.auto._buildOntology (/app/node_modules/sails-hook-orm/lib/initialize.js:456:7)
sails_1 | at runTask (/app/node_modules/async/dist/async.js:1660:17)
sails_1 | at /app/node_modules/async/dist/async.js:1602:17
sails_1 | at processQueue (/app/node_modules/async/dist/async.js:1612:17)
sails_1 | at taskComplete (/app/node_modules/async/dist/async.js:1630:13)
sails_1 | at /app/node_modules/async/dist/async.js:1653:21
sails_1 | at /app/node_modules/async/dist/async.js:339:31
sails_1 | at /app/node_modules/async/dist/async.js:847:20
sails_1 | at async.auto._checkAdapterCompatibility (/app/node_modules/sails-hook-orm/lib/initialize.js:428:14)
sails_1 | at runTask (/app/node_modules/async/dist/async.js:1660:17)
sails_1 | at /app/node_modules/async/dist/async.js:1602:17
sails_1 | at processQueue (/app/node_modules/async/dist/async.js:1612:17)
sails_1 | at taskComplete (/app/node_modules/async/dist/async.js:1630:13)
sails_1 | at /app/node_modules/async/dist/async.js:1653:21
sails_1 |
sails_1 | error: Could not load Sails app.
sails_1 | error:
sails_1 | error: Tips:
sails_1 | error: • First, take a look at the error message above.
sails_1 | error: • Make sure you've installed dependencies with `npm install`.
sails_1 | error: • Check that this app was built for a compatible version of Sails.
sails_1 | error: • Have a question or need help? (http://sailsjs.com/support)
Your datastores.js references the database host as localhost, it should be the name of the docker service postgres when running from the compose file. The migration part in compose/sails/entrypoint.sh references the host as postgres
I’m trying to configure an ecosystem with RealmObjectServer + Postgresql + DataAdapter using Docker Compose, but I can’t connect the Data Adapter server with the ROS because the adapter requires the HTTPS protocol. It keeps showing this error on my ros-adapter container:
ros-adapter_1 | Error: only http(s) protocols are supported
ros-adapter_1 | at /usr/src/app/node_modules/node-fetch/index.js:58:10
ros-adapter_1 | at new Promise (<anonymous>)
ros-adapter_1 | at new Fetch (/usr/src/app/node_modules/node-fetch/index.js:49:9)
ros-adapter_1 | at Fetch (/usr/src/app/node_modules/node-fetch/index.js:37:10)
ros-adapter_1 | at next (/usr/src/app/node_modules/realm/lib/user-methods.js:57:9)
ros-adapter_1 | at Promise (/usr/src/app/node_modules/realm/lib/user-methods.js:72:13)
ros-adapter_1 | at new Promise (<anonymous>)
ros-adapter_1 | at performFetch (/usr/src/app/node_modules/realm/lib/user-methods.js:70:16)
ros-adapter_1 | at refreshAdminToken (/usr/src/app/node_modules/realm/lib/user-methods.js:129:5)
ros-adapter_1 | at refreshAccessToken (/usr/src/app/node_modules/realm/lib/user-methods.js:169:16)
Does anyone know how I can configure this properly in Realm.Adapter?
You can see the source code here: https://github.com/artutra/ros-adapter
I am using sane-cli in an ember app and when I try to use sane up inside the sails folder. I am getting this error. Can anyone help me to fix it?
I have already tried sudo npm install inside the server folder and globally.
Error log (image):
client | Could not start watchman
client | was unable to use: "events", fell back to: "polling"
server | debug: initializing sails-hook-dev...
server | events.js:160
server | throw er; // Unhandled 'error' event
server | ^
server |
server | Error: watch /home/ayush/Code/sails/server/api/blueprints EPERM
server | at exports._errnoException (util.js:1018:11)
server | at FSWatcher.start (fs.js:1443:19)
server | at Object.fs.watch (fs.js:1470:11)
server | at createFsWatchInstance
(/home/ayush/Code/sails/server/node_modules/chokidar/lib/nodefs-
handler.js:37:15)
server | at setFsWatchListener
(/home/ayush/Code/sails/server/node_modules/chokidar/lib/nodefs-
handler.js:80:15)
server | at FSWatcher.NodeFsHandler._watchWithNodeFs
(/home/ayush/Code/sails/server/node_modules/chokidar/lib/nodefs-
handler.js:228:14)
server | at FSWatcher.NodeFsHandler._handleDir
(/home/ayush/Code/sails/server/node_modules/chokidar/lib/nodefs-
handler.js:407:19)
server | at FSWatcher.<anonymous>
(/home/ayush/Code/sails/server/node_modules/chokidar/lib/nodefs-
handler.js:455:19)
server | at FSWatcher.<anonymous>
(/home/ayush/Code/sails/server/node_modules/chokidar/lib/nodefs-
handler.js:460:16)
server | at FSReqWrap.oncomplete (fs.js:123:15)
client | Proxying to http://127.0.0.1:1337
client | Livereload server on http ://localhost:49153
client | Serving on http ://localhost:4200/
I'm running three node.js applications on one server, all using Foreman to start up the apps through always.js.
Under zero load, one of my apps consistently throws the error Error: watch EMFILE and restarts. That application still works though, despite constantly throwing that error... I've tried to find more information about this error, but there's not a whole lot out there ("too many files open" or "increase ulimit".)
My question is: why would this be happening on an idle web application - and why just one out of three? It's not doing anything... Is it an issue with always.js? (There are two other node apps running through always on this machine though...) Just looking for some info as to what is causing this error, if it's serious, and how it can be resolved.
Thanks!
FYI, here is a relevant excerpt from the console:
01:17:07 web.1 | app listening on http://0.0.0.0:5000
01:17:07 web.1 | NODE_ENV = development
01:17:07 web.1 | opened connection to database!
01:17:07 web.1 | [always] Error: watch EMFILE
01:17:07 web.1 | [always] Error: watch EMFILE
01:17:07 web.1 | at errnoException (fs.js:636:11)
01:17:07 web.1 | at FSWatcher.start (fs.js:663:11)
01:17:07 web.1 | at Object.watch (fs.js:691:11)
01:17:07 web.1 | at Object.oncomplete (/home/jesse/local/nodev0.6.14/lib/node_modules/always/lib/monitor.js:62:36)
01:17:07 web.1 | [always] Restarting app.js with Node
I don't have experience with always.js but got similar errors with other node utilities that use the fs.watch command, which in turn uses linux's inotify API.
You might want to try checking the settings for inotify then:
cat /proc/sys/fs/inotify/max_user_instances
If the file exists, setting it to a higher value might help:
echo 8704 > /proc/sys/fs/inotify/max_user_instances