I'm running a CouchDB server with docker and I'm trying to POST data through a Node app.
But I'm frequently prompted with a ESOCKETTIMEDOUT error (not always).
Here's the way I'm opening the connexion to the DB:
var remoteDB = new PouchDB('http://localhost:5984/dsndatabase', {
ajax: {
cache: true,
timeout: 40000 // I tried a lot of durations
}
});
And here's the code used to send the data :
exports.sendDatas = function(datas,db, time) {
console.log('> Export vers CouchDB')
db.bulkDocs(datas).then(function () {
return db.allDocs({include_docs: true});
}).then(function (){
var elapsedTime = new Date().getTime() - time;
console.log('> Export terminé en ', elapsedTime, ' ms');
}).catch(function (err) {
console.log(err);
})
};
The error doesn't show up every time but I'm unable to find a pattern.
And, timeout or not, all my data is successfully loaded in my CouchDB !
I've seen a lot of posts on this issue but none of them really answers my question ...
Any idea ?
Okay this seems to be real issue:
https://github.com/pouchdb/pouchdb/issues/3550#issuecomment-75100690
I think you can fix it by stating a reasonably longer timeout value/a retry logic using exponential backoff.
Let me know if that works for you.
Related
I'm building a small GEO application and it's using http://tile38.com/ and https://www.npmjs.com/package/tile38 node module. Everything working fine but I'm unable to get result from the NEARBY query from the node module. It looks like the call back function isn't working, I've spent lot of time but couldn't find a way out. What I want is get the result from the nearby query and assign to a variable.
Here is the code:
var Tile38 = require('tile38');
var client = new Tile38({host: 'localhost', port: 9851, debug: true });
// set a simple lat/lng coordinate
client.set('fleet', 'truck1', [33.5123, -112.2693])
// set with additional fields
client.nearbyQuery('fleet').distance().point(33.5123, -112.2693, 6000).execute((err, results) => {
console.log("########");
// this callback will be called multiple times
if (err) {
console.error("something went wrong! " + err);
} else {
console.log(results + "##########");
}
});;
but when I try the following simple query it's working fine.
client.get('fleet', 'truck1').then(data => {
console.log(data); // prints coordinates in geoJSON format
}).catch(err => {
console.log(err); // id not found
});
ALSO when I try the RAW query in the tile38-cli it's working fine.
NEARBY fleet POINT 33.5123 -112.2693 6000
Any help would appreciated.
Thanks in advance.
EDIT
I tried the following also but didn't work.
let query = client.nearbyQuery('fleet').distance().point(33.5123, -112.2693, 6000)
query.execute(function(results).then(results => {
console.dir(results); // results is an object.
}))
Receiving following error
query.execute(function(results).then(results => {
^
SyntaxError: Unexpected token .
the author of the node library for Tile38 here. Sorry for the trouble getting this working. I noticed a typo in the readme which may have thrown you off. I will correct this.
The execute() method returns a Promise, and (as you already figured out) the example should have stated
query.execute().then(results => {
console.dir(results);
});
instead of
query.execute(function(results).then(results => {
console.dir(results);
});
After long time debugging I found that following code is working:
let query = client.nearbyQuery('fleet').distance().point(33.5123, -112.2693, 6000)
query.execute().then(data => {
console.dir(results); // results is an object.
}))
So I've set up a node.js backend that is to be used to move physical items in our warehouse. The database hosting our software is oracle, and our older version of this web application is written in PHP which works fine, but has some weird glitches and is slow as all hell.
The node.js backend works fine for moving single items, but once I try moving a box (which will then move anything from 20-100 items), the entire backend stops at the .commit() part.
Anyone have any clue as to why this happens, and what I can do to remedy it? Suggestions for troubleshooting would be most welcome as well!
Code:
function move(barcode,location) {
var p = new Promise(function(resolve,reject) {
console.log("Started");
exports.findOwner(barcode).then(function(data) {
console.log("Got data");
// console.log(barcode);
var type = data[0];
var info = data[1];
var sql;
sql = "update pitems set location = '"+location+"' where barcode = '"+barcode+"' and status = 0"; // status = 0 is goods in store.
ora.norway.getConnection(function(e,conn) {
if(e) {
reject({"status": 0, "msg": "Failed to get connection", "error":e});
}
else {
console.log("Got connection");
conn.execute(sql,[],{}, function(err,results) {
console.log("Executed");
if(err) {
conn.release();
reject({"status": 0, "msg": "Failed to execute sql"+sql, "error": err});
}
else {
console.log("Execute was successfull"); // This is the last message logged to the console.
conn.commit(function(e) {
conn.release(function(err) {
console.log("Failed to release");
})
if(e) {
console.log("Failed to commit!");
reject({"status": 0, "msg": "Failed to commit sql"+sql, "error": e});
}
else {
console.log("derp6");
resolve({"status": 1, "msg": "Relocated "+results.rowsAffected+" items."});
}
});
}
});
}
});
});
});
return p;
}
Please be aware that your code is open to SQL injection vulnerabilities. Even more so since you posted it online. ;)
I recommend updating your statement to something like this:
update pitems
set location = :location
where barcode = :barcode
and status = 0
Then update your conn.execute as follows:
conn.execute(
sql,
{
location: location,
barcode: barcode
},
{},
function(err,results) {...}
);
Oracle automatically escapes bind variables. But there's another advantage in that you'll avoid hard parses when the values of the bind variables change.
Also, I'm happy to explore the issue you're encountering more with commit. But it would really help if you could provide a reproducible test case that I could run on my end.
I think this is an issue on the database level, an update on multiple items without providing an ID is maybe not allowed.
You should do two things:
1) for debugging purposes, add console.log(JSON.stringify(error)) where you expect an error. Then you'll find the error that your database provides back
2) at the line that says
conn.release(function(err) {
console.log("Failed to release");
})
Check if err is defined:
conn.release(function(err) {
if(err){
console.log("Failed to release");
}
else{console.log("conn released");}
})
This sounds like similar to an issue that I'm having. Node.js is hanging while updating oracle db using oracledb library. It looks like when there are 167 updates to make, it works fine. The program hangs when I have 168 updates. The structure of the program goes like this:
When 168 records from local sqlite db, for each record returned as callback from sqlite: 1.) get an Oracle connection; 2.) do 2 updates to two tables (one update to each table with an autoCommit on the latter execute). All 1st update completed but none can start the second update. They are just hanging there. With 167 records, they will run to completion.
The strange thing observed is that none of the 168 could get started on 2nd update (they finished 1st update) so some will have a chance to go forward to commit. It looks like they are all queued up in some way.
Given the following node.js + oriento sample code, I have an issue of running into a timeout, [OrientDB.ConnectionError [2]: read ETIMEDOUT], first time I make a DB query after a longish inactivity period. Right after the timeout error the connection is somehow re-initialized and the next query runs fine.
var oriento = require("oriento"),
server = oriento({...}),
db = server.use("users");
var getData = function(statement, opts, callback) {
db.query(statement, opts).then(function(data) {
callback(null, data);
}).catch(callback);
};
So I have the following questions:
Is this the right way to go or should I call oriento({...}).use("users") every time I make a query rather than reusing the connection object?
If this is the right way, why the connection is not validated and refreshed automatically?
How can I manually check that I am not going to run into a timeout (i.e. validate the connection) and force a connection refresh?
Any suggestions better than the following fairly ugly hack to keep the transport socket by pinging the DB every minute?
setInterval(function(db) {
db.query("select from user").then(function(data) {
console.log("still alive");
}).catch(function(err) {
console.error(err);
});
}, 60000, db);
I have a web app that is published via ExpressJS on NodeJS, of course. It uses CouchDB as it's data source. I implemented long polling to keep the app in sync at all times between all users. To accomplish this I use the following logic:
User logs into app and an initial long poll request is made to Node via an Express route.
Node in turn makes a long poll request to CouchDB.
When Couch is updated it responds to the request from Node.
Lastly Node responds to the browser.
Simple. What is happening, though, is that when I refresh the browser it freezes up on every fifth refresh. Huh? very wierd. But I can reproduce it over and over, even in my test environment. Every fifth refresh without fail freezes up Node and causes the app to freeze. Rebooting Node fixes the issue.
After much hair pulling I THOUGHT I solved it by changing this:
app.get('/_changes/:since*', security, routes.changes);
To this:
app.get('/_changes/:since*', security, function () { routes.changes });
However, after further testing this is just failing to run routes.changes. So no actual solution. Any ideas why long polling CouchDb from Node would do such a strange thing? On the fifth refresh I can have a break point in node on the first line of my routing code and it never get's hit. However, in the browser I can break on the request to node for long polling and it seems to go out. It's like Node is not accepting the connection for some reason...
Should I be approaching long polling from Node to CouchDB in a different way? I'm using feed=longpoll, should I maybe be doing feed=continuous? If I turn down the changes_timeout in couchdb to 5 seconds it doesn't get rid of the issue, but it does make it easier to cope with since the freezes only last 5 seconds tops. So this would seem to indicate that node can't handle having several outstanding requests to couch. Maybe I will try a continuous feed and see what happens.
self.getChanges = function (since) {
Browser:
$.ajax({
url: "/_changes/" + since,
type: "GET", dataType: "json", cache: false,
success: function (data) {
try {
self.processChanges(data.results);
self.lastSeq(data.last_seq);
self.getChanges(self.lastSeq());
self.longPollErrorCount(0);
} catch (e) {
self.longPollErrorCount(self.longPollErrorCount() + 1);
if (self.longPollErrorCount() < 10) {
setTimeout(function () {
self.getChanges(self.lastSeq());
}, 3000);
} else {
alert("You have lost contact with the server. Please refresh your browser.");
}
}
},
error: function (data) {
self.longPollErrorCount(self.longPollErrorCount() + 1);
if (self.longPollErrorCount() < 10) {
setTimeout(function () {
self.getChanges(self.lastSeq());
}, 3000);
} else {
alert("You have lost contact with the server. Please refresh your browser.");
}
}
});
}
Node:
Routing:
exports.changes = function (req, res) {
var args = {};
args.since = req.params.since;
db.changes(args, function (err, body, headers) {
if (err) {
console.log("Error retrieving changes feed: "+err);
res.send(err.status_code);
} else {
//send my response... code removed here
}
})
}
Database long poll calls:
self.changes = function (args, callback) {
console.log("changes");
if (args.since == 0) {
request(self.url + '/work_orders/_changes?descending=true&limit=1', function (err, res, headers) {
var body = JSON.parse(res.body);
var since = body.last_seq;
console.log("Since change: "+since);
self.longPoll(since, callback);
});
} else {
self.longPoll(args.since, callback);
}
}
self.longPoll = function (since, callback) {
console.log("about to request with: "+since);
request(self.url + '/work_orders/_changes?feed=continuous&include_docs=true&since=' + since,
function (err, res, headers) {
console.log("finished request.")
if (err) { console.log("Error starting long poll: "+err.reason); return; } //if err send it back
callback(err, res.body);
});
}
Socket.io will automatically fall back to long polling, and doesn't have a problem like the one you are having. So just use that. Also for CouchDB changes use this https://github.com/iriscouch/follow or maybe this one https://npmjs.org/package/changes like the other guy suggested.
Its very bad practice to reinvent things when we have popular modules that already do what you need. There are currently more than 52,000 node modules on https://npmjs.org/. People make a big deal about copying and pasting code. In my mind reinventing basic stuff is even worse than that.
I know with so many modules its hard to know about all of them, so I'm not saying you can never solve the same problem as someone else. But take a look at npmjs.org first and also sites like http://node-modules.com/ which may give better search results.
Installed NodeJS on Raspberry Pi, is there a way to check if the rPi is connected to the internet via NodeJs ?
A quick and dirty way is to check if Node can resolve www.google.com:
require('dns').resolve('www.google.com', function(err) {
if (err) {
console.log("No connection");
} else {
console.log("Connected");
}
});
This isn't entire foolproof, since your RaspPi can be connected to the Internet yet unable to resolve www.google.com for some reason, and you might also want to check err.type to distinguish between 'unable to resolve' and 'cannot connect to a nameserver so the connection might be down').
While robertklep's solution works, it is far from being the best choice for this. It takes about 3 minutes for dns.resolve to timeout and give an error if you don't have an internet connection, while dns.lookup responds almost instantly with the error ENOTFOUND.
So I made this function:
function checkInternet(cb) {
require('dns').lookup('google.com',function(err) {
if (err && err.code == "ENOTFOUND") {
cb(false);
} else {
cb(true);
}
})
}
// example usage:
checkInternet(function(isConnected) {
if (isConnected) {
// connected to the internet
} else {
// not connected to the internet
}
});
This is by far the fastest way of checking for internet connectivity and it avoids all errors that are not related to internet connectivity.
I had to build something similar in a NodeJS-app some time ago. The way I did it was to first use the networkInterfaces() function is the OS-module and then check if one or more interfaces have a non-internal IP.
If that was true, then I used exec() to start ping with a well-defined server (I like Google's DNS servers). By checking the return value of exec(), I know if ping was sucessful or not. I adjusted the number of pings based on the interface type. Forking a process introduces some overhead, but since this test is not performed too frequently in my app, I can afford it. Also, by using ping and IP-adresses, you dont depend on DNS being configured. Here is an example:
var exec = require('child_process').exec, child;
child = exec('ping -c 1 128.39.36.96', function(error, stdout, stderr){
if(error !== null)
console.log("Not available")
else
console.log("Available")
});
It's not as foolproof as possible but get the job done:
var dns = require('dns');
dns.lookupService('8.8.8.8', 53, function(err, hostname, service){
console.log(hostname, service);
// google-public-dns-a.google.com domain
});
just use a simple if(err) and treat the response adequately. :)
ps.: Please don't bother telling me 8.8.8.8 is not a name to be resolved, it's just a lookup for a highly available dns server from google. The intention is to check connectivity, not name resolution.
Here is a one liner: (Node 10.6+)
let isConnected = !!await require('dns').promises.resolve('google.com').catch(()=>{});
Since I was concerned with DNS cache in other solutions here, I tried an actual connectivity test using http2.
I think this is the best way to test the internet connection as it doesn't send much data and also doesn't rely on DNS resolving alone and it is quite fast.
Note that this was added in: v8.4.0
const http2 = require('http2');
function isConnected() {
return new Promise((resolve) => {
const client = http2.connect('https://www.google.com');
client.on('connect', () => {
resolve(true);
client.destroy();
});
client.on('error', () => {
resolve(false);
client.destroy();
});
});
};
isConnected().then(console.log);
Edit: I made this into a package if anyone is interested.
As of 2019 you can use DNS promises lookup.
NOTE This API is experimental.
const dns = require('dns').promises;
exports.checkInternet = function checkInternet() {
return dns.lookup('google.com')
.then(() => true)
.catch(() => false);
};
I found a great and simple npm tool to detect internet connection. It's looks like more reliable.
First you need to install
npm i check-internet-connected
Then you can call it like follows
const checkInternetConnected = require('check-internet-connected');
const config = {
timeout: 5000, //timeout connecting to each server(A and AAAA), each try (default 5000)
retries: 5,//number of retries to do before failing (default 5)
domain: 'google.com'//the domain to check DNS record of
}
checkInternetConnected(config)
.then(() => {
console.log("Internet available");
}).catch((error) => {
console.log("No internet", error);
});
It is very helpful to check internet connection for our browser is available or not.
var internetAvailable = require("internet-available");
internetAvailable().then(function(){
console.log("Internet available",internetAvailable);
}).catch(function(){
console.log("No internet");
});
for more[internet-available][1]: https://www.npmjs.com/package/internet-available
It's a very simple function that does not import any stuff, but makes use of JS inbuilt function, and can only be executed when called, it does not monitor loss/internet connection; unlike some answers that make use of cache, this result is accurate as it does not return cached result.
const connected = fetch("https://google.com", {
method: "FET",
cache: "no-cache",
headers: { "Content-Type": "application/json" },
referrerPolicy: "no-referrer",
}).then(() => true)
.catch(() => false);
call it using await(make sure your'e inside an async function or you'll get a promise)
console.log(await connected);