I cannot run kannel(an open source sms gateway) run properly. Everytime I try I get this error. ERROR: System error 98: Address already in use
Here is my kannel.conf
group = core
admin-port = 13000
admin-password = 123
status-password = 123
smsbox-port = 13003
wapbox-port = 13004
log-file = "/home/eclipse/kannel_core.log"
log-level = 1
wdp-interface-name = "*"
#group = smsc
#smsc = fake
#port = 1000
group = smsbox
bearerbox-host = 127.0.0.1
sendsms-port = 13013
log-file = "/home/eclipse/kannel_smsbox.log"
log-level = 1
group = wapbox
bearerbox-host = 127.0.0.1
log-file = "/home/eclipse/kannel_wapbox.log"
group = sendsms-user
username = 123
password = 123
user-allow-ip = "127.0.0.1"
group = sms-service
url="http://localhost/sms/send.php?sender=%s&&text=%t"
accept-x-kannel-headers = true
and the log file for the core:
2012-02-23 21:26:36 [9194] [0] INFO: Added logfile `/home/eclipse/kannel_core.log' with level `1'.
2012-02-23 21:26:36 [9194] [0] INFO: HTTP: Opening server at port 13000.
2012-02-23 21:26:36 [9194] [0] ERROR: bind failed
2012-02-23 21:26:36 [9194] [0] ERROR: System error 98: Address already in use
2012-02-23 21:26:36 [9194] [0] INFO: Adding interface *
2012-02-23 21:26:36 [9194] [0] ERROR: Couldn't bind a UDP socket to port 9200
2012-02-23 21:26:36 [9194] [0] ERROR: System error 98: Address already in use
2012-02-23 21:26:36 [9194] [0] ERROR: Couldn't bind a UDP socket to port 9201
2012-02-23 21:26:36 [9194] [0] ERROR: System error 98: Address already in use
2012-02-23 21:26:36 [9194] [0] INFO: ----------------------------------------
2012-02-23 21:26:36 [9194] [0] INFO: Kannel bearerbox II version 1.4.3 starting
2012-02-23 21:26:36 [9194] [7] ERROR: bind failed
2012-02-23 21:26:36 [9194] [7] ERROR: System error 98: Address already in use
2012-02-23 21:26:36 [9194] [7] PANIC: Could not open wapbox port 13004
2012-02-23 21:26:36 [9194] [0] INFO: MAIN: Start-up done, entering mainloop
2012-02-23 21:26:36 [9194] [7] PANIC: /usr/sbin/bearerbox(gw_panic+0xd2) [0x80d1b62]
2012-02-23 21:26:36 [9194] [7] PANIC: /usr/sbin/bearerbox() [0x8055bc0]
2012-02-23 21:26:36 [9194] [7] PANIC: /usr/sbin/bearerbox() [0x80c8bbb]
2012-02-23 21:26:36 [9194] [7] PANIC: /lib/i386-linux-gnu/libpthread.so.0(+0x6d31) [0x7a5d31]
2012-02-23 21:26:36 [9194] [7] PANIC: /lib/i386-linux-gnu/libc.so.6(clone+0x5e) [0xbba0ce]
I have tried to do a lot of changes but none successful. Kindly let me know which point I am missing. Thanks.
Something else is running at ports 13000, 9200, 9201 and 13004 - most likely another instance of kannel.
do sudo netstat -lanp to see what process it is.
Try to stop kannel service /etc/init.d/kannel stop and start it /etc/init.d/kannel start after few seconds.
I found out that sometime if you stop kannel service, it takes some few seconds to shutdown completely.
I had this error occur without the kannel piece but the etc/hosts file had a bad host/ip configured and the DNS at the site was pointing to the wrong server so Oracle listener could not be reached.
Related
After adding some new secrets to Terraform using the 1Password provider, we saw an error without much helpful output.
$ terraform plan
...
Error: rpc error: code = Unavailable desc = transport is closing
Error: rpc error: code = Canceled desc = context canceled
...
Terraform provider:
terraform {
required_providers {
onepassword = {
source = "anasinnyk/onepassword"
version = "~> 1.2.1"
}
}
required_version = "~> 0.13"
}
Terraform yml:
data "onepassword_item_password" "search_cloud_id" {
name = "Azure Elastic Cloud ID"
vault = data.onepassword_vault.vault_name.id
}
data "onepassword_item_password" "search_api_key" {
name = "Azure Elastic Cloud API key"
vault = data.onepassword_vault.vault_name.id
}
resource "kubernetes_secret" "search" {
metadata {
name = "search"
namespace = kubernetes_namespace.production.id
}
data = {
"ELASTICSEARCH_CLOUD_ID" = data.onepassword_item_password.api_search_cloud_id.password
"ELASTICSEARCH_API_KEY" = data.onepassword_item_password.api_search_api_key.password
}
type = "Opaque"
}
We managed to get some useful output by removing one data reference at a time, which lead to the errors printing:
panic: runtime error: invalid memory address or nil pointer dereference
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x147d1bd]
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1:
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: goroutine 194 [running]:
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/anasinnyk/terraform-provider-1password/onepassword.resourceItemPasswordRead(0x19418a0, 0xc0004ac540, 0xc000096f80, 0x173d040, 0xc0007ac740, 0xc0003bce40, 0xc000119910, 0x100c9b8)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/anasinnyk/terraform-provider-1password/onepassword/resource_item_password.go:75 +0x18d
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0004613f0, 0x1941820, 0xc000384300, 0xc000096f80, 0x173d040, 0xc0007ac740, 0x0, 0x0, 0x0)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2#v2.0.0/helper/schema/resource.go:288 +0x1ec
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0xc0004613f0, 0x1941820, 0xc000384300, 0xc000304b80, 0x173d040, 0xc0007ac740, 0xc0007ac740, 0xc000304b80, 0x0, 0x0)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2#v2.0.0/helper/schema/resource.go:489 +0xff
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2/internal/helper/plugin.(*GRPCProviderServer).ReadDataSource(0xc00026e6a0, 0x1941820, 0xc000384300, 0xc0003842c0, 0xc00026e6a0, 0xc00026e6b0, 0x185a058)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2#v2.0.0/internal/helper/plugin/grpc_provider.go:1102 +0x4c5
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5._Provider_ReadDataSource_Handler.func1(0x1941820, 0xc000384300, 0x17dcd60, 0xc0003842c0, 0xc000384300, 0x1773c80, 0xc0004ac401, 0xc000304640)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2#v2.0.0/internal/tfplugin5/tfplugin5.pb.go:3348 +0x86
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2/plugin.Serve.func3.1(0x19418e0, 0xc0003d4480, 0x17dcd60, 0xc0003842c0, 0xc000304620, 0xc000304640, 0xc0007c8ba0, 0x11b81c8, 0x17c7a20, 0xc0003d4480)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2#v2.0.0/plugin/serve.go:76 +0x87
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5._Provider_ReadDataSource_Handler(0x17fdb60, 0xc00026e6a0, 0x19418e0, 0xc0003d4480, 0xc0004ac4e0, 0xc00000d080, 0x19418e0, 0xc0003d4480, 0xc000010090, 0x90)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: github.com/hashicorp/terraform-plugin-sdk/v2#v2.0.0/internal/tfplugin5/tfplugin5.pb.go:3350 +0x14b
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: google.golang.org/grpc.(*Server).processUnaryRPC(0xc00027ae00, 0x1949c60, 0xc000103380, 0xc00018e000, 0xc00020acf0, 0x1e49910, 0x0, 0x0, 0x0)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: google.golang.org/grpc#v1.30.0/server.go:1171 +0x50a
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: google.golang.org/grpc.(*Server).handleStream(0xc00027ae00, 0x1949c60, 0xc000103380, 0xc00018e000, 0x0)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: google.golang.org/grpc#v1.30.0/server.go:1494 +0xccd
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0000382e0, 0xc00027ae00, 0x1949c60, 0xc000103380, 0xc00018e000)
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: google.golang.org/grpc#v1.30.0/server.go:834 +0xa1
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: created by google.golang.org/grpc.(*Server).serveStreams.func1
2021-08-27T15:34:29.367+0930 [DEBUG] plugin.terraform-provider-onepassword_v1.2.1: google.golang.org/grpc#v1.30.0/server.go:832 +0x204
2021-08-27T15:34:29.368+0930 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021/08/27 15:34:29 [ERROR] eval: *terraform.evalReadDataRefresh, err: rpc error: code = Unavailable desc = transport is closing
2021/08/27 15:34:29 [ERROR] eval: *terraform.evalReadDataRefresh, err: rpc error: code = Unavailable desc = transport is closing
2021/08/27 15:34:29 [ERROR] eval: *terraform.evalReadDataRefresh, err: rpc error: code = Unavailable desc = transport is closing
2021/08/27 15:34:29 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2021-08-27T15:34:29.369+0930 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/anasinnyk/onepassword/1.2.1/darwin_amd64/terraform-provider-onepassword_v1.2.1 pid=17549 error="exit status 2"
2021/08/27 15:34:29 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2021/08/27 15:34:29 [TRACE] [walkRefresh] Exiting eval tree: data.onepassword_item_password.search_api_key
2021/08/27 15:34:29 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2021/08/27 15:34:29 [TRACE] vertex "data.onepassword_item_password.search_api_key": visit complete
2021/08/27 15:34:29 [TRACE] vertex "data.onepassword_item_password.search_api_key": dynamic subgraph encountered errors
2021/08/27 15:34:29 [TRACE] vertex "data.onepassword_item_password.search_api_key": visit complete
2021/08/27 15:34:29 [TRACE] vertex "data.onepassword_item_password.search_api_key (expand)": dynamic subgraph encountered errors
2021/08/27 15:34:29 [TRACE] vertex "data.onepassword_item_password.search_api_key (expand)": visit complete
2021/08/27 15:34:29 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/kubernetes\"] (close)" errored, so skipping
2021/08/27 15:34:29 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/anasinnyk/onepassword\"] (close)" errored, so skipping
2021/08/27 15:34:29 [TRACE] dag/walk: upstream of "root" errored, so skipping
2021-08-27T15:34:29.501+0930 [DEBUG] plugin: plugin exited
2021-08-27T15:34:29.502+0930 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-08-27T15:34:29.507+0930 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/kubernetes/1.13.3/darwin_amd64/terraform-provider-kubernetes_v1.13.3_x4 pid=17673
2021-08-27T15:34:29.507+0930 [DEBUG] plugin: plugin exited
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.
When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.
SECURITY WARNING: the "crash.log" file that was created may contain
sensitive information that must be redacted before it is safe to share
on the issue tracker.
[1]: https://github.com/hashicorp/terraform/issues
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
This led us to find that one of our team members managed to create two 1Password entries with the same name in the same vault.
After deleting the duplicate entry in 1Password, terraform plan ran without error again.
I am trying to perform a connection to mongodb using mongoose but it throws an error.
Here is my code:
mongoose.connect("mongodb+srv://username:password#cluster0-bocic.mongodb.net/test?retryWrites=true", {useNewUrlParser: true})
.then(()=>console.log("Database connected..."))
.catch(err=>console.log(err));
Here is the error:
[0] { MongoNetworkError: failed to connect to server [cluster0-shard-00-01-bocic
.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to cluster
0-shard-00-01-bocic.mongodb.net:27017 closed]
[0] at Pool.<anonymous> (C:\Users\Med\Desktop\jutsuapp\myapp\node_modules\mo
ngodb-core\lib\topologies\server.js:431:11)
[0] at Pool.emit (events.js:182:13)
[0] at connect (C:\Users\Med\Desktop\jutsuapp\myapp\node_modules\mongodb-cor
e\lib\connection\pool.js:557:14)
[0] at callback (C:\Users\Med\Desktop\jutsuapp\myapp\node_modules\mongodb-co
re\lib\connection\connect.js:109:5)
[0] at runCommand (C:\Users\Med\Desktop\jutsuapp\myapp\node_modules\mongodb-
core\lib\connection\connect.js:129:7)
[0] at Connection.errorHandler (C:\Users\Med\Desktop\jutsuapp\myapp\node_mod
ules\mongodb-core\lib\connection\connect.js:321:5)
[0] at Object.onceWrapper (events.js:273:13)
[0] at Connection.emit (events.js:182:13)
[0] at TLSSocket.<anonymous> (C:\Users\Med\Desktop\jutsuapp\myapp\node_modul
es\mongodb-core\lib\connection\connection.js:350:12)
[0] at Object.onceWrapper (events.js:273:13)
[0] at TLSSocket.emit (events.js:182:13)
[0] at _handle.close (net.js:610:12)
[0] at TCP.done (_tls_wrap.js:386:7)
[0] name: 'MongoNetworkError',
[0] errorLabels: [ 'TransientTransactionError' ],
[0] [Symbol(mongoErrorContextSymbol)]: {} }
I got to solve the same problem: in the last three days my applications (node.js, express, mongoose, graphql) got connected to the mongoDB (in the cloud) whithout any problem, but today I got:
Error: MongoNetworkError: failed to connect to server [cluster0-shard-00-00-cpkkj.mongodb.net:27017] on first connect [MongoNetworkError: connection 5 to cluster0-shard-00-00-cpkkj.mongodb.net:27017 closed]
I verified the IP of my PC and that's it: it has changed and is not more the same IP I added to the Whitelist, so I added the new one and the problem was solved.
If you would like to enable any host connection access, you can set to the whitelist in atlas configuration the following IP address:
0.0.0.0
Then, even when your IP address suddenly changes, you might still connect.
I have two machines, one "server" and one "client". Both are CentOS6 with NodeJS v5.8.0.
The server runs the following program:
const AppPort = 8080;
var app = require('express')();
var logger = require('log4js').getLogger();
var onFinished = require('on-finished');
var uid = require('uid');
var reqCnt = 0;
var reqFin = 0;
app.get('/', function(req, res) {
onFinished(req, function() {
reqFin++;
var ts2 = (new Date()).getTime();
logger.info(`uid=${req.uid}, dt=${ts2-req.ts1}`);
});
req.ts1 = (new Date()).getTime();
req.uid = uid();
reqCnt++;
logger.info(`ReqCnt=${reqCnt}, fins=${reqFin}`);
res.send("This is XML");
});
app.listen(AppPort);
It's only purpose to return "This is XML" string and calculate time of fulfilling the request.
On the "client" machine I run the following program:
const AppPort = 10000;
var onFinished = require('on-finished');
var async = require('async');
var request = require('request');
var logger = require('log4js').getLogger();
var app = require('express')();
var fs = require('fs');
var util = require('util');
url = "http://my-server";
var errCnt = 0;
var okCnt = 0;
var active2 = 0;
setInterval(function() {
var errFrac = Math.floor(errCnt/(okCnt+errCnt)*100);
logger.info(`${okCnt},${errCnt},${active2},${errFrac}`);
}, 1000);
app.get('/test', function(req,res) {
onFinished(res, function() {
active2--;
});
active2++;
var ts1 = (new Date()).getTime();
request(url, {timeout: 1000}, function(err, response, body ) {
var ts2 = (new Date()).getTime();
var dt = ts2-ts1;
if ( err ) {
errCnt += 1;
logger.error(`Error: ${err}, dt=${dt}, errCnt=${errCnt}`);
res.send(`Error: ${err}`);
}
else {
okCnt += 1;
logger.info(`OK: ${url}`);
res.send(`OK: ${body}`);
}
});
});
var http = app.listen(AppPort);
logger.info(`Listening on ${AppPort}, pid=${process.pid}`);
This "client" code listens by itself on port 10000 and makes request to "server" machine to get "This is XML" string. This data is transferred back to "client"'s client.
I load-test my client code with siege:
siege -v -r 100 -c 100 http://my-client:10000/test
Almost immediately I start to get ETIMEOUT errors:
[2016-03-15 18:17:05.155] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1028, errCnt=3
[2016-03-15 18:17:05.156] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1028, errCnt=4
[2016-03-15 18:17:05.156] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1027, errCnt=5
[2016-03-15 18:17:05.157] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1027, errCnt=6
[2016-03-15 18:17:05.157] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1027, errCnt=7
[2016-03-15 18:17:05.157] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1027, errCnt=8
[2016-03-15 18:17:05.158] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1027, errCnt=9
[2016-03-15 18:17:05.160] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1029, errCnt=10
[2016-03-15 18:17:05.160] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1028, errCnt=11
[2016-03-15 18:17:05.161] [ERROR] [default] - Error: Error: ETIMEDOUT, dt=1028, errCnt=12
Also, though much less frequently, getaddrinfo errors appear:
Error: Error: getaddrinfo ENOTFOUND {my-server-domain-here}:8080, dt=2, errCnt=4478
However, all requests to the "server" are processed within less then 3 milliseconds (dt values) on the server itself:
[2016-03-15 18:19:13.847] [INFO] [default] - uid=66ohx90, dt=1
[2016-03-15 18:19:13.862] [INFO] [default] - ReqCnt=5632, fins=5631
[2016-03-15 18:19:13.862] [INFO] [default] - uid=j8mpxdm, dt=0
[2016-03-15 18:19:13.865] [INFO] [default] - ReqCnt=5633, fins=5632
[2016-03-15 18:19:13.866] [INFO] [default] - uid=xcetqyj, dt=1
[2016-03-15 18:19:13.877] [INFO] [default] - ReqCnt=5634, fins=5633
[2016-03-15 18:19:13.877] [INFO] [default] - uid=i5qnbit, dt=0
[2016-03-15 18:19:13.895] [INFO] [default] - ReqCnt=5635, fins=5634
[2016-03-15 18:19:13.895] [INFO] [default] - uid=hpdmxpg, dt=1
[2016-03-15 18:19:13.930] [INFO] [default] - ReqCnt=5636, fins=5635
[2016-03-15 18:19:13.930] [INFO] [default] - uid=8g3t8md, dt=0
[2016-03-15 18:19:13.934] [INFO] [default] - ReqCnt=5637, fins=5636
[2016-03-15 18:19:13.934] [INFO] [default] - uid=8rwkad6, dt=0
[2016-03-15 18:19:14.163] [INFO] [default] - ReqCnt=5638, fins=5637
[2016-03-15 18:19:14.165] [INFO] [default] - uid=1sh2frd, dt=2
[2016-03-15 18:19:14.169] [INFO] [default] - ReqCnt=5639, fins=5638
[2016-03-15 18:19:14.170] [INFO] [default] - uid=comn76k, dt=1
[2016-03-15 18:19:14.174] [INFO] [default] - ReqCnt=5640, fins=5639
[2016-03-15 18:19:14.174] [INFO] [default] - uid=gj9e0fm, dt=0
[2016-03-15 18:19:14.693] [INFO] [default] - ReqCnt=5641, fins=5640
[2016-03-15 18:19:14.693] [INFO] [default] - uid=x0yw66n, dt=0
[2016-03-15 18:19:14.713] [INFO] [default] - ReqCnt=5642, fins=5641
[2016-03-15 18:19:14.714] [INFO] [default] - uid=e2cumjv, dt=1
[2016-03-15 18:19:14.734] [INFO] [default] - ReqCnt=5643, fins=5642
[2016-03-15 18:19:14.735] [INFO] [default] - uid=34e0ohl, dt=1
[2016-03-15 18:19:14.747] [INFO] [default] - ReqCnt=5644, fins=5643
[2016-03-15 18:19:14.749] [INFO] [default] - uid=34aau79, dt=2
So, the problem is not that the "server" processes the requests too long, but there is a problem with the client.
In NodeJS 5.8 globalAgent looks like the following:
console.log(require('http.globalAgent'))
Agent {
domain: null,
_events: { free: [Function] },
_eventsCount: 1,
_maxListeners: undefined,
defaultPort: 80,
protocol: 'http:',
options: { path: null },
requests: {},
sockets: {},
freeSockets: {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256 }
ulimits on my system look like:
root#njs testreq]# ulimit -all
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 128211
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 200000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 128211
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
What can be a reason for the timeouts?
When running a few load tests recently I ran into a similar error, however instead of ETIMEDOUT errors I saw multiple EADDRINUSE errors. At the time I was running the tests with the following HTTP agent configuration changes.
{
maxSockets: 256,
keepAlive: false
}
It turns out this configuration wastes a lot of cycles intentionally closing each connection after a single request, and the EADDRINUSE errors were due to running out of ephemeral ports.
For my tests I was still using version 0.12.9 so I'm not sure if this still holds in versions >= 4.x, but the core HTTP library will automatically maintain connections to servers based on the host/port/protocol when possible. This can greatly reduce the load on the client and server, but can also cause requests to build up if the client pool is too small to handle the rate of outbound requests. The best configuration then is one that will keep alive connections whenever possible, but still has a large enough connection pool to quickly handle each outbound request.
Additionally, Node.js is built on top of libuv which implements the event loop interface. One way or another, almost any asynchronous operation implemented by a core Node.js library will interact with libuv. In order to implement this type of interface libuv will use one of several different policies, one of which is a thread pool. The default size of this thread pool is 4, with a max of 128.
The important point here is that any calls to getaddrinfo and getnameinfo will use the thread pool, which means regardless of the size of your HTTP connection pool, DNS queries and some operations lower in the network stack will be serialized based on the thread pool size. It's possible to change the thread pool size by setting the environment variable UV_THREADPOOL_SIZE to a value in the range 4 - 128.
For my tests the ideal settings were UV_THREADPOOL_SIZE=50 with the following HTTP agent configuration.
{
maxSockets: 256,
keepAlive: true
}
This answer has more info on when and how libuv is used.
I faced with emegrency problem on my server 1 hour ago.
I user php5-fpm cgi + nginx. The following php code leads to cgi worker hung.
$fp = fsockopen("localhost", 6699, $errno, $errstr, 30);
$result = '';
if (!$fp) {
$this->SysLog2(ALERT, "Unable to send command to RmServer. $errstr ($errno)");
} else {
fwrite($fp, $command);
while (!feof($fp)) {
$result .= fgets($fp, 128);
}
fclose($fp);
}
#0 0x00007f58ac392313 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1 0x00000000006682f4 in ?? ()
#2 0x0000000000470d8a in ?? ()
#3 0x000000000065b708 in ?? ()
#4 0x000000000065c3d9 in _php_stream_get_line ()
#5 0x00000000005daefd in zif_fgets ()
#6 0x000000000070f52d in ?? ()
#7 0x00000000006c009b in execute ()
#8 0x000000000069b600 in zend_execute_scripts ()
#9 0x0000000000647c73 in php_execute_script ()
#10 0x000000000042ba55 in ?? ()
#11 0x00007f58ac2cb76d in __libc_start_main (main=0x42ab00, argc=3, ubp_av=0x7fffa0becca8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>,
stack_end=0x7fffa0becc98) at libc-start.c:226
#12 0x000000000042c2b5 in _start ()
[09-Feb-2014 21:30:28.789264] DEBUG: pid 10529, fpm_scoreboard_init_main(), line 40: got clock tick '100'
[09-Feb-2014 21:30:28.789503] DEBUG: pid 10529, fpm_event_init_main(), line 331: event module is epoll and 3 fds have been reserved
[09-Feb-2014 21:30:28.789576] NOTICE: pid 10529, fpm_init(), line 80: fpm is running, pid 10529
[09-Feb-2014 21:30:28.789603] DEBUG: pid 10529, fpm_pctl_heartbeat(), line 460: heartbeat have been set up with a timeout of 3333ms
[09-Feb-2014 21:30:28.789617] DEBUG: pid 10529, fpm_event_loop(), line 360: 1280 bytes have been reserved in SHM
[09-Feb-2014 21:30:28.789624] NOTICE: pid 10529, fpm_event_loop(), line 361: ready to handle connections
[09-Feb-2014 21:30:29.790664] DEBUG: pid 10529, fpm_pctl_perform_idle_server_maintenance(), line 362: [pool www] currently 0 active children, 0 spare children
[09-Feb-2014 21:30:31.947235] DEBUG: pid 10529, fpm_children_make(), line 421: [pool www] child 10530 started
[09-Feb-2014 21:30:31.947278] DEBUG: pid 10529, fpm_pctl_on_socket_accept(), line 536: [pool www] got accept without idle child available .... I forked
[09-Feb-2014 21:30:31.947286] DEBUG: pid 10529, fpm_event_loop(), line 409: event module triggered 1 events
[09-Feb-2014 21:30:32.794231] DEBUG: pid 10529, fpm_pctl_perform_idle_server_maintenance(), line 362: [pool www] currently 1 active children, 0 spare children
[09-Feb-2014 21:30:33.795326] DEBUG: pid 10529, fpm_pctl_perform_idle_server_maintenance(), line 362: [pool www] currently 1 active children, 0 spare children
[09-Feb-2014 21:30:42.126558] WARNING: pid 10529, fpm_request_check_timed_out(), line 271: [pool www] child 10530, script '/proj/manager/ssl-server/RPRCP/queues.php' (request: "GET /RPRCP/queues.php") executing too slow (10.179030 sec), logging
[09-Feb-2014 21:30:42.126626] DEBUG: pid 10529, fpm_got_signal(), line 72: received SIGCHLD
[09-Feb-2014 21:30:42.126640] NOTICE: pid 10529, fpm_children_bury(), line 227: child 10530 stopped for tracing
[09-Feb-2014 21:30:42.126647] NOTICE: pid 10529, fpm_php_trace(), line 142: about to trace 10530
[09-Feb-2014 21:30:42.126773] NOTICE: pid 10529, fpm_php_trace(), line 170: finished trace of 10530
[09-Feb-2014 21:30:42.126787] DEBUG: pid 10529, fpm_event_loop(), line 409: event module triggered 1 events
[09-Feb-2014 21:30:42.802499] DEBUG: pid 10529, fpm_pctl_perform_idle_server_maintenance(), line 362: [pool www] currently 1 active children, 0 spare children
Most likely workers hungs at '_php_stream_get_line'. But why it alive after SIGCHLD?
Also I can't find 'slow' log.. I have the followith statemens in config:
prefix = /var/log
slowlog = /php5-fpm.log.slow
request_slowlog_timeout = 10s
EDIT:
My remote process deadlocked by a mutex. But why php behaviour is so dangerous?
I am trying to send and receive SMS via kannel. I have set everything, no errors but still it cannot send or receive any sms.
I am using Huawei E160 modem and Ubuntu 11 as my OS.
Here is the log file:
2012-02-25 14:25:34 [9913] [0] INFO: HTTP: Opening server at port 13000.
2012-02-25 14:25:34 [9913] [0] INFO: BOXC: 'smsbox-max-pending' not set, using default (100).
2012-02-25 14:25:34 [9913] [0] INFO: Set SMS resend frequency to 60 seconds.
2012-02-25 14:25:34 [9913] [0] INFO: SMS resend retry set to unlimited.
2012-02-25 14:25:34 [9913] [0] INFO: DLR rerouting for smsc id <(null)> disabled.
2012-02-25 14:25:34 [9913] [0] INFO: AT2[/dev/ttyUSB0]: configuration shows modemtype <huawei>
2012-02-25 14:25:34 [9913] [0] INFO: AT2[/dev/ttyUSB0]: read modem definition for <huawei-e160>
2012-02-25 14:25:34 [9913] [6] INFO: AT2[/dev/ttyUSB0]: opening device
2012-02-25 14:25:34 [9913] [0] INFO: Adding interface *
2012-02-25 14:25:34 [9913] [0] INFO: ----------------------------------------
2012-02-25 14:25:34 [9913] [0] INFO: Kannel bearerbox II version 1.4.3 starting
2012-02-25 14:25:34 [9913] [0] INFO: MAIN: Start-up done, entering mainloop
2012-02-25 14:25:34 [9913] [6] INFO: AT2[/dev/ttyUSB0]: Logging in
2012-02-25 14:25:34 [9913] [6] INFO: AT2[/dev/ttyUSB0]: init device
2012-02-25 14:25:34 [9913] [6] INFO: AT2[/dev/ttyUSB0]: speed set to 115200
2012-02-25 14:25:35 [9913] [6] INFO: AT2[/dev/ttyUSB0]: AT SMSC successfully opened.
2012-02-25 14:26:28 [9913] [13] INFO: Client connected from <127.0.0.1>
My kannel.conf file
group = core
admin-port = 13000
admin-password = 123
status-password = 123
smsbox-port = 13003
wapbox-port = 13004
log-file = "/home/eclipse/kannel_core.log"
log-level = 1
wdp-interface-name = "*"
#group = smsc
#smsc = fake
#port = 1000
group = smsbox
bearerbox-host = 127.0.0.1
sendsms-port = 13013
log-file = "/home/eclipse/kannel_smsbox.log"
log-level = 1
group = smsc
smsc = at
modemtype = huawei
device = /dev/ttyUSB0
speed=115200
log-level = 0
pin = 1442
group = modems
id = huawei
name = "huawei-e160"
detect-string = "huawei"
init-string = "ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0"
speed = 115200
group = wapbox
bearerbox-host = 127.0.0.1
log-file = "/home/eclipse/kannel_wapbox.log"
group = sendsms-user
username = 123
password = 123
user-allow-ip = "127.0.0.1"
group = sms-service
keyword = default
get-url = "http://localhost/kannel/receivesms.php?sender=%p&text=%b"
accept-x-kannel-headers = true
max-messages = 3
concatenation = true
catch-all = true
My receivesms.php file is coded as
<?php
define("DBHOST","localhost",true);
define("DBUSERNAME","root",true);
define("DBPASSWORD","xxxxxx",true);
define("DBNAME","kannel_sms",true);
function insertSms($sender,$text)
{
$con = 'mysql:dbname='.DBNAME.';host='.DBHOST;
try {
$cmd = new PDO($con,DBUSERNAME,DBPASSWORD);
$stmt = $cmd->prepare("INSERT INTO kannel_tuto (number,message) VALUES (:sender,:message)");
$stmt->bindParam(':sender',$sender);
$stmt->bindParam(':message',$text);
$stmt->execute();
$cmd = null;
if($stmt->rowCount()>0)
{
echo "Hello ".$text.". Thank you for your registration.";
}
else
{
echo "Sorry an error has occured";
}
}
catch (PDOException $e) {
echo 'Connection failed: ' . $e->getMessage();
}
}
insertSms($_GET['sender'],$_GET['text']);
?>
As I indicated above, my script and configurations are correct. I just figured out that the E160 modem is not good for sms and kannel. There is loss of signals, I don't know why though. I used E173 and E220 huawei modems and they work well.