I have this script for measuring the time it takes for a query to perform:
for (var i = 0; i < 10000; i++) {
(function(start) {
Models.User.findOneAsync({
userId: 'ABCD'
}, 'age name location')
.then(function(user) {
logger.debug(Date.now() - start);
})
})(Date.now());
}
As a result, I'm getting an incremented list of results:
2015-05-19T11:09:16.204Z - debug: 4369
2015-05-19T11:09:16.205Z - debug: 4367
2015-05-19T11:09:16.205Z - debug: 4367
2015-05-19T11:09:16.206Z - debug: 4368
2015-05-19T11:09:16.206Z - debug: 4367
2015-05-19T11:09:16.206Z - debug: 4368
2015-05-19T11:09:16.206Z - debug: 4367
2015-05-19T11:09:16.206Z - debug: 4369
2015-05-19T11:09:16.206Z - debug: 4368
2015-05-19T11:09:16.206Z - debug: 4368
2015-05-19T11:09:16.206Z - debug: 4368
2015-05-19T11:09:16.206Z - debug: 4367
2015-05-19T11:09:16.212Z - debug: 4373
2015-05-19T11:09:16.212Z - debug: 4373
2015-05-19T11:09:16.248Z - debug: 4408
2015-05-19T11:09:16.376Z - debug: 4536
2015-05-19T11:09:16.459Z - debug: 4619
2015-05-19T11:09:16.475Z - debug: 4635
2015-05-19T11:09:16.493Z - debug: 4654
2015-05-19T11:09:16.552Z - debug: 4713
2015-05-19T11:09:16.636Z - debug: 4796
2015-05-19T11:09:16.794Z - debug: 4954
2015-05-19T11:09:16.830Z - debug: 4990
2015-05-19T11:09:16.841Z - debug: 5001
2015-05-19T11:09:16.845Z - debug: 5005
2015-05-19T11:09:17.133Z - debug: 5293
2015-05-19T11:09:17.176Z - debug: 5336
2015-05-19T11:09:17.182Z - debug: 5341
2015-05-19T11:09:17.230Z - debug: 5390
2015-05-19T11:09:17.421Z - debug: 5580
2015-05-19T11:09:17.437Z - debug: 5596
2015-05-19T11:09:17.441Z - debug: 5600
2015-05-19T11:09:17.513Z - debug: 5672
2015-05-19T11:09:17.569Z - debug: 5728
2015-05-19T11:09:17.658Z - debug: 5817
2015-05-19T11:09:17.697Z - debug: 5855
2015-05-19T11:09:17.708Z - debug: 5867
2015-05-19T11:09:17.712Z - debug: 5870
2015-05-19T11:09:17.732Z - debug: 5891
2015-05-19T11:09:17.805Z - debug: 5963
2015-05-19T11:09:17.852Z - debug: 6010
2015-05-19T11:09:17.890Z - debug: 6048
2015-05-19T11:09:17.985Z - debug: 6143
2015-05-19T11:09:17.989Z - debug: 6147
2015-05-19T11:09:18.013Z - debug: 6171
2015-05-19T11:09:18.016Z - debug: 6175
2015-05-19T11:09:18.031Z - debug: 6189
2015-05-19T11:09:18.170Z - debug: 6327
2015-05-19T11:09:18.196Z - debug: 6353
2015-05-19T11:09:18.205Z - debug: 6362
2015-05-19T11:09:18.209Z - debug: 6367
2015-05-19T11:09:18.224Z - debug: 6382
2015-05-19T11:09:18.317Z - debug: 6474
2015-05-19T11:09:18.360Z - debug: 6516
2015-05-19T11:09:18.369Z - debug: 6526
2015-05-19T11:09:18.433Z - debug: 6590
2015-05-19T11:09:18.460Z - debug: 6616
2015-05-19T11:09:18.513Z - debug: 6668
2015-05-19T11:09:18.541Z - debug: 6697
2015-05-19T11:09:18.553Z - debug: 6711
2015-05-19T11:09:18.586Z - debug: 6741
2015-05-19T11:09:18.672Z - debug: 6827
2015-05-19T11:09:18.688Z - debug: 6844
2015-05-19T11:09:18.693Z - debug: 6849
2015-05-19T11:09:18.729Z - debug: 6884
2015-05-19T11:09:18.817Z - debug: 6972
2015-05-19T11:09:18.823Z - debug: 6980
2015-05-19T11:09:18.828Z - debug: 6983
2015-05-19T11:09:18.882Z - debug: 7036
2015-05-19T11:09:18.919Z - debug: 7075
2015-05-19T11:09:19.016Z - debug: 7170
2015-05-19T11:09:19.020Z - debug: 7174
2015-05-19T11:09:19.043Z - debug: 7197
2015-05-19T11:09:19.066Z - debug: 7222
2015-05-19T11:09:19.177Z - debug: 7331
2015-05-19T11:09:19.182Z - debug: 7335
2015-05-19T11:09:19.189Z - debug: 7343
2015-05-19T11:09:19.189Z - debug: 7344
2015-05-19T11:09:19.191Z - debug: 7344
2015-05-19T11:09:19.280Z - debug: 7433
2015-05-19T11:09:19.340Z - debug: 7494
2015-05-19T11:09:19.344Z - debug: 7497
2015-05-19T11:09:19.358Z - debug: 7512
2015-05-19T11:09:19.362Z - debug: 7518
2015-05-19T11:09:19.455Z - debug: 7608
2015-05-19T11:09:19.499Z - debug: 7651
2015-05-19T11:09:19.504Z - debug: 7656
2015-05-19T11:09:19.515Z - debug: 7669
2015-05-19T11:09:19.569Z - debug: 7722
2015-05-19T11:09:19.574Z - debug: 7726
2015-05-19T11:09:19.574Z - debug: 7726
2015-05-19T11:09:19.667Z - debug: 7818
2015-05-19T11:09:19.672Z - debug: 7823
2015-05-19T11:09:19.678Z - debug: 7830
2015-05-19T11:09:19.689Z - debug: 7844
2015-05-19T11:09:19.716Z - debug: 7868
2015-05-19T11:09:19.835Z - debug: 7986
2015-05-19T11:09:19.839Z - debug: 7989
2015-05-19T11:09:19.845Z - debug: 7997
2015-05-19T11:09:19.978Z - debug: 8128
2015-05-19T11:09:19.989Z - debug: 8136
2015-05-19T11:09:19.995Z - debug: 8146
2015-05-19T11:09:19.999Z - debug: 8153
2015-05-19T11:09:20.012Z - debug: 8166
2015-05-19T11:09:20.023Z - debug: 8174
2015-05-19T11:09:20.026Z - debug: 8177
2015-05-19T11:09:20.116Z - debug: 8262
2015-05-19T11:09:20.127Z - debug: 8272
2015-05-19T11:09:20.136Z - debug: 8287
2015-05-19T11:09:20.154Z - debug: 8307
2015-05-19T11:09:20.179Z - debug: 8324
2015-05-19T11:09:20.262Z - debug: 8407
2015-05-19T11:09:20.275Z - debug: 8425
2015-05-19T11:09:20.279Z - debug: 8423
2015-05-19T11:09:20.306Z - debug: 8456
2015-05-19T11:09:20.309Z - debug: 8463
2015-05-19T11:09:20.396Z - debug: 8540
2015-05-19T11:09:20.422Z - debug: 8565
2015-05-19T11:09:20.424Z - debug: 8574
2015-05-19T11:09:20.441Z - debug: 8594
2015-05-19T11:09:20.452Z - debug: 8601
2015-05-19T11:09:20.455Z - debug: 8605
2015-05-19T11:09:20.461Z - debug: 8604
2015-05-19T11:09:20.549Z - debug: 8691
2015-05-19T11:09:20.555Z - debug: 8697
2015-05-19T11:09:20.565Z - debug: 8711
2015-05-19T11:09:20.598Z - debug: 8752
2015-05-19T11:09:20.602Z - debug: 8750
2015-05-19T11:09:20.659Z - debug: 8801
2015-05-19T11:09:20.689Z - debug: 8830
2015-05-19T11:09:20.701Z - debug: 8842
2015-05-19T11:09:20.707Z - debug: 8853
2015-05-19T11:09:20.712Z - debug: 8864
2015-05-19T11:09:20.752Z - debug: 8898
2015-05-19T11:09:20.798Z - debug: 8938
2015-05-19T11:09:20.829Z - debug: 8969
2015-05-19T11:09:20.844Z - debug: 8989
2015-05-19T11:09:20.850Z - debug: 8990
2015-05-19T11:09:20.869Z - debug: 9021
2015-05-19T11:09:20.880Z - debug: 9033
2015-05-19T11:09:20.893Z - debug: 9038
2015-05-19T11:09:20.939Z - debug: 9078
2015-05-19T11:09:20.965Z - debug: 9104
2015-05-19T11:09:20.979Z - debug: 9124
2015-05-19T11:09:20.984Z - debug: 9122
2015-05-19T11:09:21.051Z - debug: 9189
2015-05-19T11:09:21.057Z - debug: 9202
2015-05-19T11:09:21.057Z - debug: 9210
2015-05-19T11:09:21.096Z - debug: 9234
2015-05-19T11:09:21.112Z - debug: 9249
2015-05-19T11:09:21.121Z - debug: 9265
2015-05-19T11:09:21.130Z - debug: 9267
2015-05-19T11:09:21.133Z - debug: 9284
2015-05-19T11:09:21.195Z - debug: 9339
2015-05-19T11:09:21.200Z - debug: 9345
2015-05-19T11:09:21.239Z - debug: 9375
2015-05-19T11:09:21.247Z - debug: 9383
2015-05-19T11:09:21.270Z - debug: 9404
2015-05-19T11:09:21.283Z - debug: 9434
2015-05-19T11:09:21.334Z - debug: 9468
2015-05-19T11:09:21.337Z - debug: 9481
2015-05-19T11:09:21.348Z - debug: 9500
2015-05-19T11:09:21.352Z - debug: 9496
2015-05-19T11:09:21.378Z - debug: 9512
2015-05-19T11:09:21.385Z - debug: 9518
2015-05-19T11:09:21.416Z - debug: 9559
2015-05-19T11:09:21.419Z - debug: 9552
2015-05-19T11:09:21.470Z - debug: 9603
2015-05-19T11:09:21.475Z - debug: 9625
2015-05-19T11:09:21.490Z - debug: 9634
2015-05-19T11:09:21.517Z - debug: 9649
2015-05-19T11:09:21.522Z - debug: 9654
2015-05-19T11:09:21.554Z - debug: 9697
2015-05-19T11:09:21.562Z - debug: 9694
2015-05-19T11:09:21.575Z - debug: 9706
2015-05-19T11:09:21.622Z - debug: 9773
2015-05-19T11:09:21.635Z - debug: 9779
2015-05-19T11:09:21.656Z - debug: 9787
2015-05-19T11:09:21.683Z - debug: 9814
2015-05-19T11:09:21.688Z - debug: 9818
2015-05-19T11:09:21.691Z - debug: 9833
2015-05-19T11:09:21.716Z - debug: 9846
2015-05-19T11:09:21.720Z - debug: 9869
2015-05-19T11:09:21.751Z - debug: 9893
2015-05-19T11:09:21.778Z - debug: 9921
2015-05-19T11:09:21.786Z - debug: 9936
2015-05-19T11:09:21.790Z - debug: 9919
2015-05-19T11:09:21.820Z - debug: 9949
2015-05-19T11:09:21.830Z - debug: 9959
2015-05-19T11:09:21.864Z - debug: 10005
2015-05-19T11:09:21.870Z - debug: 9998
2015-05-19T11:09:21.884Z - debug: 10033
2015-05-19T11:09:21.928Z - debug: 10070
2015-05-19T11:09:21.931Z - debug: 10059
2015-05-19T11:09:21.949Z - debug: 10076
2015-05-19T11:09:21.984Z - debug: 10111
2015-05-19T11:09:21.987Z - debug: 10128
2015-05-19T11:09:22.015Z - debug: 10142
2015-05-19T11:09:22.019Z - debug: 10160
2015-05-19T11:09:22.046Z - debug: 10172
2015-05-19T11:09:22.060Z - debug: 10202
2015-05-19T11:09:22.064Z - debug: 10215
2015-05-19T11:09:22.089Z - debug: 10215
2015-05-19T11:09:22.123Z - debug: 10248
2015-05-19T11:09:22.130Z - debug: 10270
2015-05-19T11:09:22.152Z - debug: 10277
2015-05-19T11:09:22.156Z - debug: 10302
2015-05-19T11:09:22.180Z - debug: 10320
2015-05-19T11:09:22.183Z - debug: 10325
2015-05-19T11:09:22.186Z - debug: 10310
2015-05-19T11:09:22.228Z - debug: 10349
2015-05-19T11:09:22.260Z - debug: 10380
Ok, so it's not entirely increment but it's going up forever...
I would expect to get a list of almost the same number (queries should take almost the same time).
Anyone knows why the times keeps going up?
It's because you're not measuring just the query times.
The basic flow of this code is:
The for loop of 10000 iterations runs to completion, setting the start time for each iteration to the time each iteration occurred.
For the first 5 of those iterations (or whatever poolsize you're using with your MongoDB connection), their queries start as soon as their findOneAsync call is made.
As queries complete, their findOneAsync callbacks are put on the event queue and their connections are returned to the pool, allowing subsequent iterations' queries to start.
So all iterations' times include the time to complete the rest of the for loop after it, the time spent waiting for a connection in the pool to become available, and the time their findOneAsync callback spent waiting in the event queue.
If you want to get an accurate picture of how long the queries are taking, use MongoDB's profiling support.
Related
Any tips to debug would help, this is just the basic implementation.
I am using the exact tutorial as here to setup the ngx socket client:
https://www.npmjs.com/package/ngx-socket-io
flask-socketio for the server
https://flask-socketio.readthedocs.io/en/latest/
Server-side shows no error, but the registered handler methods to emit/receive seem blind.
Server Logs:
127.0.0.1 - - [28/Jan/2021 10:58:51] "GET /socket.io/?EIO=3&transport=polling&t=NT8Dtbg HTTP/1.1" 200 418 0.000640
(12285) accepted ('127.0.0.1', 37400)
fa1a1d40bbc349c384a121302ef567c7: Received request to upgrade to websocket
127.0.0.1 - - [28/Jan/2021 10:58:51] "GET /socket.io/?EIO=3&transport=polling&t=NT8Dtcd&sid=fa1a1d40bbc349c384a121302ef567c7 HTTP/1.1" 200 235 0.000408
127.0.0.1 - - [28/Jan/2021 10:58:51] "GET /socket.io/?EIO=3&transport=polling&t=NT8DtdE&sid=fa1a1d40bbc349c384a121302ef567c7 HTTP/1.1" 200 235 0.000268
fa1a1d40bbc349c384a121302ef567c7: Upgrade to websocket successful
5002732d42184ba6b453e7d4f35e864e: Received packet PING data None
5002732d42184ba6b453e7d4f35e864e: Sending packet PONG data None
Client logs:
config: {url: "http://127.0.0.1:5000/", options: {…}}
emptyConfig: {url: "", options: {…}}
eventObservables$: {}
ioSocket: Socket
acks: {}
connected: false
disconnected: true
flags: {}
ids: 0
io: Manager
autoConnect: true
backoff: Backoff {ms: 1000, max: 5000, factor: 2, jitter: 0.5, attempts: 0}
connecting: [Socket]
decoder: Decoder {reconstructor: null, _callbacks: {…}}
encoder: Encoder {}
encoding: false
engine: Socket {secure: false, agent: false, hostname: "127.0.0.1", port: "5000", query: {…}, …}
lastPing: Thu Jan 28 2021 11:00:56
Server code (no print)
#socketio.on('connect')
def test_connect():
print('\n\nClient connected')
Why can't I get the connection working? Thanks
Double check npm/python environment of current versions of socketio and engineio.
https://github.com/miguelgrinberg/python-socketio
d7e95928d73c42fab431e94ce2df40fc: Sending packet OPEN data {'sid': 'd7e95928d73c42fab431e94ce2df40fc', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
Client connected
d7e95928d73c42fab431e94ce2df40fc: Sending packet MESSAGE data 0
zookeeper0:
extends:
file: docker-compose-base.yml
service: zookeeper
container_name: zookeeper0
environment:
- ZOO_MY_ID=1
- ZOO_SERVERS=server.1=zookeeper0:2888:3888 server.2=zookeeper1:2888:3888 server.3=zookeeper2:2888:3888
networks:
behave:
aliases:
- ${CORE_PEER_NETWORKID}
zookeeper1:
extends:
file: docker-compose-base.yml
service: zookeeper
container_name: zookeeper1
environment:
- ZOO_MY_ID=2
- ZOO_SERVERS=server.1=zookeeper0:2888:3888 server.2=zookeeper1:2888:3888 server.3=zookeeper2:2888:3888
networks:
behave:
aliases:
- ${CORE_PEER_NETWORKID}
zookeeper2:
extends:
file: docker-compose-base.yml
service: zookeeper
container_name: zookeeper2
environment:
- ZOO_MY_ID=3
- ZOO_SERVERS=server.1=zookeeper0:2888:3888 server.2=zookeeper1:2888:3888 server.3=zookeeper2:2888:3888
networks:
behave:
aliases:
- ${CORE_PEER_NETWORKID}
kafka0:
extends:
file: docker-compose-base.yml
service: kafka
container_name: kafka0
environment:
- KAFKA_BROKER_ID=0
- KAFKA_ZOOKEEPER_CONNECT=zookeeper0:2181,zookeeper1:2181,zookeeper2:2181
- KAFKA_MESSAGE_MAX_BYTES=${KAFKA_MESSAGE_MAX_BYTES}
- KAFKA_REPLICA_FETCH_MAX_BYTES=${KAFKA_REPLICA_FETCH_MAX_BYTES}
- KAFKA_REPLICA_FETCH_RESPONSE_MAX_BYTES=${KAFKA_REPLICA_FETCH_RESPONSE_MAX_BYTES}
depends_on:
- zookeeper0
- zookeeper1
- zookeeper2
networks:
behave:
aliases:
- ${CORE_PEER_NETWORKID}
kafka1:
extends:
file: docker-compose-base.yml
service: kafka
container_name: kafka1
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper0:2181,zookeeper1:2181,zookeeper2:2181
- KAFKA_MESSAGE_MAX_BYTES=${KAFKA_MESSAGE_MAX_BYTES}
- KAFKA_REPLICA_FETCH_MAX_BYTES=${KAFKA_REPLICA_FETCH_MAX_BYTES}
- KAFKA_REPLICA_FETCH_RESPONSE_MAX_BYTES=${KAFKA_REPLICA_FETCH_RESPONSE_MAX_BYTES}
depends_on:
- zookeeper0
- zookeeper1
- zookeeper2
networks:
behave:
aliases:
- ${CORE_PEER_NETWORKID}
kafka2:
extends:
file: docker-compose-base.yml
service: kafka
container_name: kafka2
environment:
- KAFKA_BROKER_ID=2
- KAFKA_ZOOKEEPER_CONNECT=zookeeper0:2181,zookeeper1:2181,zookeeper2:2181
- KAFKA_MESSAGE_MAX_BYTES=${KAFKA_MESSAGE_MAX_BYTES}
- KAFKA_REPLICA_FETCH_MAX_BYTES=${KAFKA_REPLICA_FETCH_MAX_BYTES}
- KAFKA_REPLICA_FETCH_RESPONSE_MAX_BYTES=${KAFKA_REPLICA_FETCH_RESPONSE_MAX_BYTES}
depends_on:
- zookeeper0
- zookeeper1
- zookeeper2
networks:
behave:
aliases:
- ${CORE_PEER_NETWORKID}
kafka3:
extends:
file: docker-compose-base.yml
service: kafka
container_name: kafka3
environment:
- KAFKA_BROKER_ID=3
- KAFKA_ZOOKEEPER_CONNECT=zookeeper0:2181,zookeeper1:2181,zookeeper2:2181
- KAFKA_MESSAGE_MAX_BYTES=${KAFKA_MESSAGE_MAX_BYTES}
- KAFKA_REPLICA_FETCH_MAX_BYTES=${KAFKA_REPLICA_FETCH_MAX_BYTES}
- KAFKA_REPLICA_FETCH_RESPONSE_MAX_BYTES=${KAFKA_REPLICA_FETCH_RESPONSE_MAX_BYTES}
depends_on:
- zookeeper0
- zookeeper1
- zookeeper2
networks:
behave:
aliases:
- ${CORE_PEER_NETWORKID}
logs ::==orderer logs
vagrant#vagrant:~/workspace/kafka-ordering-master$ docker logs orderer0.example.com
2019-02-15 09:08:02.775 UTC [localconfig] completeInitialization -> INFO 001 Kafka.Version unset, setting to 0.10.2.0
2019-02-15 09:08:03.466 UTC [orderer.common.server] prettyPrintStruct -> INFO 002 Orderer config values:
General.LedgerType = "ram"
General.ListenAddress = "0.0.0.0"
General.ListenPort = 7050
General.TLS.Enabled = false
General.TLS.PrivateKey = "/var/hyperledger/tls/server.key"
General.TLS.Certificate = "/var/hyperledger/tls/server.crt"
General.TLS.RootCAs = [/var/hyperledger/tls/ca.crt]
General.TLS.ClientAuthRequired = false
General.TLS.ClientRootCAs = []
General.Cluster.RootCAs = [/etc/hyperledger/fabric/tls/ca.crt]
General.Cluster.ClientCertificate = ""
General.Cluster.ClientPrivateKey = ""
General.Cluster.DialTimeout = 5s
General.Cluster.RPCTimeout = 7s
General.Cluster.ReplicationBufferSize = 20971520
General.Cluster.ReplicationPullTimeout = 5s
General.Cluster.ReplicationRetryTimeout = 5s
General.Keepalive.ServerMinInterval = 1m0s
General.Keepalive.ServerInterval = 2h0m0s
General.Keepalive.ServerTimeout = 20s
General.GenesisMethod = "file"
General.GenesisProfile = "SampleInsecureKafka"
General.SystemChannel = "test-system-channel-name"
General.GenesisFile = "/var/hyperledger/configs/orderer.block"
General.Profile.Enabled = false
General.Profile.Address = "0.0.0.0:6060"
General.LocalMSPDir = "/var/hyperledger/msp"
General.LocalMSPID = "OrdererMSP"
General.BCCSP.ProviderName = "SW"
General.BCCSP.SwOpts.SecLevel = 256
General.BCCSP.SwOpts.HashFamily = "SHA2"
General.BCCSP.SwOpts.Ephemeral = false
General.BCCSP.SwOpts.FileKeystore.KeyStorePath = "/var/hyperledger/msp/keystore"
General.BCCSP.SwOpts.DummyKeystore =
General.BCCSP.SwOpts.InmemKeystore =
General.BCCSP.PluginOpts =
General.Authentication.TimeWindow = 15m0s
FileLedger.Location = "/var/hyperledger/production/orderer"
FileLedger.Prefix = "hyperledger-fabric-ordererledger"
RAMLedger.HistorySize = 1000
Kafka.Retry.ShortInterval = 1s
Kafka.Retry.ShortTotal = 30s
Kafka.Retry.LongInterval = 5m0s
Kafka.Retry.LongTotal = 12h0m0s
Kafka.Retry.NetworkTimeouts.DialTimeout = 10s
Kafka.Retry.NetworkTimeouts.ReadTimeout = 10s
Kafka.Retry.NetworkTimeouts.WriteTimeout = 10s
Kafka.Retry.Metadata.RetryMax = 3
Kafka.Retry.Metadata.RetryBackoff = 250ms
Kafka.Retry.Producer.RetryMax = 3
Kafka.Retry.Producer.RetryBackoff = 100ms
Kafka.Retry.Consumer.RetryBackoff = 2s
Kafka.Verbose = true
Kafka.Version = 0.10.2.0
Kafka.TLS.Enabled = false
Kafka.TLS.PrivateKey = ""
Kafka.TLS.Certificate = ""
Kafka.TLS.RootCAs = []
Kafka.TLS.ClientAuthRequired = false
Kafka.TLS.ClientRootCAs = []
Kafka.SASLPlain.Enabled = false
Kafka.SASLPlain.User = ""
Kafka.SASLPlain.Password = ""
Kafka.Topic.ReplicationFactor = 3
Debug.BroadcastTraceDir = ""
Debug.DeliverTraceDir = ""
Consensus = map[SnapDir:/var/hyperledger/production/orderer/etcdraft/snapshot WALDir:/var/hyperledger/production/orderer/etcdraft/wal]
Operations.ListenAddress = "127.0.0.1:8443"
Operations.TLS.Enabled = false
Operations.TLS.PrivateKey = ""
Operations.TLS.Certificate = ""
Operations.TLS.RootCAs = []
Operations.TLS.ClientAuthRequired = false
Operations.TLS.ClientRootCAs = []
Metrics.Provider = "disabled"
Metrics.Statsd.Network = "udp"
Metrics.Statsd.Address = "127.0.0.1:8125"
Metrics.Statsd.WriteInterval = 30s
Metrics.Statsd.Prefix = ""
2019-02-15 09:08:03.763 UTC [orderer.consensus.kafka] newChain -> INFO 003 [channel: testchainid] Starting chain with last persisted offset -3 and last recorded block 0
2019-02-15 09:08:03.787 UTC [orderer.commmon.multichannel] Initialize -> INFO 004 Starting system channel 'testchainid' with genesis block hash 5f2c3828df168808a899ecce5d7d7306c36cc615464ed0d54b4846155cc3979d and orderer type kafka
2019-02-15 09:08:03.787 UTC [orderer.common.server] Start -> INFO 005 Starting orderer:
Version: 1.4.0
Commit SHA: d700b43
Go version: go1.11.1
OS/Arch: linux/amd64
2019-02-15 09:08:03.787 UTC [orderer.common.server] Start -> INFO 006 Beginning to serve requests
2019-02-15 09:08:03.800 UTC [orderer.consensus.kafka] setupTopicForChannel -> INFO 007 [channel: testchainid] Setting up the topic for this channel...
2019-02-15 09:08:31.401 UTC [orderer.common.broadcast] ProcessMessage -> WARN 008 [channel: mychannel] Rejecting broadcast of message from 172.25.0.12:39536 with SERVICE_UNAVAILABLE: rejected by Consenter: backing Kafka cluster has not completed booting; try again later
2019-02-15 09:08:31.422 UTC [comm.grpc.server] 1 -> INFO 009 streaming call completed {"grpc.start_time": "2019-02-15T09:08:31.041Z", "grpc.service": "orderer.AtomicBroadcast", "grpc.method": "Broadcast", "grpc.peer_address": "172.25.0.12:39536", "grpc.code": "OK", "grpc.call_duration": "380.924298ms"}
2019-02-15 09:08:31.453 UTC [common.deliver] Handle -> WARN 00a Error reading from 172.25.0.12:39534: rpc error: code = Canceled desc = context canceled
2019-02-15 09:08:31.460 UTC [comm.grpc.server] 1 -> INFO 00b streaming call completed {"grpc.start_time": "2019-02-15T09:08:31.036Z", "grpc.service": "orderer.AtomicBroadcast", "grpc.method": "Deliver", "grpc.peer_address": "172.25.0.12:39534", "error": "rpc error: code = Canceled desc = context canceled", "grpc.code": "Canceled", "grpc.call_duration": "423.701471ms"}
It also happens to me. The way I solve it by: run Kitematic --> click My Images (on the right top) --> click create
I installed in Debian 8.5 the package libpam-ldapd, then I proceeded to configure the file /etc/nslcd.conf with the following configuration:
# /etc/nslcd.conf
# nslc
d configuration file. See nslcd.conf(5)
# for details.
# The user and group nslcd should run as.
uid nslcd
gid nslcd
# The location at which the LDAP server(s) should be reachable.
uri ldap://172.17.192.100
# The search base that will be used for all queries.
base DC=myorg,DC=com
# The LDAP protocol version to use.
ldap_version 3
binddn CN=ldapuser,DC=myorg,DC=com
bindpw secret
# The search scope.
#scope sub
filter passwd (objectClass=person)
map passwd uid sAMAccountName
map passwd uidNumber employeeID
map passwd gidNumber objectSid
filter shadow (objectClass=person)
map shadow uid sAMAccountName
Problem is that when logging into the server with user#myorg.com I have the following log (auth sucessfull but search fails due to the #myorg.com section, also it uses the nslcd_pam_authc() function ):
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_initialize(ldap://172.17.192.100)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_set_rebind_proc()
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_simple_bind_s("CN=isldap,DC=TI,DC=ads","***") (uri="ldap://172.17.192.100")
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_result(): end of results (0 total)
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: myldap_search(base="DC=myorg,DC=com", filter="(&(objectClass=person)(sAMAccountName=user#myorg.com))")
nslcd: [8e1f29] <passwd="user#myorg.com"> DEBUG: ldap_result(): end of results (0 total)
nslcd: [e87ccd] DEBUG: connection from pid=9046 uid=0 gid=0
nslcd: [e87ccd] <authc="user#myorg.com"> DEBUG: nslcd_pam_authc("user#myorg.com","sshd","***")
nslcd: [e87ccd] <authc="user#myorg.com"> DEBUG: myldap_search(base="DC=myorg,DC=com", filter="(&(objectClass=person)(sAMAccountName=user#myorg.com))")
nslcd: [e87ccd] <authc="user#myorg.com"> DEBUG: ldap_result(): end of results (0 total)
nslcd: [e87ccd] <authc="user#myorg.com"> DEBUG: myldap_search(base="DC=myorg,DC=com", filter="(&(objectClass=person)(sAMAccountName=user#myorg.com))")
nslcd: [e87ccd] <authc="user#myorg.com"> DEBUG: ldap_result(): end of results (0 total)
nslcd: [e87ccd] <authc="user#myorg.com"> DEBUG: "user#myorg.com": user not found: No such object
If I login using only user the search has success but the authentication does not. (Tries to authenticate using full DN and ldap_sasl_bind() function)
nslcd: [8b4567] <host=10.0.2.2> DEBUG: ldap_simple_bind_s("CN=ldapuserDC=myorg,DC=com","***") (uri="ldap://172.17.192.100")
nslcd: [8b4567] <host=10.0.2.2> DEBUG: ldap_result(): end of results (0 total)
nslcd: [8b4567] <host=10.0.2.2> DEBUG: myldap_search(base="OU=Guatemala Support Team,OU=TI_Service_Accounts,DC=TI,DC=ads", filter="(&(objectClass=ipHost)(ipHostNumber=10.0.2.2))")
nslcd: [8b4567] <host=10.0.2.2> DEBUG: ldap_result(): end of results (0 total)
nslcd: [7b23c6] DEBUG: connection from pid=9099 uid=0 gid=0
nslcd: [7b23c6] <passwd="user"> DEBUG: myldap_search(base="DC=myorg,DC=com", filter="(&(objectClass=person)(sAMAccountName=user))")
nslcd: [7b23c6] <passwd="user"> DEBUG: ldap_initialize(ldap://172.17.192.100)
nslcd: [7b23c6] <passwd="user"> DEBUG: ldap_set_rebind_proc()
nslcd: [7b23c6] <passwd="user"> DEBUG: ldap_simple_bind_s("CN=ldapuser,DC=myorg,DC=com","***") (uri="ldap://172.17.192.100")
nslcd: [7b23c6] <passwd="user"> DEBUG: ldap_result(): CN=User John Doe,DC=myorg,DC=com
nslcd: [7b23c6] <passwd="user"> CN=User John Doe,DC=myorg,DC=com: objectSid: missing
nslcd: [7b23c6] <passwd="user"> DEBUG: ldap_result(): end of results (1 total)
nslcd: [7b23c6] <passwd="user"> DEBUG: myldap_search(base="OU=Guatemala Support Team,OU=TI_Service_Accounts,DC=TI,DC=ads", filter="(&(objectClass=person)(sAMAccountName=user))")
nslcd: [7b23c6] <passwd="user"> DEBUG: ldap_result(): end of results (0 total)
nslcd: [3c9869] DEBUG: connection from pid=9099 uid=0 gid=0
nslcd: [3c9869] <passwd="user"> DEBUG: myldap_search(base="DC=myorg,DC=com", filter="(&(objectClass=person)(sAMAccountName=user))")
nslcd: [3c9869] <passwd="user"> DEBUG: ldap_result(): CN=User John Doe,DC=myorg,DC=com
nslcd: [3c9869] <passwd="user"> CN=User John Doe,DC=myorg,DC=com: objectSid: missing
nslcd: [3c9869] <passwd="user"> DEBUG: ldap_result(): end of results (1 total)
nslcd: [3c9869] <passwd="user"> DEBUG: myldap_search(base="OU=Guatemala Support Team,OU=TI_Service_Accounts,DC=TI,DC=ads", filter="(&(objectClass=person)(sAMAccountName=user))")
nslcd: [3c9869] <passwd="user"> DEBUG: ldap_result(): end of results (0 total)
nslcd: [334873] DEBUG: connection from pid=9099 uid=0 gid=0
nslcd: [334873] <passwd="user"> DEBUG: myldap_search(base="DC=myorg,DC=com", filter="(&(objectClass=person)(sAMAccountName=user))")
nslcd: [334873] <passwd="user"> DEBUG: ldap_result(): CN=User John Doe,DC=myorg,DC=com
nslcd: [334873] <passwd="user"> CN=User John Doe,DC=myorg,DC=com: objectSid: missing
nslcd: [334873] <passwd="user"> DEBUG: ldap_result(): end of results (1 total)
nslcd: [334873] <passwd="user"> DEBUG: myldap_search(base="OU=Guatemala Support Team,OU=TI_Service_Accounts,DC=TI,DC=ads", filter="(&(objectClass=person)(sAMAccountName=user))")
nslcd: [334873] <passwd="user"> DEBUG: ldap_result(): end of results (0 total)
nslcd: [b0dc51] DEBUG: connection from pid=9099 uid=0 gid=0
nslcd: [b0dc51] <authc="user"> DEBUG: nslcd_pam_authc("user","sshd","***")
nslcd: [b0dc51] <authc="user"> DEBUG: myldap_search(base="DC=myorg,DC=com", filter="(&(objectClass=person)(sAMAccountName=user))")
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_initialize(ldap://172.17.192.100)
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_set_rebind_proc()
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_simple_bind_s("CN=ldapuserDC=myorg,DC=com","***") (uri="ldap://172.17.192.100")
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_result(): CN=User John Doe,DC=myorg,DC=com
nslcd: [b0dc51] <authc="user"> DEBUG: myldap_search(base="CN=User John Doe,DC=myorg,DC=com", filter="(objectClass=*)")
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_initialize(ldap://172.17.192.100)
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_set_rebind_proc()
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_sasl_bind("CN=User John Doe,DC=myorg,DC=com","***") (uri="ldap://172.17.192.100")
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_parse_result() result: Invalid credentials: 80090308: LdapErr: DSID-0C0903D0, comment: AcceptSecurityContext error, data 52e, v2580
nslcd: [b0dc51] <authc="user"> DEBUG: failed to bind to LDAP server ldap://172.17.192.100: Invalid credentials: 80090308: LdapErr: DSID-0C0903D0, comment: AcceptSecurityContext error, data 52e, v2580
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_unbind()
nslcd: [b0dc51] <authc="user"> CN=User John Doe,DC=myorg,DC=com: Invalid credentials
nslcd: [b0dc51] <authc="user"> DEBUG: myldap_search(base="DC=myorg,DC=com", filter="(&(objectClass=person)(sAMAccountName=user))")
nslcd: [b0dc51] <authc="user"> DEBUG: ldap_result(): CN=User John Doe,DC=myorg,DC=com
Question: how should I configured nslcd.conf if I wanted to:
Login with user
Search in the sAMAccount field equal to user
Thank you in advance and sorry for the long post.
In /etc/nslcd.conf, Try changing (objectClass=person) to
(&(objectCategory=person)(objectClass=user))
-jim
Running chef 12 on Ubuntu 14.1. Using self signed certs to setup the server, when I try to run knife commands from my client it fails with following error. Any operation has same error. chef server logs do not have any errors or info during the query.
knife config
[root#ip-10-233-2-40 ~]# cat ~/.chef/knife.rb
log_level :debug
log_location STDOUT
node_name 'admin'
client_key '/root/.chef/admin.pem'
validation_client_name 'dev'
validation_key '/root/.chef/dev-validator.pem'
chef_server_url 'https://chef.example.com/organizations/dev'
syntax_check_cache_path '/root/.chef/syntax_check_cache'
root#ip-10-233-2-177:~/ssl-certs# chef-server-ctl status
run: bookshelf: (pid 1092) 1998s; run: log: (pid 1064) 1998s
run: nginx: (pid 6140) 723s; run: log: (pid 1063) 1998s
run: oc_bifrost: (pid 1077) 1998s; run: log: (pid 1058) 1998s
run: oc_id: (pid 1091) 1998s; run: log: (pid 1061) 1998s
run: opscode-erchef: (pid 1090) 1998s; run: log: (pid 1066) 1998s
run: opscode-expander: (pid 1076) 1998s; run: log: (pid 1060) 1998s
run: opscode-expander-reindexer: (pid 1096) 1998s; run: log: (pid 1059) 1998s
run: opscode-solr4: (pid 1075) 1998s; run: log: (pid 1057) 1998s
run: postgresql: (pid 1085) 1998s; run: log: (pid 1056) 1998s
run: rabbitmq: (pid 1062) 1998s; run: log: (pid 1046) 1998s
run: redis_lb: (pid 6124) 723s; run: log: (pid 1065) 1998s
[root#ip-10-233-2-40 ~]# knife environment create staging
ERROR: The object you are looking for could not be found
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:325:in `stream_check': undefined method `closed?' for nil:NilClass (NoMethodError)
from /opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:199:in `read_body'
from /opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:226:in `body'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:499:in `rescue in format_rest_error'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:497:in `format_rest_error'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:459:in `humanize_http_exception'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:418:in `humanize_exception'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:409:in `rescue in run_with_pretty_exceptions'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:400:in `run_with_pretty_exceptions'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:203:in `run'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/knife.rb:142:in `run'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/bin/knife:25:in `<top (required)>'
from /usr/bin/knife:54:in `load'
from /usr/bin/knife:54:in `<main>'`enter code here`
Update
[root#ip-10-233-2-40 ~]# knife client list -VV
INFO: Using configuration from /root/.chef/knife.rb
DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_request
DEBUG: Signing the request as admin
DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_request
DEBUG: Using 10.233.0.182:3128 for proxy
DEBUG: Initiating GET to https://chef.example.com/organizations/dev/clients
DEBUG: ---- HTTP Request Header Data: ----
DEBUG: Accept: application/json
DEBUG: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
DEBUG: X-OPS-SIGN: algorithm=sha1;version=1.0;
DEBUG: X-OPS-USERID: admin
DEBUG: X-OPS-TIMESTAMP: 2015-10-21T17:40:17Z
DEBUG: X-OPS-CONTENT-HASH: 2jmj7l5rSw0yVb/vlWAYkK/YBwk=
DEBUG: X-OPS-AUTHORIZATION-1: m/vlWcZBPE7XUN7qhX6t/T9hXTT+2x/JehpOYq6My1ffEID6n+U+Xc+lHWto
DEBUG: X-OPS-AUTHORIZATION-2: Lq4ZEfNT1ltZkkYZ9Ii8EoF3eajUQmb2buwKMWae3yvxrZ5rgllJPf5q4gy3
DEBUG: X-OPS-AUTHORIZATION-3: IEqUUst+KzmoRHCiC1LeYxKXy+oeo45F4Vw4xHlOWgS0piqXfrmXnkrxs8Um
DEBUG: X-OPS-AUTHORIZATION-4: ZDqdLvcQ10WjoW9Wz4F2+fRh/BdRHjwMF80LVPwrtylf+GbdIhmCU3xxVvOq
DEBUG: X-OPS-AUTHORIZATION-5: w1Z2p03UcpRfMZy1pQV59A0Y3yv57Db5n3PJdjD9TlitNK++/HXcqO3IfO2U
DEBUG: X-OPS-AUTHORIZATION-6: 0QbZYZaeGSkJw0ArQDeffnjbpzAhSXhUfbs+in9tRg==
DEBUG: HOST: chef.example.com:443
DEBUG: X-Ops-Server-API-Version: 1
DEBUG: X-REMOTE-REQUEST-ID: 6a00a52a-7eeb-43d6-920d-fffc685c1b2a
DEBUG: ---- End HTTP Request Header Data ----
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException)
from /opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:128:in `value'
from /opt/chef/embedded/lib/ruby/2.1.0/net/http.rb:915:in `connect'
from /opt/chef/embedded/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
from /opt/chef/embedded/lib/ruby/2.1.0/net/http.rb:852:in `start'
from /opt/chef/embedded/lib/ruby/2.1.0/net/http.rb:1375:in `request'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http/basic_client.rb:65:in `request'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:266:in `block in send_http_request'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:298:in `block in retrying_http_errors'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:296:in `loop'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:296:in `retrying_http_errors'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:260:in `send_http_request'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:143:in `request'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:110:in `get'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/api_client_v1.rb:198:in `list'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife/client_list.rb:38:in `run'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:405:in `block in run_with_pretty_exceptions'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/local_mode.rb:44:in `with_server_connectivity'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:404:in `run_with_pretty_exceptions'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/knife.rb:203:in `run'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/knife.rb:142:in `run'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/bin/knife:25:in `<top (required)>'
from /usr/bin/knife:54:in `load'
from /usr/bin/knife:54:in `<main>'
[root#ip-10-233-2-40 ~]# telnet chef.example.com 443
Trying 10.233.2.177...
Connected to chef.example.com.
Escape character is '^]'.
In continuation to the post - http://forum.spring.io/forum/spring-projects/integration/119697-unable-to-rename-file-in-sftp-remote-directory-please-help
I am using the sftp:outbound-channel-adapter to upload a file into one of the sftp server. Once file is uploaded, API is unable to rename the temporary file sample_test.pgp.writing to sample_test.pgp.
Before uploading the file I verified in the sftp remote folder for file exist and remote folder was completely empty.
When I looked at the debug level log, I could see below message and it is failing at the end with the invalid path error message.
[main] DEBUG: com.ftp.util.FileUploadUtil - Upload for file /sample_test.pgp triggered
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'ftp.uploadgateway.request.channel', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.filter.MessageFilter - org.springframework.integration.filter.MessageFilter#3970ae0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'upload.file.to.sftp', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : org.springframework.integration.handler.LoggingHandler - [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.file.remote.handler.FileTransferringMessageHandler - org.springframework.integration.file.remote.handler.FileTransferringMessageHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : com.jcraft.jsch - Connecting to remote.sever.com port 10022
[main] INFO : com.jcraft.jsch - Connection established
[main] INFO : com.jcraft.jsch - Remote version string: SSH-2.0-SSHD
[main] INFO : com.jcraft.jsch - Local version string: SSH-2.0-JSCH-0.1.49
[main] INFO : com.jcraft.jsch - CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - CheckKexes: diffie-hellman-group14-sha1
[main] INFO : com.jcraft.jsch - diffie-hellman-group14-sha1 is not available.
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT sent
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT received
[main] INFO : com.jcraft.jsch - kex: server: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: server: ssh-rsa
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc,blowfish-cbc
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,blowfish-cbc,aes192-cbc,aes256-cbc,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: client: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: client: ssh-rsa,ssh-dss
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: server->client aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - kex: client->server aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXDH_INIT sent
[main] INFO : com.jcraft.jsch - expecting SSH_MSG_KEXDH_REPLY
[main] INFO : com.jcraft.jsch - ssh_rsa_verify: signature true
[main] WARN : com.jcraft.jsch - Permanently added 'remote.sever.com' (RSA) to the list of known hosts.
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS sent
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS received
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_REQUEST sent
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_ACCEPT received
[main] INFO : com.jcraft.jsch - Authentications that can continue: publickey,keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: publickey
[main] INFO : com.jcraft.jsch - Authentications that can continue: keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: keyboard-interactive
[main] INFO : com.jcraft.jsch - Authentication succeeded (keyboard-interactive).
[main] DEBUG: org.springframework.integration.util.SimplePool - Obtained new org.springframework.integration.sftp.session.SftpSession#6e75d758.
[main] DEBUG: org.springframework.integration.sftp.session.SftpSession - Initial File rename failed, possibly because file already exists. Will attempt to delete file: /inbox/sample_test.pgp and execute rename again.
[main] DEBUG: org.springframework.integration.file.remote.session.CachingSessionFactory - Releasing Session back to the pool.
[main] DEBUG: org.springframework.integration.util.SimplePool - Releasing org.springframework.integration.sftp.session.SftpSession#6e75d758 back to the pool
[main] DEBUG: com.ftp.service.CtrlMPOJO - ERROR UPLOADING FILES EXCEPTION IS
org.springframework.integration.MessageDeliveryException: Error handling message for file [/sample_test.pgp]
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:183)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.filter.MessageFilter.handleRequestMessage(MessageFilter.java:107)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.core.MessagingTemplate.convertAndSend(MessagingTemplate.java:189)
at org.springframework.integration.gateway.MessagingGatewaySupport.send(MessagingGatewaySupport.java:183)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:309)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:269)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:260)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy5.uploadFilesToFTP(Unknown Source)
at com.ftp.util.FileUploadUtil.scanDirectoryAndUpload(FileUploadUtil.java:123)
at com.ftp.service.CtrlMPOJO.main(CtrlMPOJO.java:160)
Caused by: org.springframework.integration.MessagingException: Failed to write to '/inbox/sample_test.pgp.writing' while uploading the file
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:266)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:172)
... 28 more
Caused by: org.springframework.core.NestedIOException: Failed to delete file /inbox/sample_test.pgp; nested exception is org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:157)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.rename(CachingSessionFactory.java:137)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:262)
... 29 more
Caused by: org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.remove(SftpSession.java:71)
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:151)
... 31 more
It works if I set the use-temporary-file-name=false but I do not want to set this flag incase if there is any file watcher job which may pick up incomplete file loaded at the remote server end.
Here is the configuration i have
<int:gateway id="file.upload.gateway"
service-interface="ftp.outbound.FTPUploadGateway"
default-request-channel="ftp.uploadgateway.request.channel"
default-reply-channel="ftp.uploadgateway.response.channel" />
<int:filter
input-channel="ftp.uploadgateway.request.channel"
output-channel="ftp.file.exist.outbound.channel"
discard-channel="upload.file.to.sftp"
expression="${ftp.outbound.remote.file.check.flag:false}">
</int:filter>
<sftp:outbound-channel-adapter id="sftpOutboundAdapter"
session-factory="sftpSessionFactory"
channel="upload.file.to.sftp"
charset="UTF-8"
remote-directory="${ftp.outbound.remote.directory}"
use-temporary-file-name="${ftp.outbound.use.temporary.filename:true}"
remote-filename-generator-expression="${ftp.outbound.remote.filename.expression}"/>
Here are the property values
ftp.outbound.remote.file.check.flag=false
ftp.outbound.remote.directory=/inbox/
ftp.outbound.use.temporary.filename=true
ftp.outbound.remote.filename.expression=payload.getName()
Please show the complete configuration.
Is your sftp user chrooted? If not /inbox/... is trying to manipulate the root directory on the server, and he likely doesn't have permissions for that.
Try removing the leading / from the remote dir.