I used xades4j library for xml digital signature. Everything seems ok with signature, but I realized that verification fails if xml file has special letters or accents that are not from English alphabet, such as é or ñ. I used this code for digital signing and verification, and here is sample xml which fails to verify after signing:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<collection Id="root">
<album>
<title>Questions, unanswered</title>
<artist>Steve and the flubberblubs</artist>
<year>1989</year>
<t:tracks xmlns:t="http://test.xades4j/tracks">
<t:song tracknumber="1" length="4:05">
<t:title>What do you know?</t:title>
<t:artist>Steve é flubberblubs</t:artist>
<t:lastplayed>2006-10-17-08:31</t:lastplayed>
</t:song>
<t:song tracknumber="2" length="3:45">
<t:title>Who do you know?</t:title>
<t:artist>Steve and the flubberblubs</t:artist>
<t:lastplayed>2006-10-17-08:35</t:lastplayed>
</t:song>
<t:song tracknumber="3" length="5:14">
<t:title>When do you know?</t:title>
<t:artist>Steve and the flubberblubs</t:artist>
<t:lastplayed>2006-10-17-08:39</t:lastplayed>
</t:song>
<t:song tracknumber="4" length="4:19">
<t:title>Do you know?</t:title>
<t:artist>Steve and the flubberblubs</t:artist>
<t:lastplayed>2006-10-17-08:44</t:lastplayed>
</t:song>
</t:tracks>
</album>
</collection>
Exception I get is:
[main] WARN org.apache.xml.security.signature.Reference - Verification failed for URI "#root"
[main] WARN org.apache.xml.security.signature.Reference - Expected Digest: KHQDbFVesKu/iTx7o1ad80ADwyDo1/sw/bagFpPMNH0=
[main] WARN org.apache.xml.security.signature.Reference - Actual Digest: S+7XIBhTeXQZ8UOdz1e2gbM1mJFrN9c/pEForMZw0p4=
[main] WARN org.apache.xml.security.signature.Reference - Verification failed for URI "#root"
[main] WARN org.apache.xml.security.signature.Reference - Expected Digest: KHQDbFVesKu/iTx7o1ad80ADwyDo1/sw/bagFpPMNH0=
[main] WARN org.apache.xml.security.signature.Reference - Actual Digest: S+7XIBhTeXQZ8UOdz1e2gbM1mJFrN9c/pEForMZw0p4=
[main] WARN org.apache.xml.security.signature.Reference - Verification failed for URI "#root"
[main] WARN org.apache.xml.security.signature.Reference - Expected Digest: KHQDbFVesKu/iTx7o1ad80ADwyDo1/sw/bagFpPMNH0=
[main] WARN org.apache.xml.security.signature.Reference - Actual Digest: S+7XIBhTeXQZ8UOdz1e2gbM1mJFrN9c/pEForMZw0p4=
Exception in thread "main" xades4j.verification.ReferenceValueException: Reference '#root' cannot be validated
at xades4j.verification.XadesVerifierImpl.doCoreVerification(XadesVerifierImpl.java:313)
at xades4j.verification.XadesVerifierImpl.verify(XadesVerifierImpl.java:195)
at xadesTest.Test2.verifyBes(Test2.java:184)
at xadesTest.Test2.main(Test2.java:84)
If diacritic mark is removed from xml verification is successful. I don't know if code is missing anything or there's somewhere else problem. Thank you in advance.
thanks for share your solution, it resolved my problem for verify XAdES signature:
InputStream iStream = new FileInputStream(SIGNED);
Reader reader = new InputStreamReader(iStream, "utf-8");
InputSource is = new InputSource(reader);
is.setEncoding("utf-8");
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setNamespaceAware(true);
DocumentBuilder builder = factory.newDocumentBuilder();
//Document doc = builder.parse(new InputSource(new FileReader(SIGNED)));
Document doc = builder.parse(is);
.........
Related
I am trying to implement Node.js SDK for my Hyperledger Fabric business network. My network has one organisation, four peers, one channel, solo ordered and one CA.
I have successful managed to issue the JWT identity for user Jim and org1 organization.
When I try to invoke the the chaincode using:
curl -s -X POST \
http://localhost:4000/channels/c1/chaincodes/txman \
-H "authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1MTAwODIyNzMsInVzZXJuYW1lIjoiSmltIiwib3JnTmFtZSI6Im9yZzEiLCJpYXQiOjE1MTAwNDYyNzN9.NTDYky6jj_y1COP163Z_HA6q4v7S1_jN6BxL00Bo-Xo" \
-H "content-type: application/json" \
-d '{
"fcn":"queryTx",
"args":["TX0"]
}'
I am getting the following error:
2017-11-07 11:18:34.286] [DEBUG] SampleWebApp - Decoded from JWT token: username - Jim, orgname - org1
[2017-11-07 11:18:34.287] [DEBUG] SampleWebApp - ==================== INVOKE ON CHAINCODE ==================
[2017-11-07 11:18:34.287] [DEBUG] SampleWebApp - USERNAME: Jim
[2017-11-07 11:18:34.287] [DEBUG] SampleWebApp - ORGNAME: org1
[2017-11-07 11:18:34.287] [DEBUG] SampleWebApp - channelName : c1
[2017-11-07 11:18:34.288] [DEBUG] SampleWebApp - chaincodeName : txman
[2017-11-07 11:18:34.288] [DEBUG] SampleWebApp - fcn : queryTx
[2017-11-07 11:18:34.288] [DEBUG] SampleWebApp - args : TX0
[2017-11-07 11:18:34.292] [DEBUG] invoke-chaincode -
============ invoke transaction on organization org1 ============
[2017-11-07 11:18:34.292] [DEBUG] Helper - [FileKeyValueStore.js]: FileKeyValueStore.js - constructor
[2017-11-07 11:18:34.293] [DEBUG] Helper - [FileKeyValueStore.js]: FileKeyValueStore -- getValue
[2017-11-07 11:18:34.298] [DEBUG] Helper - [crypto_ecdsa_aes]: importKey - start
[2017-11-07 11:18:34.298] [DEBUG] Helper - [crypto_ecdsa_aes]: importKey - have the key [Circular]
[2017-11-07 11:18:34.300] [DEBUG] Helper - [utils.CryptoKeyStore]: _getKeyStore resolving store
[2017-11-07 11:18:34.300] [DEBUG] Helper - [ecdsa/key.js]: ECDSA curve param X: c3cd1a318e957b0200852e4ed6c9601eadc8225eb9f79acc11b1c65df142446a
[2017-11-07 11:18:34.300] [DEBUG] Helper - [ecdsa/key.js]: ECDSA curve param Y: f7bbe9fd43eaae1a0a1d2eb3024ecd79bd63c39a8d1b3b978bc8b4a652d0ef63
[2017-11-07 11:18:34.301] [DEBUG] Helper - [FileKeyValueStore.js]: FileKeyValueStore -- setValue
[2017-11-07 11:18:34.303] [DEBUG] Helper - [utils.CryptoKeyStore]: _getKeyStore resolving store
[2017-11-07 11:18:34.303] [DEBUG] Helper - [FileKeyValueStore.js]: FileKeyValueStore -- getValue
[2017-11-07 11:18:34.304] [DEBUG] Helper - [ecdsa/key.js]: ECDSA curve param X: c3cd1a318e957b0200852e4ed6c9601eadc8225eb9f79acc11b1c65df142446a
[2017-11-07 11:18:34.304] [DEBUG] Helper - [ecdsa/key.js]: ECDSA curve param Y: f7bbe9fd43eaae1a0a1d2eb3024ecd79bd63c39a8d1b3b978bc8b4a652d0ef63
[2017-11-07 11:18:34.305] [DEBUG] Helper - [ecdsa/key.js]: ECDSA curve param X: c3cd1a318e957b0200852e4ed6c9601eadc8225eb9f79acc11b1c65df142446a
[2017-11-07 11:18:34.306] [DEBUG] Helper - [ecdsa/key.js]: ECDSA curve param Y: f7bbe9fd43eaae1a0a1d2eb3024ecd79bd63c39a8d1b3b978bc8b4a652d0ef63
[2017-11-07 11:18:34.308] [DEBUG] Helper - [ecdsa/key.js]: ECDSA curve param X: c3cd1a318e957b0200852e4ed6c9601eadc8225eb9f79acc11b1c65df142446a
[2017-11-07 11:18:34.308] [DEBUG] Helper - [ecdsa/key.js]: ECDSA curve param Y: f7bbe9fd43eaae1a0a1d2eb3024ecd79bd63c39a8d1b3b978bc8b4a652d0ef63
[2017-11-07 11:18:34.309] [DEBUG] Helper - [FileKeyValueStore.js]: FileKeyValueStore -- setValue
[2017-11-07 11:18:34.309] [INFO] Helper - Successfully loaded member from persistence
[2017-11-07 11:18:34.313] [DEBUG] invoke-chaincode - Sending transaction "{"_nonce":{"type":"Buffer","data":[218,130,209,154,87,123,73,36,118,155,194,91,27,250,14,216,117,202,164,5,16,237,96,208]},"_transaction_id":"dba107be47745c51eb75b6a927d68f0afd58b325c5e36326b40a729f9d858f43"}"
[2017-11-07 11:18:34.341] [DEBUG] Helper - [crypto_ecdsa_aes]: ecdsa signature: Signature {
r: <BN: cf5e0aaef0fd10d9abf468e025c6f6c1e8991d0dcddf5f5aa6b285ebdcd920ad>,
s: <BN: 8de356dcfda3ba3fb0d6decfc0e43995b3297bc7037fd3997bfba561e6d626d>,
recoveryParam: 0 }
error: [client-utils.js]: sendPeersProposal - Promise is rejected: Error: Failed to deserialize creator identity, err The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "ca.org1.example.com")
at /home/brankoterzic/Desktop/devGC/txman/node_modules/grpc/src/client.js:554:15
error: [client-utils.js]: sendPeersProposal - Promise is rejected: Error: Failed to deserialize creator identity, err The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "ca.org1.example.com")
at /home/brankoterzic/Desktop/devGC/txman/node_modules/grpc/src/client.js:554:15
error: [client-utils.js]: sendPeersProposal - Promise is rejected: Error: Failed to deserialize creator identity, err The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "ca.org1.example.com")
at /home/brankoterzic/Desktop/devGC/txman/node_modules/grpc/src/client.js:554:15
error: [client-utils.js]: sendPeersProposal - Promise is rejected: Error: Failed to deserialize creator identity, err The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "ca.org1.example.com")
at /home/brankoterzic/Desktop/devGC/txman/node_modules/grpc/src/client.js:554:15
error: [client-utils.js]: sendPeersProposal - Promise is rejected: Error: Failed to deserialize creator identity, err The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "ca.org1.example.com")
at /home/brankoterzic/Desktop/devGC/txman/node_modules/grpc/src/client.js:554:15
[2017-11-07 11:18:34.400] [ERROR] invoke-chaincode - transaction proposal was bad
[2017-11-07 11:18:34.401] [ERROR] invoke-chaincode - transaction proposal was bad
[2017-11-07 11:18:34.401] [ERROR] invoke-chaincode - transaction proposal was bad
[2017-11-07 11:18:34.401] [ERROR] invoke-chaincode - transaction proposal was bad
[2017-11-07 11:18:34.401] [ERROR] invoke-chaincode - transaction proposal was bad
[2017-11-07 11:18:34.401] [ERROR] invoke-chaincode - Failed to send Proposal or receive valid response. Response null or status is not 200. exiting...
[2017-11-07 11:18:34.401] [ERROR] invoke-chaincode - Failed to order the transaction. Error code: undefined
I was following the balance-transfer Node.js example: https://github.com/hyperledger/fabric-samples/tree/release/balance-transfer
Thank you for your answers!
I have solved this by setting up the generated crypto for CA
crypto-config/peerOrganizations/org1.example.com/ca/361a9254173cb1a97c057432094a20db9578ca69a152725f96323e00379c5ad6_sk
to
FABRIC_CA_SERVER_CA_KEYFILE=/etc/hyperledger/fabric-ca-server-config/361a9254173cb1a97c057432094a20db9578ca69a152725f96323e00379c5ad6_sk
in docker-compose-base.yaml env variable.
I followed https://wiki.apache.org/nutch/NutchTutorial and trying to install and integrate Nutch 1.12 with Solr 5.5.2. I installed Nutch by following steps mentioned in tutorial but when trying to integrate with solr by running below command. It is throwing the below exception.
bin/nutch index http://10.209.18.213:8983/solr crawl/crawldb/ -linkdb crawl/linkdb/ crawl/segments/* -filter -normalize
Exception
2016-08-11 09:18:40,076 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-08-11 09:18:40,383 WARN segment.SegmentChecker - The input path at crawldb is not a segment... skipping
2016-08-11 09:18:40,397 INFO segment.SegmentChecker - Segment dir is complete: crawl/segments/20160810110110.
2016-08-11 09:18:40,403 INFO segment.SegmentChecker - Segment dir is complete: crawl/segments/20160810112551.
2016-08-11 09:18:40,408 INFO segment.SegmentChecker - Segment dir is complete: crawl/segments/20160810112952.
2016-08-11 09:18:40,409 INFO indexer.IndexingJob - Indexer: starting at 2016-08-11 09:18:40
2016-08-11 09:18:40,415 INFO indexer.IndexingJob - Indexer: deleting gone documents: false
2016-08-11 09:18:40,415 INFO indexer.IndexingJob - Indexer: URL filtering: true
2016-08-11 09:18:40,415 INFO indexer.IndexingJob - Indexer: URL normalizing: true
2016-08-11 09:18:40,672 INFO indexer.IndexWriters - Adding org.apache.nutch.indexwriter.solr.SolrIndexWriter
2016-08-11 09:18:40,672 INFO indexer.IndexingJob - Active IndexWriters :
SOLRIndexWriter
solr.server.url : URL of the SOLR instance
solr.zookeeper.hosts : URL of the Zookeeper quorum
solr.commit.size : buffer size when sending to SOLR (default 1000)
solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
solr.auth : use authentication (default false)
solr.auth.username : username for authentication
solr.auth.password : password for authentication
2016-08-11 09:18:40,677 INFO indexer.IndexerMapReduce - IndexerMapReduce: crawldb: http://10.209.18.213:8983/solr
2016-08-11 09:18:40,677 INFO indexer.IndexerMapReduce - IndexerMapReduce: linkdb: crawl/linkdb
2016-08-11 09:18:40,677 INFO indexer.IndexerMapReduce - IndexerMapReduces: adding segment: crawl/segments/20160810110110
2016-08-11 09:18:40,683 INFO indexer.IndexerMapReduce - IndexerMapReduces: adding segment: crawl/segments/20160810112551
2016-08-11 09:18:40,684 INFO indexer.IndexerMapReduce - IndexerMapReduces: adding segment: crawl/segments/20160810112952
2016-08-11 09:18:41,362 ERROR indexer.IndexingJob - Indexer: java.io.IOException: No FileSystem for scheme: http
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2385)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:256)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304)
at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:833)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:145)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:228)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:237)
The tutorial still mentions the deprecated solrindex command. The index command should be
bin/nutch index -Dsolr.server.url=http://.../solr crawldb/ -linkdb linkdb/ segments/*
Without argument Nutch commands show a command-line help:
bin/nutch index
Usage: Indexer <crawldb> [-linkdb <linkdb>] [-params k1=v1&k2=v2...] (<segment> ... | -dir <segments>) [-noCommit] [-deleteGone] [-filter] [-normalize] [-addBinaryContent] [-base64]
Active IndexWriters :
SOLRIndexWriter
solr.server.url : URL of the SOLR instance
solr.zookeeper.hosts : URL of the Zookeeper quorum
solr.commit.size : buffer size when sending to SOLR (default 1000)
solr.mapping.file : name of the mapping file for fields (default solrindex-mapping.xml)
solr.auth : use authentication (default false)
solr.auth.username : username for authentication
solr.auth.password : password for authentication
I am using Spark's FP-growth algorithm. I was getting OOM errors when I was doing a collect, I then changed the code so that I can save the results in a text file on HDFS rather than collecting them on the driver node. Here is the related code:
// Model building:
val fpg = new FPGrowth()
.setMinSupport(0.01)
.setNumPartitions(10)
val model = fpg.run(transaction_distinct)
Here is a transformation that should give me RDD[Strings].
val mymodel = model.freqItemsets.map { itemset =>
val model_res = itemset.items.mkString("[", ",", "]") + ", " + itemset.freq
model_res
}
I then save the model results as. Unfortunately, this is really SLOW!!!
mymodel.saveAsTextFile("fpm_model")
I get these errors:
16/02/04 14:47:28 ERROR ErrorMonitor: AssociationError[akka.tcp://sparkDriver#ipaddress:46811] -> [akka.tcp://sparkExecutor#hostname:39720]: Error [Association failed with [akka.tcp://sparkExecutor#hostname:39720]][akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor#hostname:39720]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: hostname/ipaddress:39720] akka.event.Logging$Error$NoCause$
16/02/04 14:47:28 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(3, hostname, 58683)
16/02/04 14:47:28 INFO BlockManagerMaster: Removed 3 successfully in removeExecutor
16/02/04 14:47:28 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver#ipaddress:46811] ->[akka.tcp://sparkExecutor#hostname:39720]: Error [Association failed with [akka.tcp://sparkExecutor#hostname:39720]][akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor#hostname:39720]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: hostname/ipaddress:39720
How to integrate a Pollable channel in the Integration flow. As my flow is using a NullChannel instead of Pollable Channel.
Can some one kindly provide any pointer on this.
This is my Integration flow.
#Bean
public IntegrationFlow sftpInboundFlow() {
System.out.println("enter sftpInboundFlow....."
+ sftpSessionFactory.getSession());
System.out.println("the channel is " + sftpInboundResultChannel);
return IntegrationFlows
.from(Sftp.inboundAdapter(this.sftpSessionFactory)
.preserveTimestamp(true).remoteDirectory(remDir)
.regexFilter(".*\\.txt$")
.localFilenameExpression("#this.toUpperCase()")
.localDirectory(new File(localDir))
.remoteFileSeparator("/"),
new Consumer<SourcePollingChannelAdapterSpec>() {
#Override
public void accept(SourcePollingChannelAdapterSpec e) {
e.id("sftpInboundAdapter")
.autoStartup(true)
.poller(Pollers.fixedRate(1000)
.maxMessagesPerPoll(1));
}
})
.channel(MessageChannels.queue("sftpInboundResultChannel"))
.get();
}
And I want to use this..
#Autowired
private PollableChannel sftpInboundResultChannel;
in the .channel() method.
And the logs are...
[2015-07-22 18:48:05.497] - 2796 INFO [main] --- com.jcraft.jsch: SSH_MSG_NEWKEYS sent
[2015-07-22 18:48:05.497] - 2796 INFO [main] --- com.jcraft.jsch: SSH_MSG_NEWKEYS received
[2015-07-22 18:48:05.500] - 2796 INFO [main] --- com.jcraft.jsch: SSH_MSG_SERVICE_REQUEST sent
[2015-07-22 18:48:05.501] - 2796 INFO [main] --- com.jcraft.jsch: SSH_MSG_SERVICE_ACCEPT received
[2015-07-22 18:48:15.517] - 2796 INFO [main] --- com.jcraft.jsch: Authentications that can continue: gssapi-with-mic,publickey,keyboard-interactive,password
[2015-07-22 18:48:15.517] - 2796 INFO [main] --- com.jcraft.jsch: Next authentication method: gssapi-with-mic
[2015-07-22 18:48:15.522] - 2796 INFO [main] --- com.jcraft.jsch: Authentications that can continue: publickey,keyboard-interactive,password
[2015-07-22 18:48:15.522] - 2796 INFO [main] --- com.jcraft.jsch: Next authentication method: publickey
[2015-07-22 18:48:15.523] - 2796 INFO [main] --- com.jcraft.jsch: Authentications that can continue: password
[2015-07-22 18:48:15.523] - 2796 INFO [main] --- com.jcraft.jsch: Next authentication method: password
[2015-07-22 18:48:15.537] - 2796 INFO [main] --- com.jcraft.jsch: Authentication succeeded (password).
enter sftpSessionFactory.....org.springframework.integration.sftp.session.SftpSession#1ffcd26
enter sftpInboundFlow.....org.springframework.integration.sftp.session.SftpSession#13a462c
the channel is org.springframework.integration.channel.NullChannel#d532ee
[2015-07-22 18:48:15.827] - 2796 INFO [main] --- org.springframework.context.support.DefaultLifecycleProcessor: Starting beans in phase -2147483648
[2015-07-22 18:48:15.828] - 2796 INFO [main] --- org.springframework.context.support.DefaultLifecycleProcessor: Starting beans in phase 0
[2015-07-22 18:48:15.828] - 2796 INFO [main] --- org.springframework.integration.endpoint.EventDrivenConsumer: Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
[2015-07-22 18:48:15.828] - 2796 INFO [main] --- org.springframework.integration.channel.PublishSubscribeChannel: Channel 'application.errorChannel' has 1 subscriber(s).
[2015-07-22 18:48:15.829] - 2796 INFO [main] --- org.springframework.integration.endpoint.EventDrivenConsumer: started _org.springframework.integration.errorLogger
[2015-07-22 18:48:15.829] - 2796 INFO [main] --- org.springframework.context.support.DefaultLifecycleProcessor: Starting beans in phase 1073741823
[2015-07-22 18:48:15.832] - 2796 INFO [main] --- org.springframework.integration.endpoint.SourcePollingChannelAdapter: started sftpInboundAdapter
[2015-07-22 18:48:15.834] - 2796 INFO [main] --- src.MainSftpInBoundDsl: Started MainSftpInBoundDsl in 11.019 seconds (JVM running for 11.182)
enter main.....
enter main.....
[2015-07-22 18:48:16.040] - 2796 INFO [task-scheduler-1] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\AMARJEET.TXT, headers={timestamp=1437571096040, id=ddc20f6d-26ee-7eef-f59e-b5207dcdc8c7}]]
[2015-07-22 18:48:16.832] - 2796 INFO [task-scheduler-1] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\INFO.TXT, headers={timestamp=1437571096832, id=361b55d5-884e-9001-d513-ddc81926bf85}]]
[2015-07-22 18:48:17.832] - 2796 INFO [task-scheduler-2] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\KKKK.TXT, headers={timestamp=1437571097832, id=49deb17c-50e3-4315-dfa2-cc443a35bd1c}]]
[2015-07-22 18:48:18.832] - 2796 INFO [task-scheduler-1] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\LA.TXT, headers={timestamp=1437571098832, id=88def298-ab86-1d63-039e-c81c70ae6819}]]
[2015-07-22 18:48:19.832] - 2796 INFO [task-scheduler-3] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\POC.TXT, headers={timestamp=1437571099832, id=d78b36eb-10f5-ce83-ba4c-cd24a806480b}]]
[2015-07-22 18:48:20.832] - 2796 INFO [task-scheduler-2] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\PPPP.TXT, headers={timestamp=1437571100832, id=4e727ebb-b4de-be19-6a92-afb3be24576f}]]
[2015-07-22 18:48:21.832] - 2796 INFO [task-scheduler-4] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\REFERENCE.TXT, headers={timestamp=1437571101832, id=3d611371-431c-6b1e-0e17-377f5607de96}]]
[2015-07-22 18:48:22.831] - 2796 INFO [task-scheduler-1] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\RRRRR.TXT, headers={timestamp=1437571102831, id=707e566c-ccfa-6c32-16d6-1f27ec73abaf}]]
[2015-07-22 18:48:23.832] - 2796 INFO [task-scheduler-5] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\SFTP.TXT, headers={timestamp=1437571103832, id=b3c979ff-fcd2-e1ff-8b37-6b8f2803637a}]]
[2015-07-22 18:48:24.832] - 2796 INFO [task-scheduler-3] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\TEST.TXT, headers={timestamp=1437571104832, id=98d43e62-8e5f-69bc-7617-4706761eaf86}]]
[2015-07-22 18:48:25.832] - 2796 INFO [task-scheduler-3] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\XSS.TXT, headers={timestamp=1437571105832, id=77b28259-0287-0ef5-825f-5345d6c21c21}]]
[2015-07-22 18:48:26.832] - 2796 INFO [task-scheduler-3] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\ZZZZ.TXT, headers={timestamp=1437571106832, id=b5644a62-8c8c-86d2-e016-75fff01c70b4}]]
Just use the autowired field directly in the flow...
.channel(this.sftpInboundResultChannel)
EDIT:
It looks like MessageChannels.queue("sftpInboundResultChannel") is not registering the channel as a bean so Spring is autowiring NullChannel into that field (because it's the only PollableChannel in the context).
We need to take a look at that; in the meantime, add
#Bean
public PollableChannel sftpInboundResultChannel() {
return new QueueChannel();
}
In continuation to the post - http://forum.spring.io/forum/spring-projects/integration/119697-unable-to-rename-file-in-sftp-remote-directory-please-help
I am using the sftp:outbound-channel-adapter to upload a file into one of the sftp server. Once file is uploaded, API is unable to rename the temporary file sample_test.pgp.writing to sample_test.pgp.
Before uploading the file I verified in the sftp remote folder for file exist and remote folder was completely empty.
When I looked at the debug level log, I could see below message and it is failing at the end with the invalid path error message.
[main] DEBUG: com.ftp.util.FileUploadUtil - Upload for file /sample_test.pgp triggered
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'ftp.uploadgateway.request.channel', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.filter.MessageFilter - org.springframework.integration.filter.MessageFilter#3970ae0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'upload.file.to.sftp', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : org.springframework.integration.handler.LoggingHandler - [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.file.remote.handler.FileTransferringMessageHandler - org.springframework.integration.file.remote.handler.FileTransferringMessageHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : com.jcraft.jsch - Connecting to remote.sever.com port 10022
[main] INFO : com.jcraft.jsch - Connection established
[main] INFO : com.jcraft.jsch - Remote version string: SSH-2.0-SSHD
[main] INFO : com.jcraft.jsch - Local version string: SSH-2.0-JSCH-0.1.49
[main] INFO : com.jcraft.jsch - CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - CheckKexes: diffie-hellman-group14-sha1
[main] INFO : com.jcraft.jsch - diffie-hellman-group14-sha1 is not available.
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT sent
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT received
[main] INFO : com.jcraft.jsch - kex: server: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: server: ssh-rsa
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc,blowfish-cbc
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,blowfish-cbc,aes192-cbc,aes256-cbc,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: client: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: client: ssh-rsa,ssh-dss
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: server->client aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - kex: client->server aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXDH_INIT sent
[main] INFO : com.jcraft.jsch - expecting SSH_MSG_KEXDH_REPLY
[main] INFO : com.jcraft.jsch - ssh_rsa_verify: signature true
[main] WARN : com.jcraft.jsch - Permanently added 'remote.sever.com' (RSA) to the list of known hosts.
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS sent
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS received
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_REQUEST sent
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_ACCEPT received
[main] INFO : com.jcraft.jsch - Authentications that can continue: publickey,keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: publickey
[main] INFO : com.jcraft.jsch - Authentications that can continue: keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: keyboard-interactive
[main] INFO : com.jcraft.jsch - Authentication succeeded (keyboard-interactive).
[main] DEBUG: org.springframework.integration.util.SimplePool - Obtained new org.springframework.integration.sftp.session.SftpSession#6e75d758.
[main] DEBUG: org.springframework.integration.sftp.session.SftpSession - Initial File rename failed, possibly because file already exists. Will attempt to delete file: /inbox/sample_test.pgp and execute rename again.
[main] DEBUG: org.springframework.integration.file.remote.session.CachingSessionFactory - Releasing Session back to the pool.
[main] DEBUG: org.springframework.integration.util.SimplePool - Releasing org.springframework.integration.sftp.session.SftpSession#6e75d758 back to the pool
[main] DEBUG: com.ftp.service.CtrlMPOJO - ERROR UPLOADING FILES EXCEPTION IS
org.springframework.integration.MessageDeliveryException: Error handling message for file [/sample_test.pgp]
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:183)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.filter.MessageFilter.handleRequestMessage(MessageFilter.java:107)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.core.MessagingTemplate.convertAndSend(MessagingTemplate.java:189)
at org.springframework.integration.gateway.MessagingGatewaySupport.send(MessagingGatewaySupport.java:183)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:309)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:269)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:260)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy5.uploadFilesToFTP(Unknown Source)
at com.ftp.util.FileUploadUtil.scanDirectoryAndUpload(FileUploadUtil.java:123)
at com.ftp.service.CtrlMPOJO.main(CtrlMPOJO.java:160)
Caused by: org.springframework.integration.MessagingException: Failed to write to '/inbox/sample_test.pgp.writing' while uploading the file
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:266)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:172)
... 28 more
Caused by: org.springframework.core.NestedIOException: Failed to delete file /inbox/sample_test.pgp; nested exception is org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:157)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.rename(CachingSessionFactory.java:137)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:262)
... 29 more
Caused by: org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.remove(SftpSession.java:71)
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:151)
... 31 more
It works if I set the use-temporary-file-name=false but I do not want to set this flag incase if there is any file watcher job which may pick up incomplete file loaded at the remote server end.
Here is the configuration i have
<int:gateway id="file.upload.gateway"
service-interface="ftp.outbound.FTPUploadGateway"
default-request-channel="ftp.uploadgateway.request.channel"
default-reply-channel="ftp.uploadgateway.response.channel" />
<int:filter
input-channel="ftp.uploadgateway.request.channel"
output-channel="ftp.file.exist.outbound.channel"
discard-channel="upload.file.to.sftp"
expression="${ftp.outbound.remote.file.check.flag:false}">
</int:filter>
<sftp:outbound-channel-adapter id="sftpOutboundAdapter"
session-factory="sftpSessionFactory"
channel="upload.file.to.sftp"
charset="UTF-8"
remote-directory="${ftp.outbound.remote.directory}"
use-temporary-file-name="${ftp.outbound.use.temporary.filename:true}"
remote-filename-generator-expression="${ftp.outbound.remote.filename.expression}"/>
Here are the property values
ftp.outbound.remote.file.check.flag=false
ftp.outbound.remote.directory=/inbox/
ftp.outbound.use.temporary.filename=true
ftp.outbound.remote.filename.expression=payload.getName()
Please show the complete configuration.
Is your sftp user chrooted? If not /inbox/... is trying to manipulate the root directory on the server, and he likely doesn't have permissions for that.
Try removing the leading / from the remote dir.