In continuation to the post - http://forum.spring.io/forum/spring-projects/integration/119697-unable-to-rename-file-in-sftp-remote-directory-please-help
I am using the sftp:outbound-channel-adapter to upload a file into one of the sftp server. Once file is uploaded, API is unable to rename the temporary file sample_test.pgp.writing to sample_test.pgp.
Before uploading the file I verified in the sftp remote folder for file exist and remote folder was completely empty.
When I looked at the debug level log, I could see below message and it is failing at the end with the invalid path error message.
[main] DEBUG: com.ftp.util.FileUploadUtil - Upload for file /sample_test.pgp triggered
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'ftp.uploadgateway.request.channel', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.filter.MessageFilter - org.springframework.integration.filter.MessageFilter#3970ae0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'upload.file.to.sftp', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : org.springframework.integration.handler.LoggingHandler - [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.file.remote.handler.FileTransferringMessageHandler - org.springframework.integration.file.remote.handler.FileTransferringMessageHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : com.jcraft.jsch - Connecting to remote.sever.com port 10022
[main] INFO : com.jcraft.jsch - Connection established
[main] INFO : com.jcraft.jsch - Remote version string: SSH-2.0-SSHD
[main] INFO : com.jcraft.jsch - Local version string: SSH-2.0-JSCH-0.1.49
[main] INFO : com.jcraft.jsch - CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - CheckKexes: diffie-hellman-group14-sha1
[main] INFO : com.jcraft.jsch - diffie-hellman-group14-sha1 is not available.
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT sent
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT received
[main] INFO : com.jcraft.jsch - kex: server: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: server: ssh-rsa
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc,blowfish-cbc
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,blowfish-cbc,aes192-cbc,aes256-cbc,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: client: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: client: ssh-rsa,ssh-dss
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: server->client aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - kex: client->server aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXDH_INIT sent
[main] INFO : com.jcraft.jsch - expecting SSH_MSG_KEXDH_REPLY
[main] INFO : com.jcraft.jsch - ssh_rsa_verify: signature true
[main] WARN : com.jcraft.jsch - Permanently added 'remote.sever.com' (RSA) to the list of known hosts.
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS sent
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS received
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_REQUEST sent
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_ACCEPT received
[main] INFO : com.jcraft.jsch - Authentications that can continue: publickey,keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: publickey
[main] INFO : com.jcraft.jsch - Authentications that can continue: keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: keyboard-interactive
[main] INFO : com.jcraft.jsch - Authentication succeeded (keyboard-interactive).
[main] DEBUG: org.springframework.integration.util.SimplePool - Obtained new org.springframework.integration.sftp.session.SftpSession#6e75d758.
[main] DEBUG: org.springframework.integration.sftp.session.SftpSession - Initial File rename failed, possibly because file already exists. Will attempt to delete file: /inbox/sample_test.pgp and execute rename again.
[main] DEBUG: org.springframework.integration.file.remote.session.CachingSessionFactory - Releasing Session back to the pool.
[main] DEBUG: org.springframework.integration.util.SimplePool - Releasing org.springframework.integration.sftp.session.SftpSession#6e75d758 back to the pool
[main] DEBUG: com.ftp.service.CtrlMPOJO - ERROR UPLOADING FILES EXCEPTION IS
org.springframework.integration.MessageDeliveryException: Error handling message for file [/sample_test.pgp]
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:183)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.filter.MessageFilter.handleRequestMessage(MessageFilter.java:107)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.core.MessagingTemplate.convertAndSend(MessagingTemplate.java:189)
at org.springframework.integration.gateway.MessagingGatewaySupport.send(MessagingGatewaySupport.java:183)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:309)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:269)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:260)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy5.uploadFilesToFTP(Unknown Source)
at com.ftp.util.FileUploadUtil.scanDirectoryAndUpload(FileUploadUtil.java:123)
at com.ftp.service.CtrlMPOJO.main(CtrlMPOJO.java:160)
Caused by: org.springframework.integration.MessagingException: Failed to write to '/inbox/sample_test.pgp.writing' while uploading the file
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:266)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:172)
... 28 more
Caused by: org.springframework.core.NestedIOException: Failed to delete file /inbox/sample_test.pgp; nested exception is org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:157)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.rename(CachingSessionFactory.java:137)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:262)
... 29 more
Caused by: org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.remove(SftpSession.java:71)
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:151)
... 31 more
It works if I set the use-temporary-file-name=false but I do not want to set this flag incase if there is any file watcher job which may pick up incomplete file loaded at the remote server end.
Here is the configuration i have
<int:gateway id="file.upload.gateway"
service-interface="ftp.outbound.FTPUploadGateway"
default-request-channel="ftp.uploadgateway.request.channel"
default-reply-channel="ftp.uploadgateway.response.channel" />
<int:filter
input-channel="ftp.uploadgateway.request.channel"
output-channel="ftp.file.exist.outbound.channel"
discard-channel="upload.file.to.sftp"
expression="${ftp.outbound.remote.file.check.flag:false}">
</int:filter>
<sftp:outbound-channel-adapter id="sftpOutboundAdapter"
session-factory="sftpSessionFactory"
channel="upload.file.to.sftp"
charset="UTF-8"
remote-directory="${ftp.outbound.remote.directory}"
use-temporary-file-name="${ftp.outbound.use.temporary.filename:true}"
remote-filename-generator-expression="${ftp.outbound.remote.filename.expression}"/>
Here are the property values
ftp.outbound.remote.file.check.flag=false
ftp.outbound.remote.directory=/inbox/
ftp.outbound.use.temporary.filename=true
ftp.outbound.remote.filename.expression=payload.getName()
Please show the complete configuration.
Is your sftp user chrooted? If not /inbox/... is trying to manipulate the root directory on the server, and he likely doesn't have permissions for that.
Try removing the leading / from the remote dir.
Related
I have a integration flow.
IntegrationFlows.from(
Sftp.inboundAdapter(inboundSftp)
.localDirectory(this.getlocalDirectory(config.getId()))
.deleteRemoteFiles(true)
.autoCreateLocalDirectory(true)
.filter(new CompositeFileListFilter().addFilter(new LastModifiedLsEntryFileListFilter(config.getRegexFilter())))
.remoteDirectory(config.getInboundDirectory())
, e -> e.poller(
Pollers.cron(config.getCron())
.errorChannel(MessageHeaders.ERROR_CHANNEL).errorHandler((ex) -> {
try {
this.destroy(String.valueOf(config.getId()));
configurationService.removeConfigurationChannelById(config.getId());
//loggin here
}
} catch (Exception ex1) {
Logger.getLogger(ExceptionAspect.class.getName()).log(Level.SEVERE, null, ex1);
}
})
.advice(startup.scanRemoteDirectory())
))
.transform(
file -> util.transform((File) file, config.getSourceEncoding(), config.getTargetEncoding(), doEncoding, doZip))
.publishSubscribeChannel(s -> s
.subscribe(f -> {
f.handle(Sftp.outboundAdapter(outboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getOutboundDirectory()), c -> c.advice(startup.deleteFileAdvice()));
})
.subscribe(f -> {
if(doArchive) {
f.handle(Sftp.outboundAdapter(inboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getInboundArchiveDirectory()));
} else {
f.handle(m -> {});
}
})
.subscribe(f -> f
.handle(m -> {
// file transfer logging here
if(doArchive) {
// file archived logging here
}
}
})
)
)
.get();
in 1st subscriber, if file is failed to upload it still printing logs that file is transferred but actually its not.
what i understand message will be pass to each on subscriber and when 1st subscriber complete its work it will sent to next one.
In my case first subscriber is actually fail to upload file. Advice is not removing file.
I did try few things.
IntegrationFlows.from(
Sftp.inboundAdapter(inboundSftp)
..........
))
.transform(....
)
.publishSubscribeChannel(s -> s
.subscribe(f -> {
f.handle(Sftp.outboundAdapter(outboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getOutboundDirectory()), c -> c.advice(startup.deleteFileAdvice()));
}).publishSubcribeChannel(s -> s
.subscribe(f -> {
if(doArchive) {
f.handle(Sftp.outboundAdapter(inboundSftp)
.useTemporaryFileName(false)
.autoCreateDirectory(true)
.remoteDirectory(config.getInboundArchiveDirectory()));
} else {
f.handle(m -> {});
}
})
.subscribe(f -> f
.handle(m -> {
// file transfer logging here
if(doArchive) {
// file archived logging here
}
}
})
)
)
)
.get();
I try to publish again after outboundAdapter subscriber, but in error it still published.
Also try to add .errorChannel in outboundAdapter subscriber, didn't work for me.
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.2.RELEASE)
2020-08-20 11:10:53,265 INFO c.t.i.s.I2SftpApplication - Starting I2SftpApplication on GSEUC5CG8393GR5 with PID 21636 (C:\Users\MuhammadUmair\IdeaProjects\i2sftpinboudservice\target\classes started by MuhammadUmair in C:\Users\MuhammadUmair\IdeaProjects\i2sftpinboudservice)
2020-08-20 11:10:53,269 INFO c.t.i.s.I2SftpApplication - No active profile set, falling back to default profiles: default
2020-08-20 11:10:54,464 INFO o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2020-08-20 11:10:54,478 INFO o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2020-08-20 11:10:54,577 INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'integrationChannelResolver' of type [org.springframework.integration.support.channel.BeanFactoryChannelResolver] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-20 11:10:54,584 INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'integrationDisposableAutoCreatedBeans' of type [org.springframework.integration.config.annotation.Disposables] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-20 11:10:54,597 INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.integration.config.IntegrationManagementConfiguration' of type [org.springframework.integration.config.IntegrationManagementConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-08-20 11:10:54,906 INFO o.s.b.w.e.t.TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
2020-08-20 11:10:54,917 INFO o.a.c.h.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
2020-08-20 11:10:54,917 INFO o.a.c.c.StandardService - Starting service [Tomcat]
2020-08-20 11:10:54,917 INFO o.a.c.c.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.29]
2020-08-20 11:10:55,095 INFO o.a.c.c.C.[.[.[/] - Initializing Spring embedded WebApplicationContext
2020-08-20 11:10:55,095 INFO o.s.w.c.ContextLoader - Root WebApplicationContext: initialization completed in 1705 ms
2020-08-20 11:10:56,920 INFO o.s.s.c.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-08-20 11:10:57,243 INFO o.s.s.c.ThreadPoolTaskScheduler - Initializing ExecutorService 'taskScheduler'
2020-08-20 11:10:57,415 INFO o.s.i.e.EventDrivenConsumer - Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2020-08-20 11:10:57,416 INFO o.s.i.c.PublishSubscribeChannel - Channel 'application.errorChannel' has 1 subscriber(s).
2020-08-20 11:10:57,416 INFO o.s.i.e.EventDrivenConsumer - started bean '_org.springframework.integration.errorLogger'
2020-08-20 11:10:57,416 INFO o.s.i.e.EventDrivenConsumer - Adding {message-handler:startup.resultFileHandler.serviceActivator} as a subscriber to the 'fromSftpChannel' channel
2020-08-20 11:10:57,416 INFO o.s.i.c.DirectChannel - Channel 'application.fromSftpChannel' has 1 subscriber(s).
2020-08-20 11:10:57,417 INFO o.s.i.e.EventDrivenConsumer - started bean 'startup.resultFileHandler.serviceActivator'
2020-08-20 11:10:57,422 INFO o.s.i.e.SourcePollingChannelAdapter - started bean 'startup.sftpMessageSource.inboundChannelAdapter'
2020-08-20 11:10:57,441 INFO o.a.c.h.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8080"]
2020-08-20 11:10:57,498 INFO o.s.b.w.e.t.TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path ''
2020-08-20 11:10:57,508 INFO c.t.i.s.I2SftpApplication - Started I2SftpApplication in 9.228 seconds (JVM running for 13.133)
Registering an Integration Flow with id : 1
2020-08-20 11:10:57,689 INFO o.s.i.c.DirectChannel - Channel 'application.1.subFlow#2.channel#0' has 1 subscriber(s).
2020-08-20 11:10:57,690 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#2.org.springframework.integration.config.ConsumerEndpointFactoryBean#1'
2020-08-20 11:10:57,691 INFO o.s.i.e.EventDrivenConsumer - Adding {bridge} as a subscriber to the '1.subFlow#0.channel#0' channel
2020-08-20 11:10:57,691 INFO o.s.i.c.PublishSubscribeChannel - Channel 'application.1.subFlow#0.channel#0' has 1 subscriber(s).
2020-08-20 11:10:57,691 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#2.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-08-20 11:10:57,691 INFO o.s.i.c.DirectChannel - Channel 'application.1.subFlow#1.channel#0' has 1 subscriber(s).
2020-08-20 11:10:57,691 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#1.org.springframework.integration.config.ConsumerEndpointFactoryBean#1'
2020-08-20 11:10:57,691 INFO o.s.i.e.EventDrivenConsumer - Adding {bridge} as a subscriber to the '1.subFlow#0.channel#0' channel
2020-08-20 11:10:57,692 INFO o.s.i.c.PublishSubscribeChannel - Channel 'application.1.subFlow#0.channel#0' has 2 subscriber(s).
2020-08-20 11:10:57,692 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#1.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-08-20 11:10:57,693 INFO o.s.i.e.EventDrivenConsumer - Adding {message-handler} as a subscriber to the '1.subFlow#0.channel#1' channel
2020-08-20 11:10:57,694 INFO o.s.i.c.DirectChannel - Channel 'application.1.subFlow#0.channel#1' has 1 subscriber(s).
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#0.org.springframework.integration.config.ConsumerEndpointFactoryBean#1'
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - Adding {bridge} as a subscriber to the '1.subFlow#0.channel#0' channel
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - started bean '1.subFlow#0.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - Adding {transformer} as a subscriber to the '1.channel#0' channel
2020-08-20 11:10:57,694 INFO o.s.i.c.DirectChannel - Channel 'application.1.channel#0' has 1 subscriber(s).
2020-08-20 11:10:57,694 INFO o.s.i.e.EventDrivenConsumer - started bean '1.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2020-08-20 11:10:57,700 INFO o.s.i.e.SourcePollingChannelAdapter - started bean '1.org.springframework.integration.config.SourcePollingChannelAdapterFactoryBean#0'
2020-08-20 11:10:58,017 INFO c.j.jsch - Connecting to *.*.*.114 port 22
2020-08-20 11:10:58,341 INFO c.j.jsch - Connection established
2020-08-20 11:10:58,672 INFO c.j.jsch - Remote version string: SSH-2.0-OpenSSH_7.4
2020-08-20 11:10:58,673 INFO c.j.jsch - Local version string: SSH-2.0-JSCH-0.1.54
2020-08-20 11:10:58,674 INFO c.j.jsch - CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
2020-08-20 11:10:58,728 INFO c.j.jsch - CheckKexes: diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
2020-08-20 11:10:58,815 INFO c.j.jsch - CheckSignatures: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
2020-08-20 11:10:58,823 INFO c.j.jsch - SSH_MSG_KEXINIT sent
2020-08-20 11:10:59,036 INFO c.j.jsch - SSH_MSG_KEXINIT received
2020-08-20 11:10:59,038 INFO c.j.jsch - kex: server: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
2020-08-20 11:10:59,038 INFO c.j.jsch - kex: server: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
2020-08-20 11:10:59,039 INFO c.j.jsch - kex: server: aes256-ctr,aes192-ctr,aes128-ctr
2020-08-20 11:10:59,040 INFO c.j.jsch - kex: server: aes256-ctr,aes192-ctr,aes128-ctr
2020-08-20 11:10:59,040 INFO c.j.jsch - kex: server: hmac-sha2-512,hmac-sha2-256
2020-08-20 11:10:59,041 INFO c.j.jsch - kex: server: hmac-sha2-512,hmac-sha2-256
2020-08-20 11:10:59,042 INFO c.j.jsch - kex: server: none,zlib#openssh.com
2020-08-20 11:10:59,042 INFO c.j.jsch - kex: server: none,zlib#openssh.com
2020-08-20 11:10:59,042 INFO c.j.jsch - kex: server:
2020-08-20 11:10:59,043 INFO c.j.jsch - kex: server:
2020-08-20 11:10:59,043 INFO c.j.jsch - kex: client: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
2020-08-20 11:10:59,044 INFO c.j.jsch - kex: client: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
2020-08-20 11:10:59,044 INFO c.j.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-cbc
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: none
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client: none
2020-08-20 11:10:59,045 INFO c.j.jsch - kex: client:
2020-08-20 11:10:59,046 INFO c.j.jsch - kex: client:
2020-08-20 11:10:59,046 INFO c.j.jsch - kex: server->client aes128-ctr hmac-sha2-256 none
2020-08-20 11:10:59,047 INFO c.j.jsch - kex: client->server aes128-ctr hmac-sha2-256 none
2020-08-20 11:10:59,053 INFO c.j.jsch - SSH_MSG_KEX_ECDH_INIT sent
2020-08-20 11:10:59,053 INFO c.j.jsch - expecting SSH_MSG_KEX_ECDH_REPLY
2020-08-20 11:10:59,385 INFO c.j.jsch - ssh_rsa_verify: signature true
2020-08-20 11:10:59,389 INFO o.s.i.s.s.DefaultSftpSessionFactory - The authenticity of host '*.*.*.114' can't be established.
RSA key fingerprint is 1f:3e:c9:5f:37:00:a1:00:ef:50:59:af:42:98:99:e9.
Are you sure you want to continue connecting?
2020-08-20 11:10:59,390 WARN c.j.jsch - Permanently added '*.*.*.114' (RSA) to the list of known hosts.
2020-08-20 11:10:59,390 INFO c.j.jsch - SSH_MSG_NEWKEYS sent
2020-08-20 11:10:59,390 INFO c.j.jsch - SSH_MSG_NEWKEYS received
2020-08-20 11:10:59,395 INFO c.j.jsch - SSH_MSG_SERVICE_REQUEST sent
2020-08-20 11:10:59,718 INFO c.j.jsch - SSH_MSG_SERVICE_ACCEPT received
2020-08-20 11:11:00,047 INFO c.j.jsch - Authentications that can continue: gssapi-with-mic,publickey,keyboard-interactive,password
2020-08-20 11:11:00,047 INFO c.j.jsch - Next authentication method: gssapi-with-mic
2020-08-20 11:11:00,399 INFO c.j.jsch - Authentications that can continue: password
2020-08-20 11:11:00,399 INFO c.j.jsch - Next authentication method: password
2020-08-20 11:11:00,746 INFO c.j.jsch - Authentication succeeded (password).
Total Files: 1
2020-08-20 11:11:27,623 INFO c.t.i.s.s.LastModifiedLsEntryFileListFilter - [OB.xml] old size [null] increased to [0]...
Total Files: 1
2020-08-20 11:11:29,636 INFO c.t.i.s.s.LastModifiedLsEntryFileListFilter - [OB.xml] old size [0] increased to [38709]...
Total Files: 1
2020-08-20 11:11:34,256 INFO c.j.jsch - Connecting to *.*.*.115 port 22
2020-08-20 11:11:55,282 INFO c.t.i.c.f.a.LoggingAspect - log message Integration Name=CXML invoice DK Service Name=FileTransferService Source=External Source Interface=/home/umair/accruals/14minute/ Target=NA Target Interface=/home/umair/accruals/15minute/ Content ID=OB.xml Message ID=N/A Category=SUCCESS Timestamp=2020-08-20T11:11:55.281 Message=file OB.xml transferred
as per Artem Bilan suggestion, i have created a demo app and uploaded to git repo here
server configs are here
integration flow is defined here
README file
I found your problem:
#Bean
public Advice deleteFileAdvice() {
ExpressionEvaluatingRequestHandlerAdvice advice = new ExpressionEvaluatingRequestHandlerAdvice();
advice.setOnSuccessExpressionString(AppConstants.SUCCESS_EXPRESSION);
advice.setTrapException(true);
return advice;
}
See its JavaDocs:
/**
* If true, any exception will be caught and null returned.
* Default false.
* #param trapException true to trap Exceptions.
*/
public void setTrapException(boolean trapException) {
So, any exceptions from the message handler, including an SFTP conneciton failure is going to be swallowed. Therefore a request message is really moved to the next subscriber in your publishSubcribeChannel configuration.
I tried to download files using the following code:
<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
auto-startup="true" channel="receiveChannel" session-factory="sftpSessionFactory"
local-directory="file:${directory.files.local}" remote-directory="${directory.files.remote}"
auto-create-local-directory="true" delete-remote-files="true"
filename-pattern="*.txt">
<int:poller fixed-delay="${sftp.interval.request}"
max-messages-per-poll="-1" error-channel="sftp.in.error.channel" />
</int-sftp:inbound-channel-adapter>
<bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${sftp.host}" />
<property name="port" value="${sftp.port}" />
<property name="user" value="${user}" />
<property name="password" value="${password}" />
<property name="allowUnknownKeys" value="true" />
</bean>
I'm sure that the user is authorized because i tried it with shell:
sftp user#x.x.x.x
then i write the password and the download succeed with "get".
but i can't download files, the error is:
DEBUG LOG:
jsch:52 - Authentications that can continue: gssapi-with-mic,publickey,keyboard-interactive,password
2016-02-02 07:54:04 INFO jsch:52 - Next authentication method: gssapi-with-mic
2016-02-02 07:54:04 INFO jsch:52 - Authentications that can continue: publickey,keyboard-interactive,password
2016-02-02 07:54:04 INFO jsch:52 - Next authentication method: publickey
2016-02-02 07:54:04 INFO jsch:52 - Authentications that can continue: password
2016-02-02 07:54:04 INFO jsch:52 - Next authentication method: password
2016-02-02 07:54:04 INFO jsch:52 - Authentication succeeded (password).
2016-02-02 07:54:05 DEBUG SimplePool:190 - Obtained new org.springframework.integration.sftp.session.SftpSession#39c9c99a.
2016-02-02 07:54:05 DEBUG CachingSessionFactory:187 - Releasing Session org.springframework.integration.sftp.session.SftpSession#39c9c99a back to the pool.
2016-02-02 07:54:05 INFO jsch:52 - Disconnecting from x.x.x.x port 22
I would enable DEBUG logging for jsch as well as org.springframework.integration.
That last message is coming from this code...
#Override
public boolean promptYesNo(String message) {
logger.info(message); // <<<<<<<<< INFO message in your log line 538
if (hasDelegate()) {
return getDelegate().promptYesNo(message);
}
else {
if (logger.isDebugEnabled()) {
logger.debug("No UserInfo provided - " + message + ", returning:"
+ DefaultSftpSessionFactory.this.allowUnknownKeys);
}
return DefaultSftpSessionFactory.this.allowUnknownKeys;
}
}
Since you have not provided a delegate UserInfo (according to your configuration in the question), it should return true (because you have allowUnknownKeys set to true).
If you can't figure it out; edit your question with the appropriate part of the log.
EDIT
You removed the most useful part of the log you posted on your first edit:
2016-02-01 18:28:27 DEBUG DefaultSftpSessionFactory:544 - No UserInfo provided - The authenticity of host '192.168.21.36' can't be established.
RSA key fingerprint is 98:1d:7e:73:77:97:f6:af:f9:2a:fc:2b:21:8e:8e:bf.
Are you sure you want to continue connecting?, returning:false
"Returning:false" means that the allowUnknownKeys property is false, not true as you show in your configuration. Perhaps you have another session factory bean that's overriding this one?
How to integrate a Pollable channel in the Integration flow. As my flow is using a NullChannel instead of Pollable Channel.
Can some one kindly provide any pointer on this.
This is my Integration flow.
#Bean
public IntegrationFlow sftpInboundFlow() {
System.out.println("enter sftpInboundFlow....."
+ sftpSessionFactory.getSession());
System.out.println("the channel is " + sftpInboundResultChannel);
return IntegrationFlows
.from(Sftp.inboundAdapter(this.sftpSessionFactory)
.preserveTimestamp(true).remoteDirectory(remDir)
.regexFilter(".*\\.txt$")
.localFilenameExpression("#this.toUpperCase()")
.localDirectory(new File(localDir))
.remoteFileSeparator("/"),
new Consumer<SourcePollingChannelAdapterSpec>() {
#Override
public void accept(SourcePollingChannelAdapterSpec e) {
e.id("sftpInboundAdapter")
.autoStartup(true)
.poller(Pollers.fixedRate(1000)
.maxMessagesPerPoll(1));
}
})
.channel(MessageChannels.queue("sftpInboundResultChannel"))
.get();
}
And I want to use this..
#Autowired
private PollableChannel sftpInboundResultChannel;
in the .channel() method.
And the logs are...
[2015-07-22 18:48:05.497] - 2796 INFO [main] --- com.jcraft.jsch: SSH_MSG_NEWKEYS sent
[2015-07-22 18:48:05.497] - 2796 INFO [main] --- com.jcraft.jsch: SSH_MSG_NEWKEYS received
[2015-07-22 18:48:05.500] - 2796 INFO [main] --- com.jcraft.jsch: SSH_MSG_SERVICE_REQUEST sent
[2015-07-22 18:48:05.501] - 2796 INFO [main] --- com.jcraft.jsch: SSH_MSG_SERVICE_ACCEPT received
[2015-07-22 18:48:15.517] - 2796 INFO [main] --- com.jcraft.jsch: Authentications that can continue: gssapi-with-mic,publickey,keyboard-interactive,password
[2015-07-22 18:48:15.517] - 2796 INFO [main] --- com.jcraft.jsch: Next authentication method: gssapi-with-mic
[2015-07-22 18:48:15.522] - 2796 INFO [main] --- com.jcraft.jsch: Authentications that can continue: publickey,keyboard-interactive,password
[2015-07-22 18:48:15.522] - 2796 INFO [main] --- com.jcraft.jsch: Next authentication method: publickey
[2015-07-22 18:48:15.523] - 2796 INFO [main] --- com.jcraft.jsch: Authentications that can continue: password
[2015-07-22 18:48:15.523] - 2796 INFO [main] --- com.jcraft.jsch: Next authentication method: password
[2015-07-22 18:48:15.537] - 2796 INFO [main] --- com.jcraft.jsch: Authentication succeeded (password).
enter sftpSessionFactory.....org.springframework.integration.sftp.session.SftpSession#1ffcd26
enter sftpInboundFlow.....org.springframework.integration.sftp.session.SftpSession#13a462c
the channel is org.springframework.integration.channel.NullChannel#d532ee
[2015-07-22 18:48:15.827] - 2796 INFO [main] --- org.springframework.context.support.DefaultLifecycleProcessor: Starting beans in phase -2147483648
[2015-07-22 18:48:15.828] - 2796 INFO [main] --- org.springframework.context.support.DefaultLifecycleProcessor: Starting beans in phase 0
[2015-07-22 18:48:15.828] - 2796 INFO [main] --- org.springframework.integration.endpoint.EventDrivenConsumer: Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
[2015-07-22 18:48:15.828] - 2796 INFO [main] --- org.springframework.integration.channel.PublishSubscribeChannel: Channel 'application.errorChannel' has 1 subscriber(s).
[2015-07-22 18:48:15.829] - 2796 INFO [main] --- org.springframework.integration.endpoint.EventDrivenConsumer: started _org.springframework.integration.errorLogger
[2015-07-22 18:48:15.829] - 2796 INFO [main] --- org.springframework.context.support.DefaultLifecycleProcessor: Starting beans in phase 1073741823
[2015-07-22 18:48:15.832] - 2796 INFO [main] --- org.springframework.integration.endpoint.SourcePollingChannelAdapter: started sftpInboundAdapter
[2015-07-22 18:48:15.834] - 2796 INFO [main] --- src.MainSftpInBoundDsl: Started MainSftpInBoundDsl in 11.019 seconds (JVM running for 11.182)
enter main.....
enter main.....
[2015-07-22 18:48:16.040] - 2796 INFO [task-scheduler-1] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\AMARJEET.TXT, headers={timestamp=1437571096040, id=ddc20f6d-26ee-7eef-f59e-b5207dcdc8c7}]]
[2015-07-22 18:48:16.832] - 2796 INFO [task-scheduler-1] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\INFO.TXT, headers={timestamp=1437571096832, id=361b55d5-884e-9001-d513-ddc81926bf85}]]
[2015-07-22 18:48:17.832] - 2796 INFO [task-scheduler-2] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\KKKK.TXT, headers={timestamp=1437571097832, id=49deb17c-50e3-4315-dfa2-cc443a35bd1c}]]
[2015-07-22 18:48:18.832] - 2796 INFO [task-scheduler-1] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\LA.TXT, headers={timestamp=1437571098832, id=88def298-ab86-1d63-039e-c81c70ae6819}]]
[2015-07-22 18:48:19.832] - 2796 INFO [task-scheduler-3] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\POC.TXT, headers={timestamp=1437571099832, id=d78b36eb-10f5-ce83-ba4c-cd24a806480b}]]
[2015-07-22 18:48:20.832] - 2796 INFO [task-scheduler-2] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\PPPP.TXT, headers={timestamp=1437571100832, id=4e727ebb-b4de-be19-6a92-afb3be24576f}]]
[2015-07-22 18:48:21.832] - 2796 INFO [task-scheduler-4] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\REFERENCE.TXT, headers={timestamp=1437571101832, id=3d611371-431c-6b1e-0e17-377f5607de96}]]
[2015-07-22 18:48:22.831] - 2796 INFO [task-scheduler-1] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\RRRRR.TXT, headers={timestamp=1437571102831, id=707e566c-ccfa-6c32-16d6-1f27ec73abaf}]]
[2015-07-22 18:48:23.832] - 2796 INFO [task-scheduler-5] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\SFTP.TXT, headers={timestamp=1437571103832, id=b3c979ff-fcd2-e1ff-8b37-6b8f2803637a}]]
[2015-07-22 18:48:24.832] - 2796 INFO [task-scheduler-3] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\TEST.TXT, headers={timestamp=1437571104832, id=98d43e62-8e5f-69bc-7617-4706761eaf86}]]
[2015-07-22 18:48:25.832] - 2796 INFO [task-scheduler-3] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\XSS.TXT, headers={timestamp=1437571105832, id=77b28259-0287-0ef5-825f-5345d6c21c21}]]
[2015-07-22 18:48:26.832] - 2796 INFO [task-scheduler-3] --- org.springframework.integration.file.FileReadingMessageSource: Created message: [GenericMessage [payload=D:\local_copy\ZZZZ.TXT, headers={timestamp=1437571106832, id=b5644a62-8c8c-86d2-e016-75fff01c70b4}]]
Just use the autowired field directly in the flow...
.channel(this.sftpInboundResultChannel)
EDIT:
It looks like MessageChannels.queue("sftpInboundResultChannel") is not registering the channel as a bean so Spring is autowiring NullChannel into that field (because it's the only PollableChannel in the context).
We need to take a look at that; in the meantime, add
#Bean
public PollableChannel sftpInboundResultChannel() {
return new QueueChannel();
}
I'm trying to write an RDD[String] to HDFS (in spark-shell) using:
output.saveAsTextFile("hdfs://localhost:9000/datasets/result")
However, it just hangs - and the job doesn't even appear in the web UI. I have to kill the SparkSubmit process.
I'm reading data successful from HDFS using:
val input = sc.textFile("hdfs://localhost:9000/datasets/data.csv")
I can output.collect successfully, and writing to local files works as expected.
I'm using Spark 1.4 and Hadoop 2.6. Everything is running on a local machine.
Any ideas?
Comments made me realize that, I should switch on DEBUG level logs. Initial log extract below. There's something about the connection being closed. But, I read the data from HDFS at the start of the short script, so I'm confused.
Solution (kind of)
Must be something to do with local routing. I replaced localhost with 127.0.0.1 and it's working now.
Debug logs
15/06/30 17:04:42 DEBUG ClosureCleaner: +++ Cleaning closure <function1> (org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30}) +++
15/06/30 17:04:42 DEBUG ClosureCleaner: + declared fields: 1
15/06/30 17:04:42 DEBUG ClosureCleaner: public static final long org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30.serialVersionUID
15/06/30 17:04:42 DEBUG ClosureCleaner: + declared methods: 2
15/06/30 17:04:42 DEBUG ClosureCleaner: public final java.lang.Object org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30.apply(java.lang.Object)
15/06/30 17:04:42 DEBUG ClosureCleaner: public final scala.collection.Iterator org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30.apply(scala.collection.Iterator)
15/06/30 17:04:42 DEBUG ClosureCleaner: + inner classes: 1
15/06/30 17:04:42 DEBUG ClosureCleaner: org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30$$anonfun$apply$49
15/06/30 17:04:42 DEBUG ClosureCleaner: + outer classes: 0
15/06/30 17:04:42 DEBUG ClosureCleaner: + outer objects: 0
15/06/30 17:04:42 DEBUG ClosureCleaner: + populating accessed fields because this is the starting closure
15/06/30 17:04:42 DEBUG ClosureCleaner: + fields accessed by starting closure: 0
15/06/30 17:04:42 DEBUG ClosureCleaner: + there are no enclosing objects!
15/06/30 17:04:42 DEBUG ClosureCleaner: +++ closure <function1> (org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1$$anonfun$30) is now cleaned +++
15/06/30 17:04:42 DEBUG BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
15/06/30 17:04:42 DEBUG BlockReaderLocal: dfs.client.read.shortcircuit = false
15/06/30 17:04:42 DEBUG BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
15/06/30 17:04:42 DEBUG BlockReaderLocal: dfs.domain.socket.path =
15/06/30 17:04:42 DEBUG DFSClient: No KeyProvider found.
15/06/30 17:04:43 DEBUG Client: IPC Client (832019786) connection to localhost/127.0.0.1:9000 from user: closed
15/06/30 17:04:43 DEBUG Client: IPC Client (832019786) connection to localhost/127.0.0.1:9000 from user: stopped, remaining connections 0
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#4eb1e256,BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$L]
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#4eb1e256,BlockManagerId(driver, localhost, 51349)),true)
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.290877 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#4eb1e256,BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$L]
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$M]
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true)
15/06/30 17:04:48 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.257991 ms) AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$M]
15/06/30 17:04:57 DEBUG RetryUtils: multipleLinearRandomRetry = null
15/06/30 17:04:57 DEBUG Client: getting client out of cache: org.apache.hadoop.ipc.Client#7d1fc150
15/06/30 17:04:57 DEBUG DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
15/06/30 17:04:57 DEBUG PairRDDFunctions: Saving as hadoop file of type (NullWritable, Text)
15/06/30 17:04:57 DEBUG Client: The ping interval is 60000 ms.
15/06/30 17:04:57 DEBUG Client: Connecting to localhost/81.200.64.50:9000
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7aba4180,BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$N]
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7aba4180,BlockManagerId(driver, localhost, 51349)),true)
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.277965 ms) AkkaMessage(Heartbeat(driver,[Lscala.Tuple2;#7aba4180,BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$N]
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] received message AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$O]
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: Received RPC message: AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true)
15/06/30 17:04:58 DEBUG AkkaRpcEnv$$anonfun$actorRef$lzycompute$1$1$$anon$1: [actor] handled message (0.258037 ms) AkkaMessage(BlockManagerHeartbeat(BlockManagerId(driver, localhost, 51349)),true) from Actor[akka://sparkDriver/temp/$O]
My testing stack consists of the latest version of Selenium Server (2.33.0, aka selenium-server-standalone-2.33.0.jar), Mocha, Node.js, and PhantomJS.
My question regards the following code:
var webdriver = require('../../../lib/selenium/node_modules/selenium-webdriver/'),
driver = new webdriver.Builder().
withCapabilities({'browserName': 'phantomjs'}).
build();
driver.manage().timeouts().implicitlyWait(15000);
describe('Wordpress', function() {
it('should be able to log in', function(done) {
driver.get('http://#### REDACTED ####/wp-login.php');
driver.findElement(webdriver.By.css('#user_login')).sendKeys('#### REDACTED ####');
driver.findElement(webdriver.By.css('#user_pass')).sendKeys('#### REDACTED ####');
driver.findElement(webdriver.By.css('#wp-submit')).click();
// #wpwrap is an element on the Wordpress dashboard that is displayed once
// the user is logged in. By testing for its presence, we can determine
// if the login attempt succeeded.
driver.findElement(webdriver.By.css('#wpwrap')).then(function(v) {
done();
});
});
});
On my local system, OS X, the test runs fine consistently. However, once the test is uploaded to our CentOS server (where we hope to do Continuous Integration testing), the test behaves extremely strangely.
After Selenium Server is started, the test runs successfully once. From that point on, the test only succeeds every one out of ten times or so. Restarting Selenium Server guarantees that the test will run successfully. In fact, if Selenium Server is restarted every time the test is run, the test will succeed every time.
How can I get this test to succeed without restarting Selenium Server every time?
Thank you so much for your help! :)
UPDATE: In addition to the error log below, I'm also occasionally getting the following error:
Exception in thread "Thread-21" java.lang.OutOfMemoryError: unable to create new native thread
Details on the error messages follow below:
A successful test yields the following output from Mocha:
[s5rich#host ~]$ mocha test/selenium/acceptance/simple.js
Wordpress
✓ should be able to log in (2604ms)
1 passing (3 seconds)
A successful test also yields the following output from Selenium Server:
23:21:50.517 INFO - Executing: [new session: {browserName=phantomjs}] at URL: /session)
23:21:50.527 INFO - Creating a new session for Capabilities [{browserName=phantomjs}]
23:21:50.547 INFO - executable: /usr/local/bin/phantomjs
23:21:50.547 INFO - port: 26515
23:21:50.547 INFO - arguments: [--webdriver=26515, --webdriver-logfile=/home/s5rich/phantomjsdriver.log]
23:21:50.547 INFO - environment: {}
PhantomJS is launching GhostDriver...
[INFO - 2013-07-24T05:21:50.923Z] GhostDriver - Main - running on port 26515
[INFO - 2013-07-24T05:21:51.435Z] Session [f235d040-f420-11e2-8d90-f50327bc3449] - CONSTRUCTOR - Desired Capabilities: {"browserName":"phantomjs"}
[INFO - 2013-07-24T05:21:51.435Z] Session [f235d040-f420-11e2-8d90-f50327bc3449] - CONSTRUCTOR - Negotiated Capabilities: {"browserName":"phantomjs","version":"1.9.1","driverName":"ghostdriver","driverVersion":"1.0.3","platform":"linux-unknown-32bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
[INFO - 2013-07-24T05:21:51.435Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: f235d040-f420-11e2-8d90-f50327bc3449
23:21:51.495 INFO - Done: /session
23:21:51.504 INFO - Executing: org.openqa.selenium.remote.server.handler.GetSessionCapabilities#46b78d at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2)
23:21:51.505 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2
23:21:51.520 INFO - Executing: [get: http://#### REDACTED ####/wp-login.php] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/url)
23:21:51.821 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/url
23:21:51.827 INFO - Executing: [find element: By.selector: #user_login] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element)
23:21:51.874 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element
23:21:51.883 INFO - Executing: [send keys: 0 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#20788bf8, [#### REDACTED ####]] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/0/value)
23:21:51.939 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/0/value
23:21:51.948 INFO - Executing: [find element: By.selector: #user_pass] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element)
23:21:51.965 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element
23:21:52.001 INFO - Executing: [send keys: 1 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#20788bf9, [#### REDACTED ####]] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/1/value)
23:21:52.065 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/1/value
23:21:52.074 INFO - Executing: [find element: By.selector: #wp-submit] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element)
23:21:52.099 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element
23:21:52.106 INFO - Executing: [click: 2 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#20788bfa] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/2/click)
23:21:52.842 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element/2/click
23:21:52.850 INFO - Executing: [find element: By.selector: #wpwrap] at URL: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element)
23:21:52.871 INFO - Done: /session/8750a6b4-ec7d-4313-a86d-04ac344b74f2/element
A failed test yields the following output from Mocha:
[s5rich#host ~]$ mocha test/selenium/acceptance/simple.js
Wordpress
1) should be able to log in
0 passing (2 seconds)
1 failing
1) Wordpress should be able to log in:
Uncaught UnknownError: Error Message => 'Unable to find element with css selector '#wpwrap''
caused by Request => {"headers":{"Accept":"application/json, image/png","Connection":"Keep-Alive","Content-Length":"42","Content-Type":"application/json; charset=utf-8","Host":"localhost:2897"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#wpwrap\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/77a0a110-f421-11e2-a6fd-61cd002d7d02/element"}
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Driver info: driver.version: unknown
at new bot.Error (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/atoms/error.js:108:18)
at Object.bot.response.checkResponse (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/atoms/response.js:106:9)
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/webdriver.js:262:20
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/goog/base.js:1112:15
at webdriver.promise.ControlFlow.runInNewFrame_ (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:1431:20)
at notify (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:315:12)
at notifyAll (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:284:7)
at fulfill (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:389:7)
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:1298:10
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/goog/base.js:1112:15
at webdriver.promise.ControlFlow.runInNewFrame_ (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:1431:20)
at notify (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:315:12)
at notifyAll (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:284:7)
at fulfill (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:389:7)
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/goog/base.js:1112:15
at webdriver.promise.ControlFlow.runInNewFrame_ (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:1431:20)
at notify (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:315:12)
at notifyAll (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:284:7)
at fulfill (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:389:7)
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/promise.js:600:51
at /home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/http/http.js:96:5
at IncomingMessage.<anonymous> (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/http/index.js:113:7)
at IncomingMessage.EventEmitter.emit (events.js:117:20)
at _stream_readable.js:910:16
at process._tickCallback (node.js:415:13)
==== async task ====
WebDriver.findElement(By.cssSelector("#wpwrap"))
at webdriver.WebDriver.schedule (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/webdriver.js:246:15)
at webdriver.WebDriver.findElement (/home/s5rich/lib/selenium/node_modules_osx/selenium-webdriver/lib/webdriver/webdriver.js:685:17)
at Context.<anonymous> (/home/s5rich/test/selenium/acceptance/simple.js:12:10)
at Test.Runnable.run (/usr/local/lib/node_modules/mocha/lib/runnable.js:194:15)
at Runner.runTest (/usr/local/lib/node_modules/mocha/lib/runner.js:355:10)
at /usr/local/lib/node_modules/mocha/lib/runner.js:401:12
at next (/usr/local/lib/node_modules/mocha/lib/runner.js:281:14)
at /usr/local/lib/node_modules/mocha/lib/runner.js:290:7
at next (/usr/local/lib/node_modules/mocha/lib/runner.js:234:23)
at Object._onImmediate (/usr/local/lib/node_modules/mocha/lib/runner.js:258:5)
at processImmediate [as _immediateCallback] (timers.js:330:15)
A failed test also yields the following output from Selenium Server:
23:25:34.742 INFO - Executing: [new session: {browserName=phantomjs}] at URL: /session)
23:25:34.743 INFO - Creating a new session for Capabilities [{browserName=phantomjs}]
23:25:34.744 INFO - executable: /usr/local/bin/phantomjs
23:25:34.744 INFO - port: 2897
23:25:34.744 INFO - arguments: [--webdriver=2897, --webdriver-logfile=/home/s5rich/phantomjsdriver.log]
23:25:34.744 INFO - environment: {}
PhantomJS is launching GhostDriver...
[INFO - 2013-07-24T05:25:34.879Z] GhostDriver - Main - running on port 2897
[INFO - 2013-07-24T05:25:35.270Z] Session [77a0a110-f421-11e2-a6fd-61cd002d7d02] - CONSTRUCTOR - Desired Capabilities: {"browserName":"phantomjs"}
[INFO - 2013-07-24T05:25:35.270Z] Session [77a0a110-f421-11e2-a6fd-61cd002d7d02] - CONSTRUCTOR - Negotiated Capabilities: {"browserName":"phantomjs","version":"1.9.1","driverName":"ghostdriver","driverVersion":"1.0.3","platform":"linux-unknown-32bit","javascriptEnabled":true,"takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
[INFO - 2013-07-24T05:25:35.270Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: 77a0a110-f421-11e2-a6fd-61cd002d7d02
23:25:35.275 INFO - Done: /session
23:25:35.283 INFO - Executing: org.openqa.selenium.remote.server.handler.GetSessionCapabilities#13b4ce4 at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee)
23:25:35.284 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee
23:25:35.297 INFO - Executing: [get: http://#### REDACTED ####/wp-login.php] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/url)
23:25:35.592 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/url
23:25:35.597 INFO - Executing: [find element: By.selector: #user_login] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element)
23:25:35.619 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element
23:25:35.631 INFO - Executing: [send keys: 0 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#240035bc, [#### REDACTED ####]] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/0/value)
23:25:35.683 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/0/value
23:25:35.695 INFO - Executing: [find element: By.selector: #user_pass] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element)
23:25:35.712 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element
23:25:35.723 INFO - Executing: [send keys: 1 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#240035bd, [#### REDACTED ####]] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/1/value)
23:25:35.783 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/1/value
23:25:35.800 INFO - Executing: [find element: By.selector: #wp-submit] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element)
23:25:35.822 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element
23:25:35.832 INFO - Executing: [click: 2 org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWebElement#240035be] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/2/click)
23:25:36.105 INFO - Done: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element/2/click
23:25:36.121 INFO - Executing: [find element: By.selector: #wpwrap] at URL: /session/b62d0c67-2000-439c-a0bd-f2c100350dee/element)
e = java.awt.HeadlessException:
No X11 DISPLAY variable was set, but this program performed an operation which requires it.
23:25:36.285 WARN - Exception thrown
org.openqa.selenium.NoSuchElementException: Error Message => 'Unable to find element with css selector '#wpwrap''
caused by Request => {"headers":{"Accept":"application/json, image/png","Connection":"Keep-Alive","Content-Length":"42","Content-Type":"application/json; charset=utf-8","Host":"localhost:2897"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#wpwrap\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/77a0a110-f421-11e2-a6fd-61cd002d7d02/element"}
Command duration or timeout: 149 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Session ID: 77a0a110-f421-11e2-a6fd-61cd002d7d02
Driver info: org.openqa.selenium.phantomjs.PhantomJSDriver
Capabilities [{platform=LINUX, acceptSslCerts=false, javascriptEnabled=true, browserName=phantomjs, rotatable=false, driverVersion=1.0.3, locationContextEnabled=false, version=1.9.1, databaseEnabled=false, cssSelectorsEnabled=true, handlesAlerts=false, browserConnectionEnabled=false, proxy={proxyType=direct}, webStorageEnabled=false, nativeEvents=true, driverName=ghostdriver, applicationCacheEnabled=false, takesScreenshot=true}]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:191)
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:145)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:554)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:307)
at org.openqa.selenium.remote.RemoteWebDriver.findElementByCssSelector(RemoteWebDriver.java:396)
at org.openqa.selenium.By$ByCssSelector.findElement(By.java:407)
at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:299)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.openqa.selenium.support.events.EventFiringWebDriver$2.invoke(EventFiringWebDriver.java:101)
at $Proxy1.findElement(Unknown Source)
at org.openqa.selenium.support.events.EventFiringWebDriver.findElement(EventFiringWebDriver.java:180)
at org.openqa.selenium.remote.server.handler.FindElement.call(FindElement.java:47)
at org.openqa.selenium.remote.server.handler.FindElement.call(FindElement.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at org.openqa.selenium.remote.server.DefaultSession$1.run(DefaultSession.java:169)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.openqa.selenium.remote.ScreenshotException: Screen shot has been taken
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Driver info: driver.version: EventFiringWebDriver
at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:125)
... 20 more
Caused by: org.openqa.selenium.remote.ErrorHandler$UnknownServerException: Error Message => 'Unable to find element with css selector '#wpwrap''
caused by Request => {"headers":{"Accept":"application/json, image/png","Connection":"Keep-Alive","Content-Length":"42","Content-Type":"application/json; charset=utf-8","Host":"localhost:2897"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#wpwrap\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/77a0a110-f421-11e2-a6fd-61cd002d7d02/element"}
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Driver info: driver.version: unknown
23:25:36.293 WARN - Exception: Error Message => 'Unable to find element with css selector '#wpwrap''
caused by Request => {"headers":{"Accept":"application/json, image/png","Connection":"Keep-Alive","Content-Length":"42","Content-Type":"application/json; charset=utf-8","Host":"localhost:2897"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#wpwrap\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/77a0a110-f421-11e2-a6fd-61cd002d7d02/element"}
Build info: version: '2.33.0', revision: '4e90c97', time: '2013-05-22 15:32:38'
System info: os.name: 'Linux', os.arch: 'i386', os.version: '2.6.32-042stab076.8', java.version: '1.7.0'
Driver info: driver.version: unknown
From what I see, it is a simple problem to solve. You have a unhandled exception that is being thrown when there is a failure: NoSuchElementException . I don't think you need to restart your grid node every time (assuming you are running your grid hub separate from your grid node instances). It may be that successive runs of the browser from your Grid Node use just enough more memory to cause a minor delay in the page, breaking your script. All you need to do is handle the NoSuchElementException gracefully by wrapping it in a retry try loop. You can also effectively do similar using FluentWait with the .ignoring method.