I tried to download files using the following code:
<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
auto-startup="true" channel="receiveChannel" session-factory="sftpSessionFactory"
local-directory="file:${directory.files.local}" remote-directory="${directory.files.remote}"
auto-create-local-directory="true" delete-remote-files="true"
filename-pattern="*.txt">
<int:poller fixed-delay="${sftp.interval.request}"
max-messages-per-poll="-1" error-channel="sftp.in.error.channel" />
</int-sftp:inbound-channel-adapter>
<bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${sftp.host}" />
<property name="port" value="${sftp.port}" />
<property name="user" value="${user}" />
<property name="password" value="${password}" />
<property name="allowUnknownKeys" value="true" />
</bean>
I'm sure that the user is authorized because i tried it with shell:
sftp user#x.x.x.x
then i write the password and the download succeed with "get".
but i can't download files, the error is:
DEBUG LOG:
jsch:52 - Authentications that can continue: gssapi-with-mic,publickey,keyboard-interactive,password
2016-02-02 07:54:04 INFO jsch:52 - Next authentication method: gssapi-with-mic
2016-02-02 07:54:04 INFO jsch:52 - Authentications that can continue: publickey,keyboard-interactive,password
2016-02-02 07:54:04 INFO jsch:52 - Next authentication method: publickey
2016-02-02 07:54:04 INFO jsch:52 - Authentications that can continue: password
2016-02-02 07:54:04 INFO jsch:52 - Next authentication method: password
2016-02-02 07:54:04 INFO jsch:52 - Authentication succeeded (password).
2016-02-02 07:54:05 DEBUG SimplePool:190 - Obtained new org.springframework.integration.sftp.session.SftpSession#39c9c99a.
2016-02-02 07:54:05 DEBUG CachingSessionFactory:187 - Releasing Session org.springframework.integration.sftp.session.SftpSession#39c9c99a back to the pool.
2016-02-02 07:54:05 INFO jsch:52 - Disconnecting from x.x.x.x port 22
I would enable DEBUG logging for jsch as well as org.springframework.integration.
That last message is coming from this code...
#Override
public boolean promptYesNo(String message) {
logger.info(message); // <<<<<<<<< INFO message in your log line 538
if (hasDelegate()) {
return getDelegate().promptYesNo(message);
}
else {
if (logger.isDebugEnabled()) {
logger.debug("No UserInfo provided - " + message + ", returning:"
+ DefaultSftpSessionFactory.this.allowUnknownKeys);
}
return DefaultSftpSessionFactory.this.allowUnknownKeys;
}
}
Since you have not provided a delegate UserInfo (according to your configuration in the question), it should return true (because you have allowUnknownKeys set to true).
If you can't figure it out; edit your question with the appropriate part of the log.
EDIT
You removed the most useful part of the log you posted on your first edit:
2016-02-01 18:28:27 DEBUG DefaultSftpSessionFactory:544 - No UserInfo provided - The authenticity of host '192.168.21.36' can't be established.
RSA key fingerprint is 98:1d:7e:73:77:97:f6:af:f9:2a:fc:2b:21:8e:8e:bf.
Are you sure you want to continue connecting?, returning:false
"Returning:false" means that the allowUnknownKeys property is false, not true as you show in your configuration. Perhaps you have another session factory bean that's overriding this one?
Related
Spring Integration and Spring Integration AMQP.
I have the following code:
<bean id="allHeadersMapper" class="org.springframework.integration.amqp.support.DefaultAmqpHeaderMapper">
<property name="requestHeaderNames" value="*" />
<property name="replyHeaderNames" value="*" />
</bean>
<int-amqp:inbound-channel-adapter channel="ResponseChannel" queue-names="OutputQueue" connection-factory="amqpConnectionFactory" error-channel="274ErrorChannel" header-mapper="allHeadersMapper" />
I am converting the EDI message to JSON and Placing in "OutputQueue" . While I try to Read the mesage from OutputQueue and Place in Channel, I try to map all standard and non Standard Header.
Eg.
[payload={abcPayload}, headers={amqp_consumerQueue=OutputQueue, amqp_receivedExchange=XExchange, VERSION_ID=1.0, amqp_contentEncoding=UTF-8, contentType=text/plain, amqp_redelivered=true, STATUS=ABC, timestamp=1432235031343, id=793599d5-6236-c6a1-717e-4f85f9cd9f7b, history=JSONChannel,ResponseLogging,FilterChannel1, amqp_receivedRoutingKey=OutputQueueKey, MESSAGE_ID=MESSAGE_ID, amqp_deliveryMode=NON_PERSISTENT, RECORD_ID=RECORD_ID, amqp_consumerTag=amq.ctag-7jyApLDNT5s_HbxOU8gkPQ, FLOW_NAME=BCD, amqp_deliveryTag=1}]
I am getting the Exception:
Incorrect type specified for header 'history'. Expected [class org.springframework.integration.history.MessageHistory] but actual type is [class java.util.ArrayList]
I am trying to sftp file with Spring integration using a maven web projecy
Need a polling facility. If I am starting the SftpInbound.java, the polling is working. Need to have the polling at server start up.
The content of java file and configuration
SftpInbound.java
package com.myproj.integration.bsy.sftp;
import java.io.File;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import org.springframework.integration.endpoint.SourcePollingChannelAdapter;
import org.springframework.integration.file.remote.RemoteFileTemplate;
import org.springframework.integration.file.remote.session.CachingSessionFactory;
import org.springframework.integration.file.remote.session.SessionFactory;
import org.springframework.messaging.Message;
import org.springframework.messaging.PollableChannel;
import org.springframework.scheduling.annotation.Scheduled;
import com.myproj.integration.bsy.sftp.*;
import com.jcraft.jsch.ChannelSftp.LsEntry;
public class SftpInboundReceive {
#Scheduled(fixedRate=5000)
public void inboundSftpPoll(){
ConfigurableApplicationContext context =
new ClassPathXmlApplicationContext("/META-INF/spring/integration/sftp/SftpInboundReceive-context.xml", this.getClass());
RemoteFileTemplate<LsEntry> template = null;
String file1 = "a.txt";
String file2 = "b.txt";
String file3 = "c.bar";
new File("local-dir", file1).delete();
new File("local-dir", file2).delete();
try {
PollableChannel localFileChannel = context.getBean("receiveChannel", PollableChannel.class);
#SuppressWarnings("unchecked")
SessionFactory<LsEntry> sessionFactory = context.getBean(CachingSessionFactory.class);
template = new RemoteFileTemplate<LsEntry>(sessionFactory);
System.out.println("here 1" +template);
SourcePollingChannelAdapter adapter = context.getBean("sftpInbondAdapter",SourcePollingChannelAdapter.class);
adapter.start();
Message<?> received = localFileChannel.receive();
System.out.println("Received first file message 1: " + received);
received = localFileChannel.receive();
System.out.println("Received second file message: " + received);
received = localFileChannel.receive(1000);
System.out.println("Third file was received as expected" +received);
}catch(Exception e){
e.printStackTrace();
}
finally {
SftpUtils.cleanUp(template, file1, file2, file3);
//context.close();
}
}
public static void main(String args[])
{
SftpInboundReceive oInboundReceiveSample = new SftpInboundReceive();
oInboundReceiveSample.inboundSftpPoll();
}
}
The xml file SftpInboundReceive-context.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-sftp="http://www.springframework.org/schema/integration/sftp"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.1.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-4.1.xsd
http://www.springframework.org/schema/integration/sftp http://www.springframework.org/schema/integration/sftp/spring-integration-sftp-4.1.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-4.1.xsd">
<!-- <import resource="SftpSampleCommon.xml"/> -->
<context:property-placeholder order="1"
location="classpath:/sftpuser.properties" ignore-unresolvable="true"/>
<bean id="sftpSessionFactory"
class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<constructor-arg ref="defaultSftpSessionFactory" />
</bean>
<!-- host=xxx.xx.128.143 port=22 username=xxxuser passphrase= private.keyfile=classpath:META-INF/keys/sftp_rsa -->
<bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${sftp.host}" />
<property name="port" value="${sftp.port}" />
<property name="user" value="${sftp.username}" />
<property name="privateKey" value="${private.keyfile}" />
<property name="privateKeyPassphrase" value="${passphrase}" />
</bean>
<!-- username & password from property file... tested <bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${sftp.host}"/> <property name="port" value="${sftp.port}"/>
<property name="user" value="${sftp.username}"/> <property name="password"
value="${sftp.password}"/> </bean> -->
<!-- hardcoded, username & password... tested <bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="xxx.xx.128.143"/> <property name="port" value="22"/>
<property name="user" value="xxxuser"/> <property name="password" value="xxxuser#123"/>
</bean> -->
<!-- Inbound channel adapter for SFTP call . with poll facility -->
<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
auto-startup="true" channel="receiveChannel" session-factory="sftpSessionFactory"
local-directory="file:/target/foo" remote-directory="${sftp.inboundremotedir}"
auto-create-local-directory="true" delete-remote-files="false"
filename-pattern="*.txt">
<int:poller fixed-rate="100000" max-messages-per-poll="1" />
</int-sftp:inbound-channel-adapter>
<int:channel id="receiveChannel">
<int:queue />
</int:channel>
</beans>
STackTrace
p-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 5 of 12 in additional filter chain; firing Filter: 'DefaultLoginPageGeneratingFilter'
16:02:58.368 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 6 of 12 in additional filter chain; firing Filter: 'BasicAuthenticationFilter'
16:02:58.368 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 7 of 12 in additional filter chain; firing Filter: 'RequestCacheAwareFilter'
16:02:58.368 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 8 of 12 in additional filter chain; firing Filter: 'SecurityContextHolderAwareRequestFilter'
16:02:58.370 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 9 of 12 in additional filter chain; firing Filter: 'AnonymousAuthenticationFilter'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.authentication.AnonymousAuthenticationFilter] Populated SecurityContextHolder with anonymous token: 'org.springframework.security.authentication.AnonymousAuthenticationToken#9055e4a6: Principal: anonymousUser; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails#957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: ROLE_ANONYMOUS'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 10 of 12 in additional filter chain; firing Filter: 'SessionManagementFilter'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 11 of 12 in additional filter chain; firing Filter: 'ExceptionTranslationFilter'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 12 of 12 in additional filter chain; firing Filter: 'FilterSecurityInterceptor'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.util.matcher.AntPathRequestMatcher] Checking match of request : '/'; against '/services/employee/*'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.access.intercept.FilterSecurityInterceptor] Public object - authentication not attempted
16:02:58.371 TRACE [http-bio-8080-exec-3][org.springframework.web.context.support.XmlWebApplicationContext] Publishing event in Root WebApplicationContext: org.springframework.security.access.event.PublicInvocationEvent[source=FilterInvocation: URL: /]
16:02:58.372 DEBUG [http-bio-8080-exec-3][org.springframework.beans.factory.support.DefaultListableBeanFactory] Returning cached instance of singleton bean 'org.springframework.integration.internalMessagingAnnotationPostProcessor'
16:02:58.372 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / reached end of additional filter chain; proceeding with original chain
16:02:58.375 TRACE [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] Bound request context to thread: SecurityContextHolderAwareRequestWrapper[ org.springframework.security.web.context.HttpSessionSecurityContextRepository$Servlet3SaveToSessionRequestWrapper#1b0b34e]
16:02:58.376 DEBUG [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] DispatcherServlet with name 'Information Exchange Gateway Integration' processing GET request for [/DummyDataIntg/]
16:02:58.376 TRACE [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] Testing handler map [org.springframework.integration.http.inbound.IntegrationRequestMappingHandlerMapping#cdca7] in DispatcherServlet with name 'Information Exchange Gateway Integration'
16:02:58.378 WARN [http-bio-8080-exec-3][org.springframework.web.servlet.PageNotFound] No mapping found for HTTP request with URI [/DummyDataIntg/] in DispatcherServlet with name 'Information Exchange Gateway Integration'
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.context.HttpSessionSecurityContextRepository] SecurityContext is empty or contents are anonymous - context will not be stored in HttpSession.
16:02:58.378 TRACE [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] Cleared thread-bound request context: SecurityContextHolderAwareRequestWrapper[ org.springframework.security.web.context.HttpSessionSecurityContextRepository$Servlet3SaveToSessionRequestWrapper#1b0b34e]
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] Successfully completed request
16:02:58.378 TRACE [http-bio-8080-exec-3][org.springframework.web.context.support.XmlWebApplicationContext] Publishing event in WebApplicationContext for namespace 'Information Exchange Gateway Integration-servlet': ServletRequestHandledEvent: url=[/DummyDataIntg/]; client=[127.0.0.1]; method=[GET]; servlet=[Information Exchange Gateway Integration]; session=[null]; user=[null]; time=[6ms]; status=[OK]
16:02:58.378 TRACE [http-bio-8080-exec-3][org.springframework.web.context.support.XmlWebApplicationContext] Publishing event in Root WebApplicationContext: ServletRequestHandledEvent: url=[/DummyDataIntg/]; client=[127.0.0.1]; method=[GET]; servlet=[Information Exchange Gateway Integration]; session=[null]; user=[null]; time=[6ms]; status=[OK]
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.beans.factory.support.DefaultListableBeanFactory] Returning cached instance of singleton bean 'org.springframework.integration.internalMessagingAnnotationPostProcessor'
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.access.ExceptionTranslationFilter] Chain processed normally
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.context.SecurityContextPersistenceFilter] SecurityContextHolder now cleared, as request processing completed
16:04:37.403 INFO [task-scheduler-4][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\b.txt, headers={timestamp=1420540477403, id=f8c32928-411b-99b7-f4a0-0dd1b119fc44}]]
16:04:37.403 ERROR [task-scheduler-4][org.springframework.integration.handler.LoggingHandler] \target\foo\b.txt
16:06:17.403 INFO [task-scheduler-9][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\brjb.txt, headers={timestamp=1420540577403, id=061634bf-0562-962b-e583-53a302cdb0d4}]]
16:06:17.403 ERROR [task-scheduler-9][org.springframework.integration.handler.LoggingHandler] \target\foo\brjb.txt
16:07:57.403 INFO [task-scheduler-10][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\d.txt, headers={timestamp=1420540677403, id=abea1188-fc5a-15b0-c40c-73ea686a88c0}]]
16:07:57.403 ERROR [task-scheduler-10][org.springframework.integration.handler.LoggingHandler] \target\foo\d.txt
16:09:37.403 INFO [task-scheduler-4][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\g.txt, headers={timestamp=1420540777403, id=461a8e48-6ebf-5d1c-ab3f-ce28e54e00b8}]]
16:09:37.403 ERROR [task-scheduler-4][org.springframework.integration.handler.LoggingHandler] \target\foo\g.txt
16:11:17.403 INFO [task-scheduler-3][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\h.txt, headers={timestamp=1420540877403, id=4cbeeceb-5949-0e1c-1492-45ec17172480}]]
16:11:17.403 ERROR [task-scheduler-3][org.springframework.integration.handler.LoggingHandler] \target\foo\h.txt
16:12:57.403 INFO [task-scheduler-9][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\p.txt, headers={timestamp=1420540977403, id=3767b4bb-4de4-3178-f693-5ac0bd94a766}]]
16:12:57.403 ERROR [task-scheduler-9][org.springframework.integration.handler.LoggingHandler] \target\foo\p.txt
16:14:37.403 INFO [task-scheduler-6][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\wiki.txt, headers={timestamp=1420541077403, id=bd3d0082-9788-fc31-1596-ab7a79186c17}]]
16:14:37.403 ERROR [task-scheduler-6][org.springframework.integration.handler.LoggingHandler] \target\foo\wiki.txt
error log in java file**strong text**
20:25:52.807 ERROR [task-scheduler-1][org.springframework.integration.handler.LoggingHandler] org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory; nested exception is org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: Failed to list files; nested exception is 2: No such file
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:209)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:167)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:57)
at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:64)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:124)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:192)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:149)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:298)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:292)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: Failed to list files; nested exception is 2: No such file
at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:343)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:167)
... 22 more
Caused by: org.springframework.core.NestedIOException: Failed to list files; nested exception is 2: No such file
at org.springframework.integration.sftp.session.SftpSession.list(SftpSession.java:103)
at org.springframework.integration.sftp.session.SftpSession.list(SftpSession.java:50)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.list(CachingSessionFactory.java:205)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:171)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:167)
at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:334)
... 23 more
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2198)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2215)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1565)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1526)
at org.springframework.integration.sftp.session.SftpSession.list(SftpSession.java:91)
... 28 more
You need to take a look how to start Spring Context from Web application using web.xml or WebApplicationInitializer for Servlet 3 environment. In this case the SftpInboundReceive-context.xml can be a part of common ApplicationContext and the polling facility (<int-sftp:inbound-channel-adapter>) will start automatically on application startup, which is caused on server start, when the last one see the web context of your application.
Please, read more docs for Spring Framework: http://projects.spring.io/spring-framework/
Spring Integration is just an EIP extension and follow with the same configuration and lifecycle rules.
I see that you just use the SFTP sample from Spring Intregration. You can found there samples for Tomcat and for Spring Boot as well.
I have a working SAML2.0 Single Sign On system built in coffeescript on express / nodejs.
I use SSOCircle to test my SSO and can successfully authenticate there with either HTTP-POST or HTTP-REDIRECT binding. So far so good.
Now I need to digitally sign my authentication request.
To do this I have used the nodejs xml-crytpo library. This results in SAML that looks like this:
<?xml version="1.0"?>
<samlp:AuthnRequest Version="2.0" ID="_1C8A2EFA-2644-4330-9A36-2B45547ECFAF" IssueInstant="2014-10-14T16:06:27.769Z" Destination="https://idp.ssocircle.com:443/sso/SSOPOST/metaAlias/ssocircle" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">
<saml:Issuer>http://app.localhost</saml:Issuer>
<samlp:NameIDPolicy AllowCreate="true" />
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
<SignedInfo>
<CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
<SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
<Reference URI="#_1C8A2EFA-2644-4330-9A36-2B45547ECFAF">
<Transforms>
<Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
</Transforms>
<DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
<DigestValue>m4qHiXM82TuxY31l6+QSECHEHc0=</DigestValue>
</Reference>
</SignedInfo>
<SignatureValue>fps0I0Rp02qDK0BPTK7Lh+ ...</SignatureValue>
<KeyInfo>
<X509Data>MIICuDCCAaCgAwIBAgIQEVFtJk ...</X509Data>
</KeyInfo>
</Signature>
</samlp:AuthnRequest>
The certificate information contained in the digital signature has been removed for brevity but it comes from a pem file I generated locally (self-signed).
I also updated my Service Provider metadata supplied to SSOCircle to include the SAML attributes AuthnRequestsSigned and WantAssertionsSigned, like this:
<md:EntityDescriptor entityID="http://app.localhost" xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata">
<md:SPSSODescriptor AuthnRequestsSigned="true" WantAssertionsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
...
</md:SPSSODescriptor>
</md:EntityDescriptor>
Sending my AuthnRequest without the digital signature works fine. When I include the digital signature it fails with a statuscode of 500 and a generic 'Invalid Request' message.
How do I correctly digitally sign my AuthnRequest so it works against SSOCircle?
Thanks.
Edit 1 - Add Cert Key to the SP Metadata
Following suggestions in the comments I have now added the appropriate KeyDescriptor attributes to my sp-metadata to include my self-signed certificate's public-key.
<md:EntityDescriptor entityID="http://app.localhost" xmlns:md="urn:oasis:names:tc:SAML:2.0:metadata">
<md:SPSSODescriptor AuthnRequestsSigned="true" WantAssertionsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
<md:KeyDescriptor use="signing">
<ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
<ds:X509Data>
<ds:X509SubjectName>CN=app.localhost</ds:X509SubjectName>
<ds:X509Certificate>MIICu ... wg==</ds:X509Certificate>
</ds:X509Data>
</ds:KeyInfo>
</md:KeyDescriptor>
<md:KeyDescriptor use="encryption">
<ds:KeyInfo xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
<ds:X509Data>
<ds:X509Certificate>MIICu ... wg==</ds:X509Certificate>
</ds:X509Data>
</ds:KeyInfo>
</md:KeyDescriptor>
...
</md:EntityDescriptor>
I have included two KeyDescriptor elements, one specifying the use = signing and the other specifying the use = encryption, although I think the latter is redundant (as I am running over TLS) and will be ignored by the IdP.
This does not work and I get the same error:
Reason: The SAML Request is invalid.
Edit 2 - Self Signed Cerificates
Theory: I cannot use a self-signed certificate. Instead I must use a certificate that is known to and trusted by SSOCircle
Result: This was disproved. You can use a self-signed cert just as long as the correct public key is included in the sp-metadata.
Edit 3 - Added Correct Transform Algorithm
I have now updated my signature to specify the #enveloped-signature transformation algorithm.
<Transforms>
<Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
</Transforms>
Also I found that the correct structure for the included x509 certificate public key was the following (note the inner X509Certificate element):
<KeyInfo>
<X509Data>
<X509Certificate>MIID...MMxZ</X509Certificate>
</X509Data>
</KeyInfo>
I am now using the TestShib site to run against as this allows access to the IdP logs which provides more detailed and useful information.
Edit 4 - Swapping the SAML Element Order to Match Schema
According to answer given by #Hos (see below) the order of the SAML elements must match the Schema. Therefore the signature element must appear directly after the issuer element, like this:
<samlp:AuthnRequest Version="2.0" ID="_782F6F1E-1B80-4D7D-B26C-AC85030D9300" IssueInstant="2014-10-28T11:45:49.412Z" Destination="https://idp.ssocircle.com:443/sso/SSOPOST/metaAlias/ssocircle" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">
<saml:Issuer>http://app.localhost9de83841</saml:Issuer>
<Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
<SignedInfo>
<CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
<SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
<Reference URI="#_782F6F1E-1B80-4D7D-B26C-AC85030D9300">
<Transforms>
<Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
</Transforms>
<DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
<DigestValue>pjXEtbAFMJA3hWhD/f6lxJTshTg=</DigestValue>
</Reference>
</SignedInfo>
<SignatureValue>FY1...Qg==</SignatureValue>
<KeyInfo>
<X509Data>
<X509Certificate>MIID...MMxZ</X509Certificate>
</X509Data>
</KeyInfo>
</Signature>
<samlp:NameIDPolicy AllowCreate="true" />
</samlp:AuthnRequest>
This still fails SSOCircle with an opaque 500 error. However TestShib shows the following log:
11:02:37.292 - DEBUG [edu.internet2.middleware.shibboleth.common.security.MetadataPKIXValidationInformationResolver:631] - Write lock over cache acquired
11:02:37.292 - DEBUG [edu.internet2.middleware.shibboleth.common.security.MetadataPKIXValidationInformationResolver:634] - Added new PKIX info to entity cache with key: [http://app.localhost9de83841,{urn:oasis:names:tc:SAML:2.0:metadata}SPSSODescriptor,urn:oasis:names:tc:SAML:2.0:protocol,SIGNING]
11:02:37.292 - DEBUG [edu.internet2.middleware.shibboleth.common.security.MetadataPKIXValidationInformationResolver:637] - Write lock over cache released
11:02:37.293 - WARN [edu.internet2.middleware.shibboleth.idp.profile.saml2.SSOProfileHandler:406] - Message did not meet security requirements
org.opensaml.ws.security.SecurityPolicyException: **Validation of protocol message signature failed**
As I read this, the public key used to validate the signature has been extracted from the sp-metadata:
Added new PKIX info to entity cache with key: [http://app.localhost9de83841,{urn:oasis:names:tc:SAML:2.0:metadata}SPSSODescriptor,urn:oasis:names:tc:SAML:2.0:protocol,SIGNING]
However the actual validation has failed, presumably because the digest values do not match. Which means the digest being calculated by xmlcrypto is different from the digest calculated by ShibTest. This can surely only be for one of two reasons:
The algorithms used by xmlcrypto and TestShib to generate the digest are somehow different
The public keys do not match.
A minute examination of the keys show they are the same. So here is my code in case you can spot anything.
Note: This code relies on the nodejs xmlbuilder and xml-crytpo libraries)
getSamlRequest:(idpUrl, requestId, next)->
request = #xmlbuilder.create
'samlp:AuthnRequest':
'#xmlns:samlp':'urn:oasis:names:tc:SAML:2.0:protocol'
'#xmlns:saml': 'urn:oasis:names:tc:SAML:2.0:assertion'
'#Version': '2.0'
'#ID': requestId
'#IssueInstant': (new Date()).toISOString()
'#Destination': idpUrl
'saml:Issuer': '##spEntityId'
,null
,headless: true
request.comment 'insert-signature'
request.element 'samlp:NameIDPolicy':
'#AllowCreate': 'true'
saml = request.end()
#Your self-signed pem file that contains both public and private keys.
#The public key must also be included in your sp-metadata
certFilePath = "certs/sp-certificate.pem"
#fs.readFile certFilePath, (err, certificate)=>
signer = new #xmlcrypto.SignedXml()
signer.signingKey = certificate
signer.addReference "//*[local-name(.)='AuthnRequest']", ['http://www.w3.org/2000/09/xmldsig#enveloped-signature']
signer.keyInfoProvider = new =>
getKeyInfo: (key)=>
public_key = /-----BEGIN CERTIFICATE-----([^-]*)-----END CERTIFICATE-----/g.exec(key)[1].replace /[\r\n|\n]/g, ''
"<X509Data><X509Certificate>#{public_key}</X509Certificate></X509Data>"
signer.computeSignature saml
signature = signer.getSignatureXml()
signed = saml.replace '<!-- insert-signature -->', signature
return next null, signed
Edit 5 - Trace of Requests
Below is a trace of the requests made during the SAML handshake with SSOCircle. The output was generated by the nodejs request module in conjunction with the request-debug module.
I have included some clarification notes below this trace output.
{ request:
{ debugId: 17,
uri: 'http://localhost:8082/security/sso/subscription/dev2/096b75a2-4a55-4eec-83c8-d2509e948b07',
method: 'GET',
headers: { host: 'localhost:8082' } } }
{ response:
{ debugId: 17,
headers:
{ 'transfer-encoding': 'chunked',
'content-type': 'application/json; charset=utf-8',
server: 'Microsoft-HTTPAPI/2.0',
'access-control-allow-origin': '*',
'access-control-allow-headers': 'origin, x-requested-with, accept, content-type',
'access-control-allow-methods': 'GET, POST, PATCH, PUT, DELETE',
date: 'Wed, 05 Nov 2014 16:55:49 GMT' },
statusCode: 200,
body: '{"RequestId":"_F7DDDD24-32C6-420E-A550-95872D30033B","SubscriptionName":"Develop-2","SubscriptionURL":"dev2","IdentityProviderCertificateFile":"ssocircle.cer","IdentityProviderDestinationUrl":"https://idp.ssocircle.com:443/sso/SSOPOST/metaAlias/ssocircle","SAMLBinding":"HttpPost"}' } }
{ request:
{ debugId: 18,
uri: 'https://idp.ssocircle.com:443/sso/SSOPOST/metaAlias/ssocircle',
method: 'POST',
headers:
{ host: 'idp.ssocircle.com:443',
'content-type': 'application/x-www-form-urlencoded; charset=utf-8',
'content-length': 3492 },
body: 'SAMLRequest=PHNhbWxwOkF1dGhuUmVxdWVzdCB4bWxuczpzYW1scD0idXJuOm9hc2lzOm5hbWVzOnRjOlNBTUw6Mi4wOnByb3RvY29sIiB4bWxuczpzYW1sPSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoyLjA6YXNzZXJ0aW9uIiBWZXJzaW9uPSIyLjAiIElEPSJfRjdEREREMjQtMzJDNi00MjBFLUE1NTAtOTU4NzJEMzAwMzNCIiBJc3N1ZUluc3RhbnQ9IjIwMTQtMTEtMDVUMTY6NTU6NDkuODkwWiIgRGVzdGluYXRpb249Imh0dHBzOi8vaWRwLnNzb2NpcmNsZS5jb206NDQzL3Nzby9TU09QT1NUL21ldGFBbGlhcy9zc29jaXJjbGUiPjxzYW1sOklzc3Vlcj5odHRwOi8vYXBwLmxvY2FsaG9zdDlkZTgzODQxPC9zYW1sOklzc3Vlcj48U2lnbmF0dXJlIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwLzA5L3htbGRzaWcjIj48U2lnbmVkSW5mbz48Q2Fub25pY2FsaXphdGlvbk1ldGhvZCBBbGdvcml0aG09Imh0dHA6Ly93d3cudzMub3JnLzIwMDEvMTAveG1sLWV4Yy1jMTRuIyIgLz48U2lnbmF0dXJlTWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMC8wOS94bWxkc2lnI3JzYS1zaGExIiAvPjxSZWZlcmVuY2UgVVJJPSIjX0Y3RERERDI0LTMyQzYtNDIwRS1BNTUwLTk1ODcyRDMwMDMzQiI%2BPFRyYW5zZm9ybXM%2BPFRyYW5zZm9ybSBBbGdvcml0aG09Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvMDkveG1sZHNpZyNlbnZlbG9wZWQtc2lnbmF0dXJlIiAvPjwvVHJhbnNmb3Jtcz48RGlnZXN0TWV0aG9kIEFsZ29yaXRobT0iaHR0cDovL3d3dy53My5vcmcvMjAwMC8wOS94bWxkc2lnI3NoYTEiIC8%2BPERpZ2VzdFZhbHVlPjBLdUtNVG1Zc29Wb1BlTXhtdDhuNml2M3RxZz08L0RpZ2VzdFZhbHVlPjwvUmVmZXJlbmNlPjwvU2lnbmVkSW5mbz48U2lnbmF0dXJlVmFsdWU%2BY1htVFJPWVBSWFZPNHpWSkxURXBBSWd0RTd1c2NsSG1QS0NFdTB0NXNzcmVYSjd1a0M4dWdGd2c4Zm1yWDZDenhib3FHNVl1enJkb2RWVVNsbU12bHZXMGpiaWtsQVBXMmtPNTcralB4TWV3UTdFdzJZdTJuRTZ2QkFhMUwxWGpsa0g0a1UrdlVWVEpFQnlsdXhOYjRQN0xoZnJTdnVrZDhWejJwbk5QTnJtY0tJZVp1LzFlZU8wWmRyRVZrdEY1REhVaFV0MEs5aFBhRXB5Z0xsYjVKYWhNZEttei9uQWk4OU04aTEyUTdrQ2hwb1UrcmhjYTQzaDBXdGk2SWI1N1lWRzgyMzE5MlNsdGR5UkZPdXl2bEJRL0FMSkdpN0hNYzRMRXVkQktOL3pxaUFpK1NZYm1ONVNuN0NucFJWbW13U3NJUElncWxCbmlrU2pIQzRQS1VBPT08L1NpZ25hdHVyZVZhbHVlPjxLZXlJbmZvPjxYNTA5RGF0YT48WDUwOUNlcnRpZmljYXRlPk1JSURnRENDQW1pZ0F3SUJBZ0lESUFTeE1BMEdDU3FHU0liM0RRRUJCUVVBTUM0eEN6QUpCZ05WQkFZVEFrUkZNUkl3RUFZRFZRUUtFd2xUVTA5RGFYSmpiR1V4Q3pBSkJnTlZCQU1UQWtOQk1CNFhEVEUwTVRBeE5qRTBOVFl3TlZvWERURTFNVEF4TmpFME5UWXdOVm93TmpFTE1Ba0dBMVVFQmhNQ1JFVXhFakFRQmdOVkJBb1RDVk5UVDBOcGNtTnNaVEVUTUJFR0ExVUVBeE1LWW1sdlpuSmhZM1JoYkRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS0Y0dDhzWVZNdHRVZkZ3eTZneFpQVjVWbWtCMDZwNXNqck5vUjUxUXZISmZFMkgyTnVTa0Nxa2paVmFwb1FRMStUVTNlelloMmxNVGNYSjU1Y0t2d0lUTlEvQWlLckxjaG4wdGxYZXFIRXhIdXBvazRGd1hqc20xSitpZTBvOUc1UDlRNTFXelRjRXYxSFRBV2Rkak9OK3Zsd3d0YndTaWRtNFBkN3hxZDdvQkhXTjJJSExQNlpGVHRPVWNpdzI1K0xtRk90V3dHdU41c1pNWDV6RDNUc216Y3ZNMFQwUzF0SVlHamhpaWNnM2UrbmhkWXhjSVNvZ1B1NjNlSWswKzM2OFU1TkhnYlJ5SVRnR2tPMFdtTU9PNDhpbFlFcWlPR1E3d3FxeTdrMzZzRGNxZXdiV1lvejFnQzBieGJBeXI1d3ZZUDdnaEVHYktSVXRWeDExZzdVQ0F3RUFBYU9CbmpDQm16QUpCZ05WSFJNRUFqQUFNQ3dHQ1dDR1NBR0crRUlCRFFRZkZoMVBjR1Z1VTFOTUlFZGxibVZ5WVhSbFpDQkRaWEowYVdacFkyRjBaVEFkQmdOVkhRNEVGZ1FVcTFyMnltMkRXb1NoYWgyRm11ekxJOWZjdDdZd1FRWURWUjBqQkRvd09LRXlwREF3TGpFTE1Ba0dBMVVFQmhNQ1JFVXhFakFRQmdOVkJBb1RDVk5UVDBOcGNtTnNaVEVMTUFrR0ExVUVBeE1DUTBHQ0FncVlNQTBHQ1NxR1NJYjNEUUVCQlFVQUE0SUJBUUFIRW5xTlVKN2VaYkkvNmozZTNmK2tFY3BQQUs2L3dXS0hSL1k2TWt1TGZpZERPREJ4eDRPTThQL1MrNFV4QkZSYWtBNFRkWkRNbjdJT2dTZ2Z3elQ0RjFxS1cvTkJKMzRtcjdndi9Yc0gxTDdHMlBEZFdUdmdGN1N5aFpCck9rbVYyZy9KYWg2U2pBREdabGdPWGJuTTlHUjlNN1NpVDBUcmFkay90dU9Zc2pxd2NJaE40eTVwSU9MNnZlemJKQThIeWZUY1lpMGFZVXFyMGl4Qkw1WWh5VDA1Qk13SUhkdFFNNGhqNjdyRDV4ME9QcmVrcTg0MjRkL0RGWmV5QTRBNG04Z3BOL0VDTFF0NzN3ajlqMVRudjBLdmlLRGZOcWJ4NngvL1o3MnN6VVhpWldETWJrNnhvdHVOZTV6Yy9xcXdhSWxLZ2lnWDJvcGRvbU9nTU14WjwvWDUwOUNlcnRpZmljYXRlPjwvWDUwOURhdGE%2BPC9LZXlJbmZvPjwvU2lnbmF0dXJlPjxzYW1scDpOYW1lSURQb2xpY3kgQWxsb3dDcmVhdGU9InRydWUiLz48L3NhbWxwOkF1dGhuUmVxdWVzdD4%3D' } }
{ response:
{ debugId: 18,
headers:
{ server: '"SSOCircle Web Server"',
date: 'Wed, 05 Nov 2014 16:55:48 GMT',
'content-type': 'text/html;charset=UTF-8',
connection: 'close',
'set-cookie': [Object],
'transfer-encoding': 'chunked' },
statusCode: 500,
body: '\n\n\n\n<html><head><title>\n Error Page\n \n </title>\n <link href="/css/bx.css" rel="stylesheet" type="text/css" /></head>\n <body>\n <div id="myheader">\n <h1><img src="/logo.png" alt="IDPee - Put your LOGO here" height="65" width="180"> </h1>\n </div>\n <div id="mycontent">\n <div id="mynav">\n <ul>\n \n </ul>\n </div>\n <div id="mybox">\n <p>\n \n <h2>Error occurred</h2>\n<p>\nReason: Invalid signature in Request.\n</p>\n\n \n </p>\n </div>\n \n </div>\n <div id="myfooter">\n \n Copyright © SSOCircle/IDPee.com\n \n \n </div>\n </body>\n</html>\n\n\n\n' } }
error: message=Something went wrong whilst trying to automatically log you in. Please try again., name=SSOFailure, message=An error occurred whilst processing the SSO Request., name=SSORequestFailure, message=The IdP failed to successfully process the SAMLRequest and returned an statuscode of 500., name=SamlRequestInvalid, url=http://app.localhost/security/saml2/request
Request debugId= 17: This is the call made by my client into my nodejs proxy server that kicks of the SAML handshake.
ResponsedebugId= 17: A 200 OK response from the proxy server back to the client. The client now waits to be redirected once the SAML handshake concludes.
Request debugId= 18: The SAMLRequest is POSTed to the SSOCircle SAML endpoint
ResponsedebugId= 18: A 500 Internal Server Error response from SSOCircle plus an HTML error page payload.
Concerning Edit2: A self signed certificate is sufficient for SSOCircle as long as it was included into the service provider's metadata.
Concerning Edit3: Please move the signature element right after the issuer element. The request must comply to the schema:
<sequence>
<element ref="saml:Issuer" minOccurs="0"/>
<element ref="ds:Signature" minOccurs="0"/>
<element ref="samlp:Extensions" minOccurs="0"/>
</sequence>
In continuation to the post - http://forum.spring.io/forum/spring-projects/integration/119697-unable-to-rename-file-in-sftp-remote-directory-please-help
I am using the sftp:outbound-channel-adapter to upload a file into one of the sftp server. Once file is uploaded, API is unable to rename the temporary file sample_test.pgp.writing to sample_test.pgp.
Before uploading the file I verified in the sftp remote folder for file exist and remote folder was completely empty.
When I looked at the debug level log, I could see below message and it is failing at the end with the invalid path error message.
[main] DEBUG: com.ftp.util.FileUploadUtil - Upload for file /sample_test.pgp triggered
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'ftp.uploadgateway.request.channel', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.filter.MessageFilter - org.springframework.integration.filter.MessageFilter#3970ae0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'upload.file.to.sftp', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - preSend on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.handler.LoggingHandler - org.springframework.integration.handler.LoggingHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : org.springframework.integration.handler.LoggingHandler - [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.channel.DirectChannel - postSend (sent=true) on channel 'logger', message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] DEBUG: org.springframework.integration.file.remote.handler.FileTransferringMessageHandler - org.springframework.integration.file.remote.handler.FileTransferringMessageHandler#0 received message: [Payload=/sample_test.pgp][Headers={timestamp=1406654118428, id=bbba360d-492d-4348-b2e7-566aec7f4209}]
[main] INFO : com.jcraft.jsch - Connecting to remote.sever.com port 10022
[main] INFO : com.jcraft.jsch - Connection established
[main] INFO : com.jcraft.jsch - Remote version string: SSH-2.0-SSHD
[main] INFO : com.jcraft.jsch - Local version string: SSH-2.0-JSCH-0.1.49
[main] INFO : com.jcraft.jsch - CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - CheckKexes: diffie-hellman-group14-sha1
[main] INFO : com.jcraft.jsch - diffie-hellman-group14-sha1 is not available.
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT sent
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXINIT received
[main] INFO : com.jcraft.jsch - kex: server: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: server: ssh-rsa
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc,blowfish-cbc
[main] INFO : com.jcraft.jsch - kex: server: aes128-cbc,blowfish-cbc,aes192-cbc,aes256-cbc,arcfour,arcfour128,arcfour256
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96,hmac-sha256,hmac-sha256#ssh.com
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server: none,zlib
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: server:
[main] INFO : com.jcraft.jsch - kex: client: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[main] INFO : com.jcraft.jsch - kex: client: ssh-rsa,ssh-dss
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client: none
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: client:
[main] INFO : com.jcraft.jsch - kex: server->client aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - kex: client->server aes128-cbc hmac-md5 none
[main] INFO : com.jcraft.jsch - SSH_MSG_KEXDH_INIT sent
[main] INFO : com.jcraft.jsch - expecting SSH_MSG_KEXDH_REPLY
[main] INFO : com.jcraft.jsch - ssh_rsa_verify: signature true
[main] WARN : com.jcraft.jsch - Permanently added 'remote.sever.com' (RSA) to the list of known hosts.
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS sent
[main] INFO : com.jcraft.jsch - SSH_MSG_NEWKEYS received
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_REQUEST sent
[main] INFO : com.jcraft.jsch - SSH_MSG_SERVICE_ACCEPT received
[main] INFO : com.jcraft.jsch - Authentications that can continue: publickey,keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: publickey
[main] INFO : com.jcraft.jsch - Authentications that can continue: keyboard-interactive,password
[main] INFO : com.jcraft.jsch - Next authentication method: keyboard-interactive
[main] INFO : com.jcraft.jsch - Authentication succeeded (keyboard-interactive).
[main] DEBUG: org.springframework.integration.util.SimplePool - Obtained new org.springframework.integration.sftp.session.SftpSession#6e75d758.
[main] DEBUG: org.springframework.integration.sftp.session.SftpSession - Initial File rename failed, possibly because file already exists. Will attempt to delete file: /inbox/sample_test.pgp and execute rename again.
[main] DEBUG: org.springframework.integration.file.remote.session.CachingSessionFactory - Releasing Session back to the pool.
[main] DEBUG: org.springframework.integration.util.SimplePool - Releasing org.springframework.integration.sftp.session.SftpSession#6e75d758 back to the pool
[main] DEBUG: com.ftp.service.CtrlMPOJO - ERROR UPLOADING FILES EXCEPTION IS
org.springframework.integration.MessageDeliveryException: Error handling message for file [/sample_test.pgp]
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:183)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.filter.MessageFilter.handleRequestMessage(MessageFilter.java:107)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:157)
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:128)
at org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:288)
at org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:149)
at org.springframework.integration.core.MessagingTemplate.convertAndSend(MessagingTemplate.java:189)
at org.springframework.integration.gateway.MessagingGatewaySupport.send(MessagingGatewaySupport.java:183)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invokeGatewayMethod(GatewayProxyFactoryBean.java:309)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.doInvoke(GatewayProxyFactoryBean.java:269)
at org.springframework.integration.gateway.GatewayProxyFactoryBean.invoke(GatewayProxyFactoryBean.java:260)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy5.uploadFilesToFTP(Unknown Source)
at com.ftp.util.FileUploadUtil.scanDirectoryAndUpload(FileUploadUtil.java:123)
at com.ftp.service.CtrlMPOJO.main(CtrlMPOJO.java:160)
Caused by: org.springframework.integration.MessagingException: Failed to write to '/inbox/sample_test.pgp.writing' while uploading the file
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:266)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.handleMessageInternal(FileTransferringMessageHandler.java:172)
... 28 more
Caused by: org.springframework.core.NestedIOException: Failed to delete file /inbox/sample_test.pgp; nested exception is org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:157)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.rename(CachingSessionFactory.java:137)
at org.springframework.integration.file.remote.handler.FileTransferringMessageHandler.sendFileToRemoteDirectory(FileTransferringMessageHandler.java:262)
... 29 more
Caused by: org.springframework.core.NestedIOException: Failed to remove file: 2: Specified file path is invalid.
at org.springframework.integration.sftp.session.SftpSession.remove(SftpSession.java:71)
at org.springframework.integration.sftp.session.SftpSession.rename(SftpSession.java:151)
... 31 more
It works if I set the use-temporary-file-name=false but I do not want to set this flag incase if there is any file watcher job which may pick up incomplete file loaded at the remote server end.
Here is the configuration i have
<int:gateway id="file.upload.gateway"
service-interface="ftp.outbound.FTPUploadGateway"
default-request-channel="ftp.uploadgateway.request.channel"
default-reply-channel="ftp.uploadgateway.response.channel" />
<int:filter
input-channel="ftp.uploadgateway.request.channel"
output-channel="ftp.file.exist.outbound.channel"
discard-channel="upload.file.to.sftp"
expression="${ftp.outbound.remote.file.check.flag:false}">
</int:filter>
<sftp:outbound-channel-adapter id="sftpOutboundAdapter"
session-factory="sftpSessionFactory"
channel="upload.file.to.sftp"
charset="UTF-8"
remote-directory="${ftp.outbound.remote.directory}"
use-temporary-file-name="${ftp.outbound.use.temporary.filename:true}"
remote-filename-generator-expression="${ftp.outbound.remote.filename.expression}"/>
Here are the property values
ftp.outbound.remote.file.check.flag=false
ftp.outbound.remote.directory=/inbox/
ftp.outbound.use.temporary.filename=true
ftp.outbound.remote.filename.expression=payload.getName()
Please show the complete configuration.
Is your sftp user chrooted? If not /inbox/... is trying to manipulate the root directory on the server, and he likely doesn't have permissions for that.
Try removing the leading / from the remote dir.
I am trying to run Kafka-0.8 Log4j appender and I am unable to make it.
I want my application to send log directly to kafka via Log4j appender.
Here is my log4j.properties.
I couldn`t find any proper encoder, so I just configure it to use default encoder.
(e.g I commented the line.)
log4j.rootLogger=INFO, stdout, KAFKA
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%-5p: %c - %m%n
log4j.appender.KAFKA.BrokerList=hnode01:9092
log4j.appender.KAFKA.Topic=DKTestEvent
#log4j.appender.KAFKA.SerializerClass=kafka.log4j.AppenderStringEncoder
And this is my sample application.
import org.apache.log4j.Logger;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.PropertyConfigurator;
public class HelloWorld {
static Logger logger = Logger.getLogger(HelloWorld.class.getName());
public static void main(String[] args) {
PropertyConfigurator.configure(args[0]);
logger.info("Entering application.");
logger.debug("Debugging!.");
logger.info("Exiting application.");
}
}
I used maven for compiling.
I included kafka_2.8.2-0.8.0 and log4j_1.2.17 in my pom.xml
And I am getting these error:
INFO [main] (Logging.scala:67) - Verifying properties
INFO [main] (Logging.scala:67) - Property metadata.broker.list is overridden to hnode01:9092
INFO [main] (Logging.scala:67) - Property serializer.class is overridden to kafka.serializer.StringEncoder
INFO [main] (HelloWorld.java:14) - Entering application.
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 0 for 1 topic(s) Set(DKTestEvent)
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 1 for 1 topic(s) Set(DKTestEvent)
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 2 for 1 topic(s) Set(DKTestEvent)
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 3 for 1 topic(s) Set(DKTestEvent)
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 4 for 1 topic(s) Set(DKTestEvent)
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 5 for 1 topic(s) Set(DKTestEvent)
.
.
.
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 60 for 1 topic(s) Set(DKTestEvent)
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 61 for 1 topic(s) Set(DKTestEvent)
INFO [main] (HelloWorld.java:14) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 62 for 1 topic(s) Set(DKTestEvent)
INFO [main] (Logging.scala:67) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 63 for 1 topic(s) Set(DKTestEvent)
INFO [main] (Logging.scala:67) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 64 for 1 topic(s) Set(DKTestEvent)
INFO [main] (Logging.scala:67) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 65 for 1 topic(s) Set(DKTestEvent)
INFO [main] (Logging.scala:67) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 66 for 1 topic(s) Set(DKTestEvent)
INFO [main] (Logging.scala:67) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 67 for 1 topic(s) Set(DKTestEvent)
.
.
.
INFO [main] (Logging.scala:67) - Fetching metadata from broker id:0,host:hnode01,port:9092 with correlation id 534 for 1 topic(s) Set(DKTestEvent)
ERROR [main] (Logging.scala:67) -
ERROR [main] (Logging.scala:67) -
ERROR [main] (Logging.scala:67) -
ERROR [main] (Logging.scala:67) -
ERROR [main] (Logging.scala:67) -
ERROR [main] (Logging.scala:67) -
java.lang.StackOverflowError
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.error(Category.java:322)
at kafka.utils.Logging$$anonfun$swallowError$1.apply(Logging.scala:105)
at kafka.utils.Logging$$anonfun$swallowError$1.apply(Logging.scala:105)
at kafka.utils.Utils$.swallow(Utils.scala:189)
at kafka.utils.Logging$class.swallowError(Logging.scala:105)
at kafka.utils.Utils$.swallowError(Utils.scala:46)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.producer.KafkaLog4jAppender.append(KafkaLog4jAppender.scala:96)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.info(Category.java:666)
at kafka.utils.Logging$class.info(Logging.scala:67)
at kafka.client.ClientUtils$.info(ClientUtils.scala:31)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:51)
at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
at kafka.utils.Utils$.swallow(Utils.scala:187)
at kafka.utils.Logging$class.swallowError(Logging.scala:105)
at kafka.utils.Utils$.swallowError(Utils.scala:46)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.producer.KafkaLog4jAppender.append(KafkaLog4jAppender.scala:96)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
.
.
.
I am getting above error continuously if i don`t terminate the program.
If I miss something, kindly let me know.
I think Jonas has identified the problem, that is the Kafka producer logging is also getting logged to the Kafka appender causing an infinite loop and eventual stack overflow (no pun intended)
You can configure all Kafka logs to go to a different appender. The following shows sending the output to stdout:
log4j.logger.kafka=INFO, stdout
So you should end up with the following in your log4j.properties
log4j.rootLogger=INFO, stdout, KAFKA
log4j.logger.kafka=INFO, stdout
log4j.logger.HelloWorld=INFO, KAFKA
I have been able to generate events via log4j in Kafka 0.8.2.2. Here is my log4j configuration:
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-5p %c{1} - %m%n" />
</layout>
</appender>
<appender name="fileAppender" class="org.apache.log4j.RollingFileAppender">
<param name="Threshold" value="INFO" />
<param name="MaxBackupIndex" value="100" />
<param name="File" value="/tmp/agna-LogFile.log" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d %-5p [%c{1}] %m %n" />
</layout>
</appender>
<appender name="kafkaAppender" class="kafka.producer.KafkaLog4jAppender">
<param name="Topic" value="kafkatopic" />
<param name="BrokerList" value="localhost:9092" />
<param name="syncSend" value="true" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n" />
</layout>
</appender>
<logger name="org.apache.kafka">
<level value="error" />
<appender-ref ref="console" />
</logger>
<logger name="com.example.kafkaLogger">
<level value="debug" />
<appender-ref ref="kafkaAppender" />
</logger>
<root>
<priority value="debug" />
<appender-ref ref="console" />
</root>
</log4j:configuration>
Here is the source code:
package com.example;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import java.util.Properties;
import java.util.concurrent.ExecutionException;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
public class JsonProducer {
static Logger defaultLogger = LoggerFactory.getLogger(JsonProducer.class);
static Logger kafkaLogger = LoggerFactory.getLogger("com.example.kafkaLogger");
public static void main(String args[]) {
JsonProducer obj = new JsonProducer();
String str = obj.getJsonObjAsString();
// Use the logger
kafkaLogger.info(str);
try {
// Construct and send message
obj.constructAndSendMessage();
} catch (InterruptedException e) {
defaultLogger.error("Caught interrupted exception " + e);
} catch (ExecutionException e) {
defaultLogger.error("Caught execution exception " + e);
}
}
private String getJsonObjAsString() {
JSONObject obj = new JSONObject();
obj.put("name", "John");
obj.put("age", new Integer(55));
obj.put("address", "123 MainSt, Palatine, IL");
JSONArray list = new JSONArray();
list.add("msg 1");
list.add("msg 2");
list.add("msg 3");
obj.put("messages", list);
return obj.toJSONString();
}
private void constructAndSendMessage() throws InterruptedException, ExecutionException {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
boolean sync = false;
String topic = "kafkatopic";
String key = "mykey";
String value = "myvalue1 mayvalue2 myvalue3";
ProducerRecord<String, String> producerRecord = new ProducerRecord<String, String>(topic, key, value);
if (sync) {
producer.send(producerRecord).get();
} else {
producer.send(producerRecord);
}
producer.close();
}
}
The whole project is a available under the following link:
https://github.com/ypant/kafka-json-producer.git
Try to set the appender async, like this:
log4j.appender.KAFKA.ProducerType=async
Seems reasonable that it goes in to an infinite loop because the kafka producer has logging in itself..