How can I get this simple (IBM MQ) MDB project to deploy to Tomee... -what configuration is required for a successful deployment - cdi

QUESTION:
What CDI annotations, tomee.xml configuration [,etc.] will allow successful deployment of this simple message driven bean?
I'm attempting to deploy a MessageDriven bean using CDI and IBM MQ - running on Tomee 7.1 Plume...
After many iterations, I am still at a loss as to how to overcome the Exception below and would appreciate some help/guidance.
--Are my "ActivationConfigProperty" values (in the "ChatBean" mdb example) correct, based on the tomee.xml file?
Exception
-
-
-
11-Oct-2018 19:20:27.427 INFO [localhost-startStop-1] org.apache.openejb.config.AutoConfig.logAutoCreateResource Auto-creating a Resource with id 'DEV.QUEUE.1' of type 'javax.jms.Queue for 'ChatBean'.
11-Oct-2018 19:20:27.427 INFO [localhost-startStop-1] org.apache.openejb.assembler.classic.Assembler.createRecipe Creating Resource(id=DEV.QUEUE.1)
11-Oct-2018 19:20:27.432 INFO [localhost-startStop-1] org.apache.openejb.config.ConfigurationFactory.configureService Configuring Service(id=Default Managed Container, type=Container, provider-id=Default Managed Container)
11-Oct-2018 19:20:27.432 INFO [localhost-startStop-1] org.apache.openejb.config.AutoConfig.createContainer Auto-creating a container for bean SimpleMDB.Comp263849086: Container(type=MANAGED, id=Default Managed Container)
11-Oct-2018 19:20:27.432 INFO [localhost-startStop-1] org.apache.openejb.assembler.classic.Assembler.createRecipe Creating Container(id=Default Managed Container)
11-Oct-2018 19:20:27.437 INFO [localhost-startStop-1] org.apache.openejb.core.managed.SimplePassivater.init Using directory /usr/local/tomee/temp for stateful session passivation
11-Oct-2018 19:20:27.445 INFO [localhost-startStop-1] org.apache.openejb.config.AutoConfig.processResourceRef Auto-linking resource-ref 'java:comp/env/imq_qcf' in bean SimpleMDB.Comp263849086 to Resource(id=imq_qcf)
11-Oct-2018 19:20:27.445 INFO [localhost-startStop-1] org.apache.openejb.config.AutoConfig.processResourceEnvRef Auto-linking resource-env-ref 'java:comp/env/imq.baq' in bean SimpleMDB.Comp263849086 to Resource(id=imq.baq)
11-Oct-2018 19:20:27.449 INFO [localhost-startStop-1] org.apache.openejb.config.AppInfoBuilder.build Enterprise application "/usr/local/tomee/webapps/SimpleMDB" loaded.
11-Oct-2018 19:20:27.449 INFO [localhost-startStop-1] org.apache.openejb.assembler.classic.Assembler.createApplication Assembling app: /usr/local/tomee/webapps/SimpleMDB
11-Oct-2018 19:20:27.471 INFO [localhost-startStop-1] org.apache.openejb.assembler.classic.JndiBuilder.bind Jndi(name=ChatBean) --> Ejb(deployment-id=ChatBean)
11-Oct-2018 19:20:27.489 INFO [localhost-startStop-1] org.apache.openejb.cdi.CdiBuilder.initSingleton Existing thread singleton service in SystemInstance(): org.apache.openejb.cdi.ThreadSingletonServiceImpl#340b9973
11-Oct-2018 19:20:27.554 INFO [localhost-startStop-1] org.apache.openejb.cdi.OpenEJBLifecycle.startApplication OpenWebBeans Container is starting...
11-Oct-2018 19:20:27.557 INFO [localhost-startStop-1] org.apache.webbeans.plugins.PluginLoader.startUp Adding OpenWebBeansPlugin : [CdiPlugin]
11-Oct-2018 19:20:27.699 INFO [localhost-startStop-1] org.apache.webbeans.config.BeansDeployer.validateInjectionPoints All injection points were validated successfully.
11-Oct-2018 19:20:27.703 INFO [localhost-startStop-1] org.apache.openejb.cdi.OpenEJBLifecycle.startApplication OpenWebBeans Container has started, it took 149 ms.
11-Oct-2018 19:20:27.838 INFO [localhost-startStop-1] org.apache.openejb.assembler.classic.Assembler.destroyApplication Undeploying app: /usr/local/tomee/webapps/SimpleMDB
11-Oct-2018 19:20:27.841 SEVERE [localhost-startStop-1] jdk.internal.reflect.NativeMethodAccessorImpl.invoke ContainerBase.removeChild: destroy:
-
-
-
11-Oct-2018 19:20:27.926 SEVERE [localhost-startStop-1] jdk.internal.reflect.NativeMethodAccessorImpl.invoke ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/SimpleMDB]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: org.apache.tomee.catalina.TomEERuntimeException: org.apache.openejb.OpenEJBException: Creating application failed: /usr/local/tomee/webapps/SimpleMDB: Error deploying 'ChatBean'. Exception: class org.apache.openejb.OpenEJBException: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.: MQJCA1011: Failed to allocate a JMS connection.: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.: MQJCA1011: Failed to allocate a JMS connection.
at org.apache.tomee.catalina.TomcatWebAppBuilder.startInternal(TomcatWebAppBuilder.java:1315)
at org.apache.tomee.catalina.TomcatWebAppBuilder.configureStart(TomcatWebAppBuilder.java:1125)
at org.apache.tomee.catalina.GlobalListenerSupport.lifecycleEvent(GlobalListenerSupport.java:133)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 10 more
Caused by: org.apache.openejb.OpenEJBException: Creating application failed: /usr/local/tomee/webapps/SimpleMDB: Error deploying 'ChatBean'. Exception: class org.apache.openejb.OpenEJBException: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.: MQJCA1011: Failed to allocate a JMS connection.: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.: MQJCA1011: Failed to allocate a JMS connection.
at org.apache.openejb.assembler.classic.Assembler.createApplication(Assembler.java:1096)
at org.apache.openejb.assembler.classic.Assembler.createApplication(Assembler.java:757)
at org.apache.tomee.catalina.TomcatWebAppBuilder.startInternal(TomcatWebAppBuilder.java:1303)
... 15 more
Caused by: org.apache.openejb.OpenEJBException: Error deploying 'ChatBean'. Exception: class org.apache.openejb.OpenEJBException: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.: MQJCA1011: Failed to allocate a JMS connection.: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.: MQJCA1011: Failed to allocate a JMS connection.
at org.apache.openejb.assembler.classic.Assembler.startEjbs(Assembler.java:1686)
at org.apache.openejb.assembler.classic.Assembler.createApplication(Assembler.java:997)
... 17 more
Caused by: org.apache.openejb.OpenEJBException: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.: MQJCA1011: Failed to allocate a JMS connection.
at org.apache.openejb.core.mdb.MdbContainer.deploy(MdbContainer.java:250)
at org.apache.openejb.assembler.classic.Assembler.startEjbs(Assembler.java:1673)
... 18 more
Caused by: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.
at com.ibm.mq.connector.services.JCAExceptionBuilder.buildException(JCAExceptionBuilder.java:174)
at com.ibm.mq.connector.services.JCAExceptionBuilder.buildException(JCAExceptionBuilder.java:135)
at com.ibm.mq.connector.inbound.ConnectionHandler.allocateConnection(ConnectionHandler.java:394)
at com.ibm.mq.connector.inbound.MessageEndpointDeployment.acquireConnection(MessageEndpointDeployment.java:289)
at com.ibm.mq.connector.inbound.MessageEndpointDeployment.<init>(MessageEndpointDeployment.java:229)
at com.ibm.mq.connector.ResourceAdapterImpl.endpointActivation(ResourceAdapterImpl.java:558)
at org.apache.openejb.core.mdb.MdbContainer$MdbActivationContext.start(MdbContainer.java:641)
at org.apache.openejb.core.mdb.MdbContainer.deploy(MdbContainer.java:232)
... 19 more
Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception occurred in the Java(tm) MQI.
The Java(tm) MQI has thrown an exception describing the problem.
See the linked exception for further information.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
at com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
at com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:226)
at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createV7ProviderConnection(WMQXAConnectionFactory.java:172)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7815)
at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createProviderXAConnection(WMQXAConnectionFactory.java:98)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createXAConnectionInternal(JmsConnectionFactoryImpl.java:354)
at com.ibm.mq.jms.MQXAConnectionFactory.createXAConnection(MQXAConnectionFactory.java:104)
at com.ibm.mq.connector.inbound.ConnectionHandler.allocateConnection(ConnectionHandler.java:268)
... 24 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2195;AMQ9546: Error return code received. [1=java.lang.NoSuchMethodException[com.ibm.mq.jmqi.remote.api.RemoteFAP.<init>(com.ibm.mq.jmqi.JmqiEnvironment, int)],3=Class.getConstructor0]
at com.ibm.mq.jmqi.JmqiEnvironment.getInstance(JmqiEnvironment.java:831)
at com.ibm.mq.jmqi.JmqiEnvironment.getMQI(JmqiEnvironment.java:676)
at com.ibm.msg.client.wmq.factories.WMQXAConnectionFactory.createV7ProviderConnection(WMQXAConnectionFactory.java:164)
... 29 more
Caused by: java.lang.NoSuchMethodException: com.ibm.mq.jmqi.remote.api.RemoteFAP.<init>(com.ibm.mq.jmqi.JmqiEnvironment, int)
at java.base/java.lang.Class.getConstructor0(Class.java:3302)
at java.base/java.lang.Class.getConstructor(Class.java:2110)
at com.ibm.mq.jmqi.JmqiEnvironment.getInstance(JmqiEnvironment.java:738)
... 31 more
11-Oct-2018 19:20:27.926 SEVERE [localhost-startStop-1] jdk.internal.reflect.NativeMethodAccessorImpl.invoke Error deploying web application archive [/usr/local/tomee/webapps/SimpleMDB.war]
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/SimpleMDB]]
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:758)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:985)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:844)
11-Oct-2018 19:20:27.927 INFO [localhost-startStop-1] jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke Deployment of web application archive [/usr/local/tomee/webapps/SimpleMDB.war] has finished in [1,926] ms
11-Oct-2018 19:20:27.927 INFO [localhost-startStop-1] jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke Deploying web application directory [/usr/local/tomee/webapps/ROOT]
11-Oct-2018 19:20:27.928 INFO [localhost-startStop-1] org.apache.tomee.catalina.TomcatWebAppBuilder.init ------------------------- localhost -> /
-
-
-
Note, my example (CDI annotated) MDB was derived from this link:
http://tomee.apache.org/examples-trunk/simple-mdb-and-cdi/
...Unfortunately (for me) I could not locate the tomee.xml or resources.xml upon which the example - "simple-mdb-and-cdi" - annotations are based...
conf/tomee.xml
<tomee>
<Container id="imq_container" type="MESSAGE">
ResourceAdapter=imq_ra
MessageListenerInterface=javax.jms.MessageListener
ActivationSpecClass=com.ibm.mq.connector.inbound.ActivationSpecImpl
</Container>
<Resource id="imq_ra" type="com.ibm.mq.connector.ResourceAdapterImpl" class-name="com.ibm.mq.connector.ResourceAdapterImpl">
connectionConcurrency=5
maxConnections=10
logWriterEnabled=false
reconnectionRetryCount=5
reconnectionRetryInterval=300000
traceEnabled=false
traceLevel=3
</Resource>
<Resource id="imq_qcf" type="javax.jms.ConnectionFactory" class-name="com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl">
TransactionSupport=none
ResourceAdapter=imq_ra
HostName=172.17.0.3
Port=1414
QueueManager=QM1
Channel=DEV.ADMIN.SVRCONN
TransportType=CLIENT
UserName=admin
Password=passw0rd
</Resource>
<Resource id="imq_container-javax.jms.QueueConnectionFactory" type="javax.jms.QueueConnectionFactory" class-name="com.ibm.mq.connector.outbound.ManagedQueueConnectionFactoryImpl">
TransactionSupport=xa
ResourceAdapter=imq_ra
</Resource>
<Resource id="imq.bar" type="javax.jms.Queue" class-name="com.ibm.mq.connector.outbound.MQQueueProxy">
arbitraryProperties
baseQueueManagerName=QM1
baseQueueName=DEV.QUEUE.1
CCSID=1208
encoding=NATIVE
expiry=APP
failIfQuiesce=true
persistence=APP
priority=APP
readAheadClosePolicy=ALL
targetClient=JMS
</Resource>
<Resource id="imq.baq" type="javax.jms.Queue" class-name="com.ibm.mq.connector.outbound.MQQueueProxy">
arbitraryProperties
baseQueueManagerName=QM1
baseQueueName=DEV.QUEUE.2
CCSID=1208
encoding=NATIVE
expiry=APP
failIfQuiesce=true
persistence=APP
priority=APP
readAheadClosePolicy=ALL
targetClient=JMS
</Resource>
</tomee>
ChatBean.java
package org.superbiz.mdb;
import javax.annotation.Resource;
import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.inject.Inject;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.DeliveryMode;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
#MessageDriven(activationConfig = {
#ActivationConfigProperty(propertyName = "destination", propertyValue = "DEV.QUEUE.1"),
#ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")
})
public class ChatBean implements MessageListener {
#Resource(name = "java:comp/env/imq_qcf")
private ConnectionFactory connectionFactory;
#Resource(name = "java:comp/env/imq.baq")
private Queue answerQueue;
#Inject
private ChatRespondCreator responder;
#Override
public void onMessage(Message message) {
try {
final TextMessage textMessage = (TextMessage) message;
final String question = textMessage.getText();
respond("...the question was: " + question);
} catch (JMSException e) {
throw new IllegalStateException(e);
}
}
private void respond(String text) throws JMSException {
Connection connection = null;
Session session = null;
try {
connection = connectionFactory.createConnection();
connection.start();
// Create a Session
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create a MessageProducer from the Session to the Topic or Queue
MessageProducer producer = session.createProducer(answerQueue);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
// Create a message
TextMessage message = session.createTextMessage(text);
// Tell the producer to send the message
producer.send(message);
} finally {
// Clean up
if (session != null) {
session.close();
}
if (connection != null) {
connection.close();
}
}
}
}
src/main/resources/beans.xml
<beans/>
IBM MQ 9.1 jars were extracted from wmq.jmsra-9.1.0.0.rar, and added to tomee/lib folder (as per: tomee.apache.org/tomee-and-webspheremq.html)
bcpkix-jdk15on.jar
bcprov-jdk15on.jar
com.ibm.mq.connector.jar
com.ibm.mq.headers.jar
com.ibm.mq.jar
com.ibm.mq.jmqi.jar
com.ibm.mqjms.jar
com.ibm.mq.pcf.jar
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>aaa.bbb.ccc</groupId>
<artifactId>SimpleMDB</artifactId>
<version>1</version>
<packaging>war</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<timestamp>${maven.build.timestamp}</timestamp>
<maven.build.timestamp.format>yyyyMMdd.HHmmss</maven.build.timestamp.format>
<skipTests>true</skipTests>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>8.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>jaxb-runtime</artifactId>
<version>2.3.0.1</version>
</dependency>
<dependency>
<groupId>com.ibm.mq</groupId>
<artifactId>com.ibm.mq.allclient</artifactId>
<version>9.1.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.tomee</groupId>
<artifactId>openejb-core</artifactId>
<version>7.1.0</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>${project.artifactId}</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<release>10</release>
<fork>true</fork>
<executable>/usr/local/jdkv10/bin/javac</executable>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>3.2.2</version>
<configuration>
<failOnMissingWebXml>false</failOnMissingWebXml>
<archive>
<manifestEntries>
<Tomee-Application-Version>${project.version}-${timestamp}</Tomee-Application-Version>
</manifestEntries>
</archive>
</configuration>
</plugin>
</plugins>
</build>
</project>
FWIW, to provide some more context: the "sendMessage" and "receiveMessage" methods in this stateless bean (below) work fine with the tomee.xml configuration and the tomee/lib MQ jars.
the issues seems to be related (indirectly?) to the CDI annotations/"wiring" of the message driven bean(MDB)... I though that perhaps my annotations may be improperly wired to the tomee.xml configuration(?)
package org.superbiz.mdb;
import javax.annotation.Resource;
import javax.ejb.Stateless;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.DeliveryMode;
import javax.jms.JMSException;
import javax.jms.MessageConsumer;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
#Stateless
public class Messages {
#Resource(name = "imq_qcf")
private ConnectionFactory connectionFactory;
#Resource(name = "imq.bar")
private Queue devQueue1;
public void sendMessage(String text) throws JMSException {
System.out.println("\n\n...........................Messages/sendMessage............................\n\n");
Connection connection = null;
Session session = null;
try {
connection = connectionFactory.createConnection();
connection.start();
// Create a Session
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create a MessageProducer from the Session to the Topic or Queue
MessageProducer producer = session.createProducer(devQueue1);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
// Create a message
TextMessage message = session.createTextMessage(text);
// Tell the producer to send the message
producer.send(message);
} finally {
// Clean up
if (session != null) session.close();
if (connection != null) connection.close();
}
}
public String receiveMessage() throws JMSException {
System.out.println("\n\n...........................Messages/receiveMessage............................\n\n");
Connection connection = null;
Session session = null;
MessageConsumer consumer = null;
try {
connection = connectionFactory.createConnection();
connection.start();
// Create a Session
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create a MessageConsumer from the Session to the Topic or Queue
consumer = session.createConsumer(devQueue1);
// Wait for a message
TextMessage message = (TextMessage) consumer.receive(1000);
return message.getText();
} finally {
if (consumer != null) consumer.close();
if (session != null) session.close();
if (connection != null) connection.close();
}
}
}
Environment:
Java 10
IBM MQ Version 9.0
Tomee 7.1.0 Plume

Can now deploy working MDB on apache tomee plume 7.x connecting to IBM MQ queue, etc.
Love Apache Tomee, so far - but, had to research/google to piece together info from various posts and/or documentation fragments to arrive this poc/solution below... -Not clear whether this solution is recommended(or, ill-advised), but, it is what I'm looking for, in that I did NOT want to include the IBM MQ ".rar" file in my war (or ear) file... -but, rather wanted to deploy to a "pre-configured" (lib, config files) tomee server, etc. (in my case, residing in a Docker container)
first placed jar files from "wmq.jmsra-9.1.0.0.rar" into tomee/lib folder
-rw-r--r--. 1 1000 1000 464030 Jul 5 21:38 com.ibm.mq.connector.jar
-rw-r--r--. 1 1000 1000 359297 Jul 5 21:37 com.ibm.mq.headers.jar
-rw-r--r--. 1 1000 1000 397171 Jul 5 21:37 com.ibm.mq.jar
-rw-r--r--. 1 1000 1000 3374322 Jul 5 21:38 com.ibm.mq.jmqi.jar
-rw-r--r--. 1 1000 1000 127566 Jul 5 21:37 com.ibm.mq.pcf.jar
-rw-r--r--. 1 1000 1000 3999810 Jul 5 21:38 com.ibm.mqjms.jar
-rw-r--r--. 1 1000 1000 792825 Jul 5 21:37 bcpkix-jdk15on.jar
-rw-r--r--. 1 1000 1000 4092400 Jul 5 21:37 bcprov-jdk15on.jar
(note: this seemed adequate for non-MDB, IBM MQ connectivity)
created folder called "ibm_rar" and placed "wmq.jmsra-9.1.0.0.rar" file into it
root#c2865c04748a:/usr/local/tomee# ls -l
total 120
-rw-r--r--. 1 root staff 52711 Jul 10 11:41 LICENSE
-rw-r--r--. 1 root staff 7692 Jul 10 11:22 NOTICE
-rw-r--r--. 1 root staff 7316 Jun 20 19:51 RELEASE-NOTES
-rw-r--r--. 1 root staff 16722 Jun 20 19:51 RUNNING.txt
drwxr-xr-x. 2 root staff 4096 Oct 15 17:55 bin
drwxr-xr-x. 4 root staff 4096 Oct 15 20:57 conf
drwxr-x---. 3 root root 19 Sep 20 14:33 data
drwxr-xr-x. 3 root root 58 Oct 15 17:55 ibm_rar <== contains ".rar" file
drwxr-xr-x. 2 root staff 8192 Oct 15 17:37 lib
drwxr-xr-x. 2 root staff 4096 Oct 16 14:11 logs
drwxr-xr-x. 2 root staff 4096 Oct 16 14:11 temp
drwxr-xr-x. 7 root staff 4096 Oct 15 22:37 webapps
drwxr-xr-x. 3 root staff 21 Sep 20 14:33 work
root#c2865c04748a:/usr/local/tomee#
contents of manually created folder - arbitrarily named: "ibm_rar"
root#c2865c04748a:/usr/local/tomee/ibm_rar# ls -l
total 12080
drwxr-x---. 3 root root 4096 Oct 15 17:55 wmq.jmsra-9.1.0.0
-rw-rw-r--. 1 1000 1000 12361911 Oct 4 21:11 wmq.jmsra-9.1.0.0.rar
root#c2865c04748a:/usr/local/tomee/ibm_rar#
(note: appears that on startup, the tomee server explodes the ".rar" file into a corresponding folder)
contents of tomee.xml
note "Deployments" entry pointing to location of .rar file
root#c2865c04748a:/usr/local/tomee/conf# cat tomee.xml
<tomee>
<!-- folder containing resource adapter -->
<Deployments dir="ibm_rar" />
<Resource id="wmqRA" type="com.ibm.mq.connector.ResourceAdapterImpl" class-name="com.ibm.mq.connector.ResourceAdapterImpl">
connectionConcurrency=5
maxConnections=10
logWriterEnabled=false
reconnectionRetryCount=5
reconnectionRetryInterval=300000
traceEnabled=false
traceLevel=3
</Resource>
<Resource id="wmqCF" type="javax.jms.ConnectionFactory" class-name="com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl">
TransactionSupport=none
ResourceAdapter=wmqRA
HostName=172.17.0.3
Port=1414
QueueManager=QM1
Channel=DEV.ADMIN.SVRCONN
TransportType=Client
UserName=admin
Password=passw0rd
</Resource>
<Resource id="wmqQCF" type="javax.jms.QueueConnectionFactory" class-name="com.ibm.mq.connector.outbound.ManagedQueueConnectionFactoryImpl">
TransactionSupport=xa
ResourceAdapter=wmqRA
</Resource>
<Container id="wmq" type="MESSAGE">
BrokerXmlConfig =
ResourceAdapter=wmqRA
MessageListenerInterface=javax.jms.MessageListener
ActivationSpecClass=com.ibm.mq.connector.inbound.ActivationSpecImpl
</Container>
<Resource id="devQueue1" type="javax.jms.Queue" class-name="com.ibm.mq.connector.outbound.MQQueueProxy">
arbitraryProperties
baseQueueManagerName=QM1
baseQueueName=DEV.QUEUE.1
CCSID=819
encoding=NATIVE
expiry=APP
failIfQuiesce=true
persistence=APP
priority=APP
readAheadClosePolicy=ALL
targetClient=JMS
</Resource>
<Resource id="devQueue2" type="javax.jms.Queue" class-name="com.ibm.mq.connector.outbound.MQQueueProxy">
arbitraryProperties
baseQueueManagerName=QM1
baseQueueName=DEV.QUEUE.2
CCSID=819
encoding=NATIVE
expiry=APP
failIfQuiesce=true
persistence=APP
priority=APP
readAheadClosePolicy=ALL
targetClient=JMS
</Resource>
</tomee>
ChatBean.java (MDB)
package org.superbiz.mdb;
import javax.annotation.Resource;
import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.DeliveryMode;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageListener;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
#MessageDriven(activationConfig = {
//...annotating "ejb-jar.xml" properties here...
#ActivationConfigProperty(propertyName = "destination",propertyValue = "DEV.QUEUE.1"),
#ActivationConfigProperty(propertyName = "destinationType",propertyValue = "javax.jms.Queue"),
#ActivationConfigProperty(propertyName = "channel",propertyValue = "DEV.ADMIN.SVRCONN"),
#ActivationConfigProperty(propertyName = "hostName",propertyValue = "172.17.0.3"),
#ActivationConfigProperty(propertyName = "port",propertyValue = "1414"),
#ActivationConfigProperty(propertyName = "queueManager",propertyValue = "QM1"),
#ActivationConfigProperty(propertyName = "transportType",propertyValue = "CLIENT"),
#ActivationConfigProperty(propertyName = "username",propertyValue = "admin"),
#ActivationConfigProperty(propertyName = "password",propertyValue = "passw0rd")
}
)
public class ChatBean implements MessageListener {
#Resource(name = "wmqCF")
private ConnectionFactory connectionFactory;
#Resource(name = "devQueue2")
private Queue answerQueue;
#Override
public void onMessage(Message message) {
try {
final TextMessage textMessage = (TextMessage) message;
final String msg = textMessage.getText();
System.out.println("\n\n\n..........ChatBean/onMessage..........incoming message: " + String.valueOf(msg));
respond("...the message received was: " + String.valueOf(msg));
} catch (JMSException e) {
throw new IllegalStateException(e);
}
}
private void respond(String text) throws JMSException {
Connection connection = null;
Session session = null;
try {
connection = connectionFactory.createConnection();
System.out.println("\n\n\n..........ChatBean/respond..........connection object:" + String.valueOf(connection));
System.out.println("\n\n\n..........ChatBean/respond..........responding with text:" + String.valueOf(text));
connection.start();
// Create a Session
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// Create a MessageProducer from the Session to the Topic or Queue
MessageProducer producer = session.createProducer(answerQueue);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
// Create a message
TextMessage textmessage = session.createTextMessage(text);
System.out.println("\n\n\n..........ChatBean/respond..........responding with textmessage:" + String.valueOf(textmessage));
System.out.println("\n\n\n");
// Tell the producer to send the message
producer.send(textmessage);
} finally {
// Clean up
if (session != null) {
session.close();
}
if (connection != null) {
connection.close();
}
}
}
}
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>aaa.bbb.ccc</groupId>
<artifactId>SimpleMDB</artifactId>
<version>1.11</version>
<packaging>war</packaging>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<timestamp>${maven.build.timestamp}</timestamp>
<maven.build.timestamp.format>yyyyMMdd.HHmmss</maven.build.timestamp.format>
<skipTests>true</skipTests>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>8.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>jaxb-runtime</artifactId>
<version>2.3.0.1</version>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.3.0</version>
</dependency>
</dependencies>
<build>
<finalName>${project.artifactId}</finalName>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<release>10</release>
<fork>true</fork>
<executable>/usr/local/jdkv10/bin/javac</executable>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>3.2.2</version>
<configuration>
<failOnMissingWebXml>false</failOnMissingWebXml>
<archive>
<manifestEntries>
<Tomee-Application-Version>${project.version}-${timestamp}</Tomee-Application-Version>
</manifestEntries>
</archive>
</configuration>
</plugin>
</plugins>
</build>
</project>
war file copied/deployed to tomee/webapps folder
root#c2865c04748a:/usr/local/tomee/webapps# ls -l
total 1668
drwxr-xr-x. 3 root staff 4096 Sep 20 14:18 ROOT
drwxr-x---. 4 root root 53 Oct 15 22:37 SimpleMDB
-rw-rw-r--. 1 1000 1000 1694696 Oct 15 22:36 SimpleMDB.war
drwxr-xr-x. 14 root staff 4096 Sep 20 14:18 docs
drwxr-xr-x. 5 root staff 82 Sep 20 14:18 host-manager
drwxr-xr-x. 5 root staff 4096 Sep 20 14:18 manager
root#c2865c04748a:/usr/local/tomee/webapps#
(tomee server explodes into corresponding folder on startup)

Related

apache pulsar : error while consuming message

I am trying to consume from pulsar topic. My client code can be seen below.
But, I am facing NoClassDefFoundError. Please let me know how to resolve this.
pulsar client version: 2.10, pulsar server version: 2.10,
installation type: standalone(not docker), OS: Ubuntu 20.04.4 LTS
My pulsar client code:
private static void consumeFromPulsarAsync() throws Exception {
logger.info("consumeFromPulsarAsync()");
PulsarClient client = PulsarClient.
builder()
.serviceUrl("pulsar://localhost:6650")
.build();
logger.info("consumeFromPulsarAsync() client");
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.topic("persistent://public/default/ack-2")
.consumerName("pulsar-consumer-id-" + Math.random())
.subscriptionName("pulsar-subscription-id-" + Math.random())
.subscriptionType(SubscriptionType.Shared).subscribe();
logger.info("consumeFromPulsarAsync() consumer");
consumer.receiveAsync().thenCompose((msg) -> {
logger.info("consumeFromPulsarAsync() consumed msg :: {}", msg.getValue());
try {
consumer.acknowledge(msg);
} catch (PulsarClientException e) {
throw new RuntimeException(e);
}
return null;
});
}
My pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.company</groupId>
<artifactId>my.work.manager</artifactId>
<version>release-1.0</version>
<description>my Worker Manager</description>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<flink.version>1.14.3</flink.version>
<pulsar.version>2.10.0</pulsar.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.12</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-pulsar</artifactId>
<version>1.15.0</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.9</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.22</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20210307</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.32</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.30</version>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>1.30</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.13</version>
</dependency>
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-client</artifactId>
<version>${pulsar.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerArgs>
<arg>-Xlint:all,-options,-path</arg>
</compilerArgs>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
<configuration>
<archive>
<manifest>
<mainClass>com.company.my.manager.flink.myWorkManager</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<resources>
<resource>
<directory>src/main/resources</directory>
<targetPath>${project.build.outputDirectory}</targetPath>
<includes>
<include>**/*.yml</include>
</includes>
</resource>
</resources>
</build>
</project>
Error it is showing:
2022-05-10 17:52:52,549 WARN
org.apache.pulsar.client.impl.MultiTopicsConsumerImpl [] -
Encountered error in partition auto update timer task for multi-topic
consumer. Another task will be scheduled.
java.lang.NoClassDefFoundError:
org/apache/pulsar/shade/com/google/common/primitives/Ints at
org.apache.pulsar.shade.com.google.common.collect.Lists.computeArrayListCapacity(Lists.java:152)
~[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.shade.com.google.common.collect.Lists.newArrayListWithExpectedSize(Lists.java:192)
~[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$TopicsPartitionChangedListener.onTopicsExtended(MultiTopicsConsumerImpl.java:1238)
~[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.client.impl.MultiTopicsConsumerImpl$1.run(MultiTopicsConsumerImpl.java:1350)
[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.shade.io.netty.util.HashedWheelTimer$HashedWheelTimeout.run(HashedWheelTimer.java:715)
[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.shade.io.netty.util.concurrent.ImmediateExecutor.execute(ImmediateExecutor.java:34)
[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.shade.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:703)
[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.shade.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:790)
[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.shade.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:503)
[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[e8fe35f4-c20c-4c3e-9b81-a3ff1a0ebaec_my.work.manager-release-1.0-jar-with-dependencies.jar:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
What type of cluster? Docker, Standalone, K8, ...
Is the Pulsar client and server both 2.10?
What JDK for server? for client?
Any errors on the server?
What server OS?
What client OS?
It says WARN, did it run or crash with the missing class? Sometimes on a MAC you will get a WARN on missing SSL and it will work with warnings.
It may be related to this https://github.com/apache/pulsar/issues/9585
Steps to debug:
Create a new topic.
Try again.
Try to produce and consume with command line client.
Create a single partitioned topic and test with that.
I ran your example with this test
bin/pulsar-client produce --key "test1" "persistent://public/default/ack-2" -m "Test this thing 4" -n 25
--
public void consumeFromPulsarAsync() throws Exception {
PulsarClient client = PulsarClient.
builder()
.serviceUrl("pulsar://localhost:6650")
.build();
Consumer<String> consumer = client.newConsumer(Schema.STRING)
.topic("persistent://public/default/ack-2")
.consumerName("pulsar-consumer-id-" + Math.random())
.subscriptionName("pulsar-subscription-id-" + Math.random())
.subscriptionType(SubscriptionType.Shared).subscribe();
consumer.receiveAsync().thenCompose((msg) -> {
log.info("consumeFromPulsarAsync() consumed msg :: {}", msg.getValue());
try {
consumer.acknowledge(msg);
} catch (PulsarClientException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
return null;
});
}
--
2022-05-11 10:00:47.321 INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConnectionPool : [[id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650]] Connected to server
2022-05-11 10:00:47.321 INFO 28575 --- [r-client-io-6-1] org.apache.pulsar.client.impl.ClientCnx : [id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650] Connected through proxy to target broker at 127.0.0.1:6650
2022-05-11 10:00:47.323 INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-2][pulsar-subscription-id-0.6345962322446594] Subscribing to topic on cnx [id: 0xf5af925d, L:/192.168.1.63:49385 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 2
2022-05-11 10:00:47.324 INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-2][pulsar-subscription-id-0.6808890118894377] Subscribing to topic on cnx [id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 2
2022-05-11 10:00:47.324 INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-1][pulsar-subscription-id-0.6345962322446594] Subscribing to topic on cnx [id: 0xf5af925d, L:/192.168.1.63:49385 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 1
2022-05-11 10:00:47.324 INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-1][pulsar-subscription-id-0.6808890118894377] Subscribing to topic on cnx [id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 1
2022-05-11 10:00:47.324 INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-0][pulsar-subscription-id-0.6345962322446594] Subscribing to topic on cnx [id: 0xf5af925d, L:/192.168.1.63:49385 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 0
2022-05-11 10:00:47.324 INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-0][pulsar-subscription-id-0.6808890118894377] Subscribing to topic on cnx [id: 0x2005ff4b, L:/192.168.1.63:49386 - R:pulsar1.fios-router.home/192.168.1.230:6650], consumerId 0
2022-05-11 10:00:47.357 INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-0][pulsar-subscription-id-0.6808890118894377] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 0
2022-05-11 10:00:47.370 INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-1][pulsar-subscription-id-0.6808890118894377] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 1
2022-05-11 10:00:47.370 INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-1][pulsar-subscription-id-0.6345962322446594] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 1
2022-05-11 10:00:47.371 INFO 28575 --- [r-client-io-6-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-2][pulsar-subscription-id-0.6808890118894377] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 2
2022-05-11 10:00:47.372 INFO 28575 --- [r-client-io-6-1] o.a.p.c.impl.MultiTopicsConsumerImpl : [persistent://public/default/ack-2] [pulsar-subscription-id-0.6808890118894377] Success subscribe new topic persistent://public/default/ack-2 in topics consumer, partitions: 3, allTopicPartitionsNumber: 3
2022-05-11 10:00:47.394 INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-0][pulsar-subscription-id-0.6345962322446594] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 0
2022-05-11 10:00:47.394 INFO 28575 --- [r-client-io-7-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/ack-2-partition-2][pulsar-subscription-id-0.6345962322446594] Subscribed to topic on pulsar1.fios-router.home/192.168.1.230:6650 -- consumer: 2
2022-05-11 10:00:47.395 INFO 28575 --- [r-client-io-7-1] o.a.p.c.impl.MultiTopicsConsumerImpl : [persistent://public/default/ack-2] [pulsar-subscription-id-0.6345962322446594] Success subscribe new topic persistent://public/default/ack-2 in topics consumer, partitions: 3, allTopicPartitionsNumber: 3
2022-05-11 10:00:52.529 INFO 28575 --- [t-internal-11-1] a.f.a.consumer.AirQualityConsumerApp : consumeFromPulsarAsync() consumed msg :: Test this thing 4
2022-05-11 10:00:52.529 INFO 28575 --- [nt-internal-9-1] a.f.a.consumer.AirQualityConsumerApp : consumeFromPulsarAsync() consumed msg :: Test this thing 4
^C2022-05-11 10:00:54.823 INFO 28575 --- [r-client-io-1-1] o.a.pulsar.client.impl.ConsumerImpl : [persistent://public/default/airquality] [airqualitysj1] Closed consumer
2022-05-11 10:00:54.824 INFO 28575 --- [ionShutdownHook] o.a.pulsar.client.impl.PulsarClientImpl : Client closing. URL: pulsar://pulsar1:6650
2022-05-11 10:00:54.826 INFO 28575 --- [r-client-io-1-1] org.apache.pulsar.client.impl.ClientCnx : [id: 0x4d247f3d, L:/192.168.1.63:49381 ! R:pulsar1.fios-router.home/192.168.1.230:6650] Disconnected
2022-05-11 10:00:54.829 INFO 28575 --- [r-client-io-1-1] org.apache.pulsar.client.impl.ClientCnx : [id: 0x5fa81a8b, L:/192.168.1.63:49382 ! R:pulsar1.fios-router.home/192.168.1.230:6650] Disconnected

Azure Key Vault Secrets Spring Boot Starter 0.1.5

I am using older version of Azure Key Vault Secrets Spring Boot Starter -- 0.1.5
Also older version of spring-boot-starter as shown below::
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.15.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web-services</artifactId>
</dependency>
There were no code changes for Azure keyvault, In application.properties we added below:
azure.keyvault.uri=<<Keyvault URL>>
azure.client-id=<<Keyvault client Id>>
azure.client-key=<<Keyvault Key>>
Now I upgraded the versions of Spring boot starter and azure keyvault as below::
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.3.RELEASE</version>
</parent>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-keyvault-secrets-spring-boot-starter</artifactId>
<version>2.3.3</version>
<scope>runtime</scope>
</dependency>
application.properties
azure.keyvault.uri=<<Keyvault URL>>
azure.keyvault.enabled=true
azure.client-id=<<Client-ID>>
azure.client-key=<<Cleint key>>
Its failing with below Errors, Can you please help what i need to do if i am upgrading to latest version.
13:36:59.270 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
java.lang.IllegalStateException: Failed to configure KeyVault property source
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:110)
at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:47)
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:200)
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:188)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:80)
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53)
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at com.honeywell.CaaS.GatewayManagerAPI.webapi.SpringBootMain.main(SpringBootMain.java:47)
Caused by: java.lang.RuntimeException: Max retries 3 times exceeded. Error Details: Network is unreachable: connect
at com.azure.core.http.policy.RetryPolicy.lambda$attemptAsync$1(RetryPolicy.java:119)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:251)
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:251)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onError(FluxPeekFuseable.java:545)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onError(FluxPeekFuseable.java:545)
at reactor.core.publisher.Operators.error(Operators.java:196)
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:134)
at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171)
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2346)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribeInner(MonoFlatMapMany.java:143)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:182)
at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:162)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156)
at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:432)
at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:518)
at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:561)
at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:448)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:594)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1287)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1324)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Network is unreachable: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:606)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
at com.azure.identity.implementation.IdentityClient.lambda$checkIMDSAvailable$14(IdentityClient.java:463)
at reactor.core.publisher.MonoCallable.call(MonoCallable.java:91)
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:126)
It seems that you miss the property azure.keyvault.tenant-id in the application.properties.
I have reproduced this error java.lang.IllegalStateException: Failed to configure KeyVault property source. See:
After adding azure.keyvault.tenant-id=<your-tenant-id> in the file. It works:
I refer to How to use the Spring Boot Starter for Azure Key Vault and the sample. To let it work, make sure the application has permissions to get and list secrets in access policies of your key vault.
application.properties
azure.keyvault.enabled=true
azure.keyvault.uri=https://{your-azure-keyvault}.vault.azure.net/
azure.keyvault.client-id=put-your-azure-client-id-here
azure.keyvault.client-key=put-your-azure-client-key-here
azure.keyvault.tenant-id=put-your-azure-tenant-id-here
pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.3.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<java.version>11</java.version>
<azure.version>2.3.3</azure.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-spring-boot-bom</artifactId>
<version>${azure.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
DemoKeyVaultApplication.java
#SpringBootApplication
#RestController
public class DemoKeyVaultApplication implements CommandLineRunner {
#Value("${test1}")
private String testSecretName;
public static void main(final String[] args) {
SpringApplication.run(DemoKeyVaultApplication.class, args);
}
public void run(final String... varl) throws Exception {
System.out.println("testSecretName stored in Azure Key Vault: " + testSecretName);
}
}

Spring-Cloud-Zipkin runs incorrectly about 'Exception in thread "main" java.lang.ClassNotFoundException'

I am new in spring-cloud.My projects are about spring-cloud-config, spring-cloud-eureka, spring-cloud-zipkin,I am running projects locally and it is normal.When I put my projects in Ubuntu, the project 'spring-cloud-zipkin' runs incorrectly. The error is about 'Exception in thread "main" java.lang.ClassNotFoundException', thanks for your answer very much.
enter image description here
and my code:
enter image description here
enter image description here
in the pom.xml:
enter image description here
I dont kown can you see my pic and I put my code:
pom.xml:
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<mainClass>com.test.itoken.zipkin.ZipKinApplication</mainClass>
</configuration>
</plugin>
</plugins>
</build>
ZipkinApplication.java:
package com.test.itoken.zipkin;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;
import zipkin.server.internal.EnableZipkinServer;
#SpringBootApplication
#EnableEurekaClient
#EnableZipkinServer
public class ZipkinApplication {
public static void main(String[] args) {
SpringApplication.run(ZipkinApplication.class, args);
}
}
The error:
Exception in thread "main" java.lang.ClassNotFoundException: com.test.itoken.zipkin.ZipKinApplication
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:46)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)

Apache Beam:'Expect srcResourceIds and destResourceIds have the same scheme, but received hdfs,filename'

I want to run a pipeline with Spark runner and data is stored on a remote machine. The following command has been used to submit the job:
./spark-submit --class org.apache.beam.examples.WindowedWordCount --master spark://192.168.1.214:6066 --deploy-mode cluster --supervise --executor-memory 2G --total-executor-cores 4 hdfs://192.168.1.214:9000/input/word-count-ck-0.1.jar --runner=SparkRunner
It is creating a temporary file '.temp-beam-2017-07-184_19-10-19-0' in the output directory.However, it is throwing an IllegalArgumentException while finalizing the write operation(See log below):
17/07/04 00:40:29 INFO Executor: Adding file:/usr/local/spark/spark-1.6.3-bin-hadoop2.6/work/app-20170704004020-0000/1/./word-count-ck-0.1.jar to class loader
17/07/04 00:40:29 INFO TorrentBroadcast: Started reading broadcast variable 0
17/07/04 00:40:29 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 7.4 KB, free 1247.2 MB)
17/07/04 00:40:29 INFO TorrentBroadcast: Reading broadcast variable 0 took 102 ms
17/07/04 00:40:30 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 15.6 KB, free 1247.2 MB)
17/07/04 00:40:30 INFO CacheManager: Partition rdd_0_1 not found, computing it
17/07/04 00:40:31 INFO MemoryStore: Block rdd_0_1 stored as values in memory (estimated size 292.9 KB, free 1246.9 MB)
17/07/04 00:40:33 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 9074 bytes result sent to driver
17/07/04 00:40:34 INFO CoarseGrainedExecutorBackend: Got assigned task 2
17/07/04 00:40:34 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
17/07/04 00:40:34 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache
17/07/04 00:40:34 INFO TorrentBroadcast: Started reading broadcast variable 1
17/07/04 00:40:34 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 8.4 KB, free 1246.9 MB)
17/07/04 00:40:34 INFO TorrentBroadcast: Reading broadcast variable 1 took 76 ms
17/07/04 00:40:34 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 18.4 KB, free 1246.9 MB)
17/07/04 00:40:34 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 1, fetching them
17/07/04 00:40:34 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker#192.168.1.214:35429)
17/07/04 00:40:34 INFO MapOutputTrackerWorker: Got the output locations
17/07/04 00:40:34 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
17/07/04 00:40:34 INFO ShuffleBlockFetcherIterator: Started 1 remote fetches in 11 ms
17/07/04 00:40:34 INFO WriteFiles: Opening writer 59cdfe11-3fff-4188-b9ca-17fce87d3ee2 for write operation TextWriteOperation{tempDirectory=hdfs://192.168.1.214:9000/beamWorks/ckoutput/.temp-beam-2017-07-184_19-10-19-0/, windowedWrites=true}, window [2017-07-03T19:30:00.000Z..2017-07-03T19:40:00.000Z) pane PaneInfo.NO_FIRING
17/07/04 00:40:34 INFO WriteFiles: Opening writer f69da197-b163-4eee-8456-12cd7435ba8d for write operation TextWriteOperation{tempDirectory=hdfs://192.168.1.214:9000/beamWorks/ckoutput/.temp-beam-2017-07-184_19-10-19-0/, windowedWrites=true}, window [2017-07-03T19:50:00.000Z..2017-07-03T20:00:00.000Z) pane PaneInfo.NO_FIRING
17/07/04 00:40:34 INFO WriteFiles: Opening writer 99089ec2-d54f-492c-8dc6-7dbdb0d6ab8d for write operation TextWriteOperation{tempDirectory=hdfs://192.168.1.214:9000/beamWorks/ckoutput/.temp-beam-2017-07-184_19-10-19-0/, windowedWrites=true}, window [2017-07-03T19:10:00.000Z..2017-07-03T19:20:00.000Z) pane PaneInfo.NO_FIRING
17/07/04 00:40:34 INFO WriteFiles: Opening writer a5f369b3-355b-4db7-b894-c8b35c20b274 for write operation TextWriteOperation{tempDirectory=hdfs://192.168.1.214:9000/beamWorks/ckoutput/.temp-beam-2017-07-184_19-10-19-0/, windowedWrites=true}, window [2017-07-03T19:20:00.000Z..2017-07-03T19:30:00.000Z) pane PaneInfo.NO_FIRING
17/07/04 00:40:34 INFO WriteFiles: Opening writer bd717b0e-af3c-461c-a554-3b472369bc84 for write operation TextWriteOperation{tempDirectory=hdfs://192.168.1.214:9000/beamWorks/ckoutput/.temp-beam-2017-07-184_19-10-19-0/, windowedWrites=true}, window [2017-07-03T19:40:00.000Z..2017-07-03T19:50:00.000Z) pane PaneInfo.NO_FIRING
17/07/04 00:40:34 INFO WriteFiles: Opening writer f0135931-d310-4e31-8b58-caa752e75e6b for write operation TextWriteOperation{tempDirectory=hdfs://192.168.1.214:9000/beamWorks/ckoutput/.temp-beam-2017-07-184_19-10-19-0/, windowedWrites=true}, window [2017-07-03T20:00:00.000Z..2017-07-03T20:10:00.000Z) pane PaneInfo.NO_FIRING
17/07/04 00:40:34 INFO WriteFiles: Opening writer e6a9e90d-ef5d-46d2-8fc2-8b374727e63e for write operation TextWriteOperation{tempDirectory=hdfs://192.168.1.214:9000/beamWorks/ckoutput/.temp-beam-2017-07-184_19-10-19-0/, windowedWrites=true}, window [2017-07-03T20:10:00.000Z..2017-07-03T20:20:00.000Z) pane PaneInfo.NO_FIRING
17/07/04 00:40:35 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 6819 bytes result sent to driver
17/07/04 00:40:35 INFO CoarseGrainedExecutorBackend: Got assigned task 5
17/07/04 00:40:35 INFO Executor: Running task 1.0 in stage 2.0 (TID 5)
17/07/04 00:40:35 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache
17/07/04 00:40:35 INFO TorrentBroadcast: Started reading broadcast variable 2
17/07/04 00:40:35 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.8 KB, free 1246.9 MB)
17/07/04 00:40:35 INFO TorrentBroadcast: Reading broadcast variable 2 took 10 ms
17/07/04 00:40:35 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 22.2 KB, free 1246.9 MB)
17/07/04 00:40:35 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 0, fetching them
17/07/04 00:40:35 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker#192.168.1.214:35429)
17/07/04 00:40:35 INFO MapOutputTrackerWorker: Got the output locations
17/07/04 00:40:35 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
17/07/04 00:40:35 INFO ShuffleBlockFetcherIterator: Started 1 remote fetches in 2 ms
17/07/04 00:40:36 INFO WriteFiles: Finalizing write operation TextWriteOperation{tempDirectory=hdfs://192.168.1.214:9000/beamWorks/ckoutput/.temp-beam-2017-07-184_19-10-19-0/, windowedWrites=true}.
17/07/04 00:40:36 ERROR Executor: Exception in task 1.0 in stage 2.0 (TID 5)
org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Expect srcResourceIds and destResourceIds have the same scheme, but received hdfs, ck2-19.
at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at org.apache.beam.sdk.io.WriteFiles$1$auxiliary$LSdUfeyo.invokeProcessElement(Unknown Source)
at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:197)
at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:158)
at org.apache.beam.runners.spark.translation.DoFnRunnerWithMetrics.processElement(DoFnRunnerWithMetrics.java:64)
at org.apache.beam.runners.spark.translation.SparkProcessContext$ProcCtxtIterator.computeNext(SparkProcessContext.java:165)
at org.apache.beam.runners.spark.repackaged.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
at org.apache.beam.runners.spark.repackaged.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:41)
at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$32.apply(RDD.scala:912)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Expect srcResourceIds and destResourceIds have the same scheme, but received hdfs, ck2-19.
at org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.apache.beam.sdk.io.FileSystems.validateSrcDestLists(FileSystems.java:398)
at org.apache.beam.sdk.io.FileSystems.copy(FileSystems.java:240)
at org.apache.beam.sdk.io.FileBasedSink$WriteOperation.copyToOutputFiles(FileBasedSink.java:641)
at org.apache.beam.sdk.io.FileBasedSink$WriteOperation.finalize(FileBasedSink.java:529)
at org.apache.beam.sdk.io.WriteFiles$1.processElement(WriteFiles.java:539)
The following are the plugins and dependencies i used in my project:
<packaging>jar</packaging>
<properties>
<beam.version>2.0.0</beam.version>
<surefire-plugin.version>2.20</surefire-plugin.version>
</properties>
<repositories>
<repository>
<id>apache.snapshots</id>
<name>Apache Development Snapshot Repository</name>
<url>https://repository.apache.org/content/repositories/snapshots/</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-spark</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-hadoop-file-system</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.6.3</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-flink_2.10</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.10</artifactId>
<version>2.8.8</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>${beam.version}</version>
<!-- <exclusions>
<exclusion>
<artifactId>beam-sdks-java-core</artifactId>
</exclusion>
</exclusions> -->
</dependency>
<!-- Adds a dependency on the Beam Google Cloud Platform IO module. -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>${beam.version}</version>
</dependency>
<!-- Dependencies below this line are specific dependencies needed by the examples code. -->
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.22.0</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-bigquery</artifactId>
<version>v2-rev295-1.22.0</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.22.0</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-pubsub</artifactId>
<version>v1-rev10-1.22.0</version>
<exclusions>
<!-- Exclude an old version of guava that is being pulled
in by a transitive dependency of google-api-client -->
<exclusion>
<groupId>com.google.guava</groupId>
<artifactId>guava-jdk5</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>20.0</version>
</dependency>
<!-- Add slf4j API frontend binding with JUL backend -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.14</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>1.7.14</version>
<!-- When loaded at runtime this will wire up slf4j to the JUL backend -->
<scope>runtime</scope>
</dependency>
<!-- Hamcrest and JUnit are required dependencies of PAssert,
which is used in the main code of DebuggingWordCount example. -->
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-hadoop-common</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-hadoop-file-system</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-hadoop-input-format</artifactId>
<version>${beam.version}</version>
</dependency>
<!-- The DirectRunner is needed for unit tests. -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-direct-java</artifactId>
<version>${beam.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>3.0.0-alpha2</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-plugin.version}</version>
<configuration>
<parallel>all</parallel>
<threadCount>4</threadCount>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
</configuration>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>${surefire-plugin.version}</version>
</dependency>
</dependencies>
</plugin>
<!-- Ensure that the Maven jar plugin runs before the Maven
shade plugin by listing the plugin higher within the file. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</plugin>
<!--
Configures `mvn package` to produce a bundled jar ("fat jar") for runners
that require this for job submission to a cluster.
-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/LICENSE</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.4.0</version>
<configuration>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>
The following is the source code of the WindowedWordCount class:
package org.apache.beam.examples;
import java.io.IOException;
import java.util.concurrent.ThreadLocalRandom;
import org.apache.beam.examples.common.ExampleBigQueryTableOptions;
import org.apache.beam.examples.common.ExampleOptions;
import org.apache.beam.examples.common.WriteOneFilePerWindow;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.io.TextIO;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.DefaultValueFactory;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.windowing.FixedWindows;
import org.apache.beam.sdk.transforms.windowing.Window;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
import org.joda.time.Duration;
import org.joda.time.Instant;
public class WindowedWordCount {
static final int WINDOW_SIZE = 10; // Default window duration in minutes
static class AddTimestampFn extends DoFn<String, String> {
private static final Duration RAND_RANGE = Duration.standardHours(1);
private final Instant minTimestamp;
private final Instant maxTimestamp;
AddTimestampFn(Instant minTimestamp, Instant maxTimestamp) {
this.minTimestamp = minTimestamp;
this.maxTimestamp = maxTimestamp;
}
#ProcessElement
public void processElement(ProcessContext c) {
Instant randomTimestamp =
new Instant(
ThreadLocalRandom.current()
.nextLong(minTimestamp.getMillis(), maxTimestamp.getMillis()));
/**
* Concept #2: Set the data element with that timestamp.
*/
c.outputWithTimestamp(c.element(), new Instant(randomTimestamp));
}
}
/** A {#link DefaultValueFactory} that returns the current system time. */
public static class DefaultToCurrentSystemTime implements DefaultValueFactory<Long> {
// #Override
public Long create(PipelineOptions options) {
return System.currentTimeMillis();
}
}
/** A {#link DefaultValueFactory} that returns the minimum timestamp plus one hour. */
public static class DefaultToMinTimestampPlusOneHour implements DefaultValueFactory<Long> {
// #Override
public Long create(PipelineOptions options) {
return options.as(Options.class).getMinTimestampMillis()
+ Duration.standardHours(1).getMillis();
}
}
/**
* Options for {#link WindowedWordCount}.
*
* <p>Inherits standard example configuration options, which allow specification of the
* runner, as well as the {#link WordCount.WordCountOptions} support for
* specification of the input and output files.
*/
public interface Options extends WordCount.WordCountOptions,
ExampleOptions, ExampleBigQueryTableOptions {
#Description("Fixed window duration, in minutes")
#Default.Integer(WINDOW_SIZE)
Integer getWindowSize();
void setWindowSize(Integer value);
#Description("Minimum randomly assigned timestamp, in milliseconds-since-epoch")
#Default.InstanceFactory(DefaultToCurrentSystemTime.class)
Long getMinTimestampMillis();
void setMinTimestampMillis(Long value);
#Description("Maximum randomly assigned timestamp, in milliseconds-since-epoch")
#Default.InstanceFactory(DefaultToMinTimestampPlusOneHour.class)
Long getMaxTimestampMillis();
void setMaxTimestampMillis(Long value);
#Description("Fixed number of shards to produce per window, or null for runner-chosen sharding")
Integer getNumShards();
void setNumShards(Integer numShards);
}
public static void main(String[] args) throws IOException {
String[] args1 =new String[]{ "--hdfsConfiguration=[{\"fs.defaultFS\" : \"hdfs://192.168.1.214:9000\"}]","--runner=SparkRunner"};
Options options = PipelineOptionsFactory.fromArgs(args1).withValidation().as(Options.class);
final String output = options.getOutput();
final Instant minTimestamp = new Instant(options.getMinTimestampMillis());
final Instant maxTimestamp = new Instant(options.getMaxTimestampMillis());
Pipeline pipeline = Pipeline.create(options);
/**
* Concept #1: the Beam SDK lets us run the same pipeline with either a bounded or
* unbounded input source.
*/
PCollection<String> input = pipeline
/** Read from the GCS file. */
.apply(TextIO.read().from(options.getInputFile()))
// Concept #2: Add an element timestamp, using an artificial time just to show windowing.
// See AddTimestampFn for more detail on this.
.apply(ParDo.of(new AddTimestampFn(minTimestamp, maxTimestamp)));
/**
* Concept #3: Window into fixed windows. The fixed window size for this example defaults to 1
* minute (you can change this with a command-line option). See the documentation for more
* information on how fixed windows work, and for information on the other types of windowing
* available (e.g., sliding windows).
*/
PCollection<String> windowedWords =
input.apply(
Window.<String>into(
FixedWindows.of(Duration.standardMinutes(options.getWindowSize()))));
/**
* Concept #4: Re-use our existing CountWords transform that does not have knowledge of
* windows over a PCollection containing windowed values.
*/
PCollection<KV<String, Long>> wordCounts = windowedWords.apply(new WordCount.CountWords());
/**
* Concept #5: Format the results and write to a sharded file partitioned by window, using a
* simple ParDo operation. Because there may be failures followed by retries, the
* writes must be idempotent, but the details of writing to files is elided here.
*/
wordCounts
.apply(MapElements.via(new WordCount.FormatAsTextFn()))
.apply(new WriteOneFilePerWindow(output, options.getNumShards()));
PipelineResult result = pipeline.run();
try {
result.waitUntilFinish();
} catch (Exception exc) {
result.cancel();
}
}
}

spring integration-sftp inbound adapter with polling facility at server startup

I am trying to sftp file with Spring integration using a maven web projecy
Need a polling facility. If I am starting the SftpInbound.java, the polling is working. Need to have the polling at server start up.
The content of java file and configuration
SftpInbound.java
package com.myproj.integration.bsy.sftp;
import java.io.File;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import org.springframework.integration.endpoint.SourcePollingChannelAdapter;
import org.springframework.integration.file.remote.RemoteFileTemplate;
import org.springframework.integration.file.remote.session.CachingSessionFactory;
import org.springframework.integration.file.remote.session.SessionFactory;
import org.springframework.messaging.Message;
import org.springframework.messaging.PollableChannel;
import org.springframework.scheduling.annotation.Scheduled;
import com.myproj.integration.bsy.sftp.*;
import com.jcraft.jsch.ChannelSftp.LsEntry;
public class SftpInboundReceive {
#Scheduled(fixedRate=5000)
public void inboundSftpPoll(){
ConfigurableApplicationContext context =
new ClassPathXmlApplicationContext("/META-INF/spring/integration/sftp/SftpInboundReceive-context.xml", this.getClass());
RemoteFileTemplate<LsEntry> template = null;
String file1 = "a.txt";
String file2 = "b.txt";
String file3 = "c.bar";
new File("local-dir", file1).delete();
new File("local-dir", file2).delete();
try {
PollableChannel localFileChannel = context.getBean("receiveChannel", PollableChannel.class);
#SuppressWarnings("unchecked")
SessionFactory<LsEntry> sessionFactory = context.getBean(CachingSessionFactory.class);
template = new RemoteFileTemplate<LsEntry>(sessionFactory);
System.out.println("here 1" +template);
SourcePollingChannelAdapter adapter = context.getBean("sftpInbondAdapter",SourcePollingChannelAdapter.class);
adapter.start();
Message<?> received = localFileChannel.receive();
System.out.println("Received first file message 1: " + received);
received = localFileChannel.receive();
System.out.println("Received second file message: " + received);
received = localFileChannel.receive(1000);
System.out.println("Third file was received as expected" +received);
}catch(Exception e){
e.printStackTrace();
}
finally {
SftpUtils.cleanUp(template, file1, file2, file3);
//context.close();
}
}
public static void main(String args[])
{
SftpInboundReceive oInboundReceiveSample = new SftpInboundReceive();
oInboundReceiveSample.inboundSftpPoll();
}
}
The xml file SftpInboundReceive-context.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration"
xmlns:int-sftp="http://www.springframework.org/schema/integration/sftp"
xmlns:context="http://www.springframework.org/schema/context"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.1.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-4.1.xsd
http://www.springframework.org/schema/integration/sftp http://www.springframework.org/schema/integration/sftp/spring-integration-sftp-4.1.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context-4.1.xsd">
<!-- <import resource="SftpSampleCommon.xml"/> -->
<context:property-placeholder order="1"
location="classpath:/sftpuser.properties" ignore-unresolvable="true"/>
<bean id="sftpSessionFactory"
class="org.springframework.integration.file.remote.session.CachingSessionFactory">
<constructor-arg ref="defaultSftpSessionFactory" />
</bean>
<!-- host=xxx.xx.128.143 port=22 username=xxxuser passphrase= private.keyfile=classpath:META-INF/keys/sftp_rsa -->
<bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${sftp.host}" />
<property name="port" value="${sftp.port}" />
<property name="user" value="${sftp.username}" />
<property name="privateKey" value="${private.keyfile}" />
<property name="privateKeyPassphrase" value="${passphrase}" />
</bean>
<!-- username & password from property file... tested <bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="${sftp.host}"/> <property name="port" value="${sftp.port}"/>
<property name="user" value="${sftp.username}"/> <property name="password"
value="${sftp.password}"/> </bean> -->
<!-- hardcoded, username & password... tested <bean id="defaultSftpSessionFactory"
class="org.springframework.integration.sftp.session.DefaultSftpSessionFactory">
<property name="host" value="xxx.xx.128.143"/> <property name="port" value="22"/>
<property name="user" value="xxxuser"/> <property name="password" value="xxxuser#123"/>
</bean> -->
<!-- Inbound channel adapter for SFTP call . with poll facility -->
<int-sftp:inbound-channel-adapter id="sftpInbondAdapter"
auto-startup="true" channel="receiveChannel" session-factory="sftpSessionFactory"
local-directory="file:/target/foo" remote-directory="${sftp.inboundremotedir}"
auto-create-local-directory="true" delete-remote-files="false"
filename-pattern="*.txt">
<int:poller fixed-rate="100000" max-messages-per-poll="1" />
</int-sftp:inbound-channel-adapter>
<int:channel id="receiveChannel">
<int:queue />
</int:channel>
</beans>
STackTrace
p-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 5 of 12 in additional filter chain; firing Filter: 'DefaultLoginPageGeneratingFilter'
16:02:58.368 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 6 of 12 in additional filter chain; firing Filter: 'BasicAuthenticationFilter'
16:02:58.368 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 7 of 12 in additional filter chain; firing Filter: 'RequestCacheAwareFilter'
16:02:58.368 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 8 of 12 in additional filter chain; firing Filter: 'SecurityContextHolderAwareRequestFilter'
16:02:58.370 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 9 of 12 in additional filter chain; firing Filter: 'AnonymousAuthenticationFilter'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.authentication.AnonymousAuthenticationFilter] Populated SecurityContextHolder with anonymous token: 'org.springframework.security.authentication.AnonymousAuthenticationToken#9055e4a6: Principal: anonymousUser; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails#957e: RemoteIpAddress: 127.0.0.1; SessionId: null; Granted Authorities: ROLE_ANONYMOUS'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 10 of 12 in additional filter chain; firing Filter: 'SessionManagementFilter'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 11 of 12 in additional filter chain; firing Filter: 'ExceptionTranslationFilter'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / at position 12 of 12 in additional filter chain; firing Filter: 'FilterSecurityInterceptor'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.util.matcher.AntPathRequestMatcher] Checking match of request : '/'; against '/services/employee/*'
16:02:58.371 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.access.intercept.FilterSecurityInterceptor] Public object - authentication not attempted
16:02:58.371 TRACE [http-bio-8080-exec-3][org.springframework.web.context.support.XmlWebApplicationContext] Publishing event in Root WebApplicationContext: org.springframework.security.access.event.PublicInvocationEvent[source=FilterInvocation: URL: /]
16:02:58.372 DEBUG [http-bio-8080-exec-3][org.springframework.beans.factory.support.DefaultListableBeanFactory] Returning cached instance of singleton bean 'org.springframework.integration.internalMessagingAnnotationPostProcessor'
16:02:58.372 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.FilterChainProxy] / reached end of additional filter chain; proceeding with original chain
16:02:58.375 TRACE [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] Bound request context to thread: SecurityContextHolderAwareRequestWrapper[ org.springframework.security.web.context.HttpSessionSecurityContextRepository$Servlet3SaveToSessionRequestWrapper#1b0b34e]
16:02:58.376 DEBUG [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] DispatcherServlet with name 'Information Exchange Gateway Integration' processing GET request for [/DummyDataIntg/]
16:02:58.376 TRACE [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] Testing handler map [org.springframework.integration.http.inbound.IntegrationRequestMappingHandlerMapping#cdca7] in DispatcherServlet with name 'Information Exchange Gateway Integration'
16:02:58.378 WARN [http-bio-8080-exec-3][org.springframework.web.servlet.PageNotFound] No mapping found for HTTP request with URI [/DummyDataIntg/] in DispatcherServlet with name 'Information Exchange Gateway Integration'
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.context.HttpSessionSecurityContextRepository] SecurityContext is empty or contents are anonymous - context will not be stored in HttpSession.
16:02:58.378 TRACE [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] Cleared thread-bound request context: SecurityContextHolderAwareRequestWrapper[ org.springframework.security.web.context.HttpSessionSecurityContextRepository$Servlet3SaveToSessionRequestWrapper#1b0b34e]
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.web.servlet.DispatcherServlet] Successfully completed request
16:02:58.378 TRACE [http-bio-8080-exec-3][org.springframework.web.context.support.XmlWebApplicationContext] Publishing event in WebApplicationContext for namespace 'Information Exchange Gateway Integration-servlet': ServletRequestHandledEvent: url=[/DummyDataIntg/]; client=[127.0.0.1]; method=[GET]; servlet=[Information Exchange Gateway Integration]; session=[null]; user=[null]; time=[6ms]; status=[OK]
16:02:58.378 TRACE [http-bio-8080-exec-3][org.springframework.web.context.support.XmlWebApplicationContext] Publishing event in Root WebApplicationContext: ServletRequestHandledEvent: url=[/DummyDataIntg/]; client=[127.0.0.1]; method=[GET]; servlet=[Information Exchange Gateway Integration]; session=[null]; user=[null]; time=[6ms]; status=[OK]
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.beans.factory.support.DefaultListableBeanFactory] Returning cached instance of singleton bean 'org.springframework.integration.internalMessagingAnnotationPostProcessor'
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.access.ExceptionTranslationFilter] Chain processed normally
16:02:58.378 DEBUG [http-bio-8080-exec-3][org.springframework.security.web.context.SecurityContextPersistenceFilter] SecurityContextHolder now cleared, as request processing completed
16:04:37.403 INFO [task-scheduler-4][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\b.txt, headers={timestamp=1420540477403, id=f8c32928-411b-99b7-f4a0-0dd1b119fc44}]]
16:04:37.403 ERROR [task-scheduler-4][org.springframework.integration.handler.LoggingHandler] \target\foo\b.txt
16:06:17.403 INFO [task-scheduler-9][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\brjb.txt, headers={timestamp=1420540577403, id=061634bf-0562-962b-e583-53a302cdb0d4}]]
16:06:17.403 ERROR [task-scheduler-9][org.springframework.integration.handler.LoggingHandler] \target\foo\brjb.txt
16:07:57.403 INFO [task-scheduler-10][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\d.txt, headers={timestamp=1420540677403, id=abea1188-fc5a-15b0-c40c-73ea686a88c0}]]
16:07:57.403 ERROR [task-scheduler-10][org.springframework.integration.handler.LoggingHandler] \target\foo\d.txt
16:09:37.403 INFO [task-scheduler-4][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\g.txt, headers={timestamp=1420540777403, id=461a8e48-6ebf-5d1c-ab3f-ce28e54e00b8}]]
16:09:37.403 ERROR [task-scheduler-4][org.springframework.integration.handler.LoggingHandler] \target\foo\g.txt
16:11:17.403 INFO [task-scheduler-3][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\h.txt, headers={timestamp=1420540877403, id=4cbeeceb-5949-0e1c-1492-45ec17172480}]]
16:11:17.403 ERROR [task-scheduler-3][org.springframework.integration.handler.LoggingHandler] \target\foo\h.txt
16:12:57.403 INFO [task-scheduler-9][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\p.txt, headers={timestamp=1420540977403, id=3767b4bb-4de4-3178-f693-5ac0bd94a766}]]
16:12:57.403 ERROR [task-scheduler-9][org.springframework.integration.handler.LoggingHandler] \target\foo\p.txt
16:14:37.403 INFO [task-scheduler-6][org.springframework.integration.file.FileReadingMessageSource] Created message: [GenericMessage [payload=\target\foo\wiki.txt, headers={timestamp=1420541077403, id=bd3d0082-9788-fc31-1596-ab7a79186c17}]]
16:14:37.403 ERROR [task-scheduler-6][org.springframework.integration.handler.LoggingHandler] \target\foo\wiki.txt
error log in java file**strong text**
20:25:52.807 ERROR [task-scheduler-1][org.springframework.integration.handler.LoggingHandler] org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory; nested exception is org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: Failed to list files; nested exception is 2: No such file
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:209)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:167)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:57)
at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:64)
at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:124)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:192)
at org.springframework.integration.endpoint.AbstractPollingEndpoint.access$000(AbstractPollingEndpoint.java:55)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:149)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$1.call(AbstractPollingEndpoint.java:146)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller$1.run(AbstractPollingEndpoint.java:298)
at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:49)
at org.springframework.integration.endpoint.AbstractPollingEndpoint$Poller.run(AbstractPollingEndpoint.java:292)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: Failed to list files; nested exception is 2: No such file
at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:343)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:167)
... 22 more
Caused by: org.springframework.core.NestedIOException: Failed to list files; nested exception is 2: No such file
at org.springframework.integration.sftp.session.SftpSession.list(SftpSession.java:103)
at org.springframework.integration.sftp.session.SftpSession.list(SftpSession.java:50)
at org.springframework.integration.file.remote.session.CachingSessionFactory$CachedSession.list(CachingSessionFactory.java:205)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:171)
at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer$1.doInSession(AbstractInboundFileSynchronizer.java:167)
at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:334)
... 23 more
Caused by: 2: No such file
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2198)
at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:2215)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1565)
at com.jcraft.jsch.ChannelSftp.ls(ChannelSftp.java:1526)
at org.springframework.integration.sftp.session.SftpSession.list(SftpSession.java:91)
... 28 more
You need to take a look how to start Spring Context from Web application using web.xml or WebApplicationInitializer for Servlet 3 environment. In this case the SftpInboundReceive-context.xml can be a part of common ApplicationContext and the polling facility (<int-sftp:inbound-channel-adapter>) will start automatically on application startup, which is caused on server start, when the last one see the web context of your application.
Please, read more docs for Spring Framework: http://projects.spring.io/spring-framework/
Spring Integration is just an EIP extension and follow with the same configuration and lifecycle rules.
I see that you just use the SFTP sample from Spring Intregration. You can found there samples for Tomcat and for Spring Boot as well.

Resources