I have multiple Cassandra clusters. Each cluster has specific set of contact points. Each cluster has separate set of tables/CF.
In my C* client I am supposed to query both clusters. I am using spring-boot version of the Cassandra. I am trying to use CassandraOperations to do the queries. How do I go about doing this?
#Bean
public CassandraOperations cassandraTemplate(Session sessionA) throws Exception {
return new CassandraTemplate(sessionA);
}
#Bean
public CassandraMappingContext mappingContext() {
return new BasicCassandraMappingContext();
}
#Bean
public CassandraConverter converter() {
return new MappingCassandraConverter(mappingContext());
}
Above is a example whereby I setup cassandraoperations using sessionA, how about doing the same for sessionB?
At any given time base don the query, it can go to either sessionA or sessionB.
Any pointers are appreciated
Thanks
Found a way to do this by creating separate classes for each cluster manager.
#Component
#Primary
public class CassandraTemplateA extends CassandraTemplate{
#Autowired
public CassandraTemplateA(CassandraConverter converter) {
super(sessionA, converter);
}
}
#Component
#Primary
public class CassandraTemplateB extends CassandraTemplate{
#Autowired
public CassandraTemplateB(CassandraConverter converter) {
super(sessionB, converter);
}
}
Related
I am new to spring integration, I have requirements similar to the below, data is being read from multiple tables. I am trying to implement jdbcpollingchanneladapter and jdbcMessageHandler, can anyone help with an example using java DSL?
Read the list of people
For each person get a list of departments
For each department get a list of managers
For each manager get the list of people
b. process the people information
c. write the information into a csv
I am using MessageSource to before DB queries and mapping result values to POJO by using a Mapper. How do I use the results as lookup values to perform another query.
#EnableIntegration
public class IntegrationConfig {
#Autowired
DataSource dataSource;
#Autowired
PlatformTransactionManager transactionManager;
#Bean
public MessageChannel FamilyChannel() {
return new DirectChannel();
}
#Bean
#InboundChannelAdapter(value = "FamilyChannel", poller = #Poller(fixedDelay = "1000"))
public MessageSource<Object> familyNumberMessageSource() {
JdbcPollingChannelAdapter jdbcPollingChannelAdapter = new JdbcPollingChannelAdapter(dataSource, NCFBulkOrderQuery.SELECT_NCF_BULK_ORDER_DATE);
return jdbcPollingChannelAdapter;
}
#Bean
public MessageHandler familyNumberHandler(){
JdbcMessageHandler jdbcMessageHandler = new JdbcMessageHandler(dataSource, NCFBulkOrderQuery.SELECT_NCF_BULK_ORDER_DATE);
return jdbcMessageHandler;
}
#Bean
public IntegrationFlow jdbFlow() {
return IntegrationFlows
.from("FamilyChannel")
.handle(familyNumberMessageSource(), "receive")
.get();
}
}
Spring Integration has ZooKeeper support as documented in https://docs.spring.io/spring-integration/reference/html/zookeeper.html
However this document is so vague.
It suggests adding below bean but does not give details on how to start/stop a poller when the node is granted leadership.
#Bean
public LeaderInitiatorFactoryBean leaderInitiator(CuratorFramework client) {
return new LeaderInitiatorFactoryBean()
.setClient(client)
.setPath("/siTest/")
.setRole("cluster");
}
Do we have any example on how to ensure below poller is run only once in a cluster at any time using zookeeper?
#Component
public class EventsPoller {
public void pullEvents() {
//pull events should be run by only one node in the cluster at any time
}
}
The LeaderInitiator emits an OnGrantedEvent and OnRevokedEvent, when it becomes leader and its leadership is revoked.
See https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#endpoint-roles and the next https://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#leadership-event-handling for more info about those events handling and how it affects your components in the particular role.
Although I agree that Zookkeper chapter must have some link to that SmartLifecycleRoleController chapter. Feel free to raise a JIRA on the matter and contribution is welcome!
UPDATE
This is what I did in our test:
#RunWith(SpringRunner.class)
#DirtiesContext
public class LeaderInitiatorFactoryBeanTests extends ZookeeperTestSupport {
private static CuratorFramework client;
#Autowired
private PollableChannel stringsChannel;
#BeforeClass
public static void getClient() throws Exception {
client = createNewClient();
}
#AfterClass
public static void closeClient() {
if (client != null) {
client.close();
}
}
#Test
public void test() {
assertNotNull(this.stringsChannel.receive(10_000));
}
#Configuration
#EnableIntegration
public static class Config {
#Bean
public LeaderInitiatorFactoryBean leaderInitiator(CuratorFramework client) {
return new LeaderInitiatorFactoryBean()
.setClient(client)
.setPath("/siTest/")
.setRole("foo");
}
#Bean
public CuratorFramework client() {
return LeaderInitiatorFactoryBeanTests.client;
}
#Bean
#InboundChannelAdapter(channel = "stringsChannel", autoStartup = "false", poller = #Poller(fixedDelay = "100"))
#Role("foo")
public Supplier<String> inboundChannelAdapter() {
return () -> "foo";
}
#Bean
public PollableChannel stringsChannel() {
return new QueueChannel();
}
}
}
And I have in logs something like this:
2018-12-14 10:12:33,542 DEBUG [Curator-LeaderSelector-0] [org.springframework.integration.support.SmartLifecycleRoleController] - Starting [leaderInitiatorFactoryBeanTests.Config.inboundChannelAdapter.inboundChannelAdapter] in role foo
2018-12-14 10:12:33,578 DEBUG [Curator-LeaderSelector-0] [org.springframework.integration.support.SmartLifecycleRoleController] - Stopping [leaderInitiatorFactoryBeanTests.Config.inboundChannelAdapter.inboundChannelAdapter] in role foo
I am using Cassandra as a datasource in my Spring boot application and would like to initialize the database before the application starts.
Up to now what I have done is, I have defined a class "CassandraConfiguration" extending "AbstractCassandraConfiguration" class as in the examples which you can see below and I have a repository extending "CassandraRepository". When I create the keyspace and the table myself, the application works fine.
However, I want to create the keyspace and tables automatically while application is starting. In order to do that, I supplied a schema.cql file under resources folder but I could not make that script work.
Does anyone have any idea what can I do to create the keyspace(s) and tables automatically?
Thanks.
Edit: I am using Cassandra 2.0.9, spring-boot 1.3.2.RELEASE and datastax cassandra driver 2.1.6 versions.
CassandraConfiguration.java
#Configuration
#PropertySource(value = { "classpath:cassandra.properties" })
#EnableCassandraRepositories(basePackages = { "bla.bla.bla.repository" })
public class CassandraConfiguration extends AbstractCassandraConfiguration {
#Autowired
private Environment environment;
#Bean
public CassandraClusterFactoryBean cluster() {
CassandraClusterFactoryBean cluster = new CassandraClusterFactoryBean();
cluster.setContactPoints( environment.getProperty( "cassandra.contactpoints" ) );
cluster.setPort( Integer.parseInt( environment.getProperty( "cassandra.port" ) ) );
return cluster;
}
#Bean
public CassandraMappingContext cassandraMapping() throws ClassNotFoundException {
return new BasicCassandraMappingContext();
}
#Bean
public CassandraConverter converter() throws ClassNotFoundException {
return new MappingCassandraConverter(cassandraMapping());
}
#Override
protected String getKeyspaceName() {
return environment.getProperty( "cassandra.keyspace" );
}
#Bean
public CassandraSessionFactoryBean session() throws Exception {
CassandraSessionFactoryBean session = new CassandraSessionFactoryBean();
session.setCluster(cluster().getObject());
session.setKeyspaceName(environment.getProperty("cassandra.keyspace"));
session.setConverter(converter());
session.setSchemaAction(SchemaAction.NONE);
return session;
}
#Override
public SchemaAction getSchemaAction() {
return SchemaAction.RECREATE_DROP_UNUSED;
}
}
If you are still having problems with this, in Spring Boot 2 and SD Cassandra 2.0.3 you can do this straightforward Java configuration and setup everything out of the box.
#Configuration
#EnableCassandraRepositories(basePackages = "com.example.repository")
public class DbConfigAutoStart extends AbstractCassandraConfiguration {
/*
* Provide a contact point to the configuration.
*/
#Override
public String getContactPoints() {
return "exampleContactPointsUrl";
}
/*
* Provide a keyspace name to the configuration.
*/
#Override
public String getKeyspaceName() {
return "exampleKeyspace";
}
/*
* Automatically creates a Keyspace if it doesn't exist
*/
#Override
protected List<CreateKeyspaceSpecification> getKeyspaceCreations() {
CreateKeyspaceSpecification specification = CreateKeyspaceSpecification
.createKeyspace("exampleKeyspace").ifNotExists()
.with(KeyspaceOption.DURABLE_WRITES, true).withSimpleReplication();
return Arrays.asList(specification);
}
/*
* Automatically configure a table if doesn't exist
*/
#Override
public SchemaAction getSchemaAction() {
return SchemaAction.CREATE_IF_NOT_EXISTS;
}
/*
* Get the entity package (where the entity class has the #Table annotation)
*/
#Override
public String[] getEntityBasePackages() {
return new String[] { "com.example.entity" };
}
And you are good to go
Your return type BasicCassandraMappingContext() might be deprecated. Use
#Bean
public CassandraMappingContext mappingContext() throws ClassNotFoundException {
CassandraMappingContext mappingContext= new CassandraMappingContext();
mappingContext.setInitialEntitySet(getInitialEntitySet());
return mappingContext;
}
#Override
public String[] getEntityBasePackages() {
return new String[]{"base-package name of all your entity, annotated
with #Table"};
}
#Override
protected Set<Class<?>> getInitialEntitySet() throws ClassNotFoundException {
return CassandraEntityClassScanner.scan(getEntityBasePackages());
}
Instead of,
#Bean
public CassandraMappingContext cassandraMapping() throws ClassNotFoundException {
return new BasicCassandraMappingContext();
}
also set:
session.setSchemaAction(SchemaAction.RECREATE_DROP_UNUSED);
and Exclude:
#Override
public SchemaAction getSchemaAction() {
return SchemaAction.RECREATE_DROP_UNUSED;
}
get reference here.
I'm working with spring-boot 1.5.10.RELEASE and cassandra 3.0.16 but you can try downscaling the versions. To create the keyspace you can import the keyspacename from you application.yml or application.properties. Using the #Table annotation your tables should be generated automatically provided you have set the entity base package.
#Value("${cassandra.keyspace}")
private String keySpace;
#Override
public String[] getEntityBasePackages() {
return new String[]{"com.example.your.entities"};
}
#Override
protected List<CreateKeyspaceSpecification> getKeyspaceCreations() {
return Arrays.asList(
CreateKeyspaceSpecification.createKeyspace()
.name(keySpace)
.ifNotExists()
);
}
Finally i got it working by adding setKeyspaceCreations(getKeyspaceCreations()) to the CassandraClusterFactoryBean Override and also make sure to enable #ComponentScan.
import com.datastax.driver.core.PlainTextAuthProvider;
import com.datastax.driver.core.policies.ConstantReconnectionPolicy;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.cassandra.config.*;
import org.springframework.data.cassandra.core.cql.keyspace.CreateKeyspaceSpecification;
import org.springframework.data.cassandra.core.cql.keyspace.DropKeyspaceSpecification;
import org.springframework.data.cassandra.core.cql.keyspace.KeyspaceOption;
import org.springframework.data.cassandra.repository.config.EnableReactiveCassandraRepositories;
import java.util.Arrays;
import java.util.List;
#Configuration
#EnableReactiveCassandraRepositories(basePackages = "com.company.domain.data")
public class CassandraConfig extends AbstractReactiveCassandraConfiguration{
#Value("${spring.data.cassandra.contactpoints}") private String contactPoints;
#Value("${spring.data.cassandra.port}") private int port;
#Value("${spring.data.cassandra.keyspace-name}") private String keyspace;
#Value("${spring.data.cassandra.username}") private String userName;
#Value("${spring.data.cassandra.password}") private String password;
#Value("${cassandra.basepackages}") private String basePackages;
#Override protected String getKeyspaceName() {
return keyspace;
}
#Override protected String getContactPoints() {
return contactPoints;
}
#Override protected int getPort() {
return port;
}
#Override public SchemaAction getSchemaAction() {
return SchemaAction.CREATE_IF_NOT_EXISTS;
}
#Override
public String[] getEntityBasePackages() {
return new String[]{"com.company.domain.data"};
}
#Override
public CassandraClusterFactoryBean cluster() {
PlainTextAuthProvider authProvider = new PlainTextAuthProvider(userName, password);
CassandraClusterFactoryBean cluster=new CassandraClusterFactoryBean();
cluster.setJmxReportingEnabled(false);
cluster.setContactPoints(contactPoints);
cluster.setPort(port);
cluster.setAuthProvider(authProvider);
cluster.setKeyspaceCreations(getKeyspaceCreations());
cluster.setReconnectionPolicy(new ConstantReconnectionPolicy(1000));
return cluster;
}
#Override
protected List<CreateKeyspaceSpecification> getKeyspaceCreations() {
CreateKeyspaceSpecification specification = CreateKeyspaceSpecification.createKeyspace(keyspace)
.ifNotExists()
.with(KeyspaceOption.DURABLE_WRITES, true);
return Arrays.asList(specification);
}
#Override
protected List<DropKeyspaceSpecification> getKeyspaceDrops() {
return Arrays.asList(DropKeyspaceSpecification.dropKeyspace(keyspace));
}
}
The previous answers are based on AbstractCassandraConfiguration from spring-data-cassandra. If you use spring-boot then it can auto-configure Cassandra for you and there's no need to extend AbstractCassandraConfiguration. However, even in this case you need to do some work to automatically create the keyspace. I've settled on an auto-configuration added to our company's spring-boot starter, but you can also define it as a regular configuration in your application.
/**
* create the configured keyspace before the first cqlSession is instantiated. This is guaranteed by running this
* autoconfiguration before the spring-boot one.
*/
#ConditionalOnClass(CqlSession.class)
#ConditionalOnProperty(name = "spring.data.cassandra.create-keyspace", havingValue = "true")
#AutoConfigureBefore(CassandraAutoConfiguration.class)
public class CassandraCreateKeyspaceAutoConfiguration {
private static final Logger logger = LoggerFactory.getLogger(CassandraCreateKeyspaceAutoConfiguration.class);
public CassandraCreateKeyspaceAutoConfiguration(CqlSessionBuilder cqlSessionBuilder, CassandraProperties properties) {
// It's OK to mutate cqlSessionBuilder because it has prototype scope.
try (CqlSession session = cqlSessionBuilder.withKeyspace((CqlIdentifier) null).build()) {
logger.info("Creating keyspace {} ...", properties.getKeyspaceName());
session.execute(CreateKeyspaceCqlGenerator.toCql(
CreateKeyspaceSpecification.createKeyspace(properties.getKeyspaceName()).ifNotExists()));
}
}
}
In my case I've also added a configuration property to control the creation, spring.data.cassandra.create-keyspace, you may leave it out if you don't need the flexibility.
Note that spring-boot auto-configuration depends on certain configuration properties, here's what I have in my dev environment:
spring:
data:
cassandra:
keyspace-name: mykeyspace
contact-points: 127.0.0.1
port: 9042
local-datacenter: datacenter1
schema-action: CREATE_IF_NOT_EXISTS
create-keyspace: true
More details: spring-boot and Cassandra
I would like to intercept all spring integration gateways via AOP.
Is it possible to do that? If not what might be best way to do log input object coming to gateway?
#ContextConfiguration
#RunWith(SpringJUnit4ClassRunner.class)
#DirtiesContext
public class AdviceExample {
#Autowired
private TestGateway testGateway;
#Test
public void testIt() {
System.out.println(this.testGateway.testIt("foo"));
}
#MessagingGateway
public interface TestGateway {
#Gateway(requestChannel = "testChannel")
#CustomAnnotation
String testIt(String payload);
}
#Configuration
#EnableIntegration
#IntegrationComponentScan
#EnableMessageHistory
#EnableAspectJAutoProxy
public static class ContextConfiguration {
LoggingHandler logger = new LoggingHandler(LoggingHandler.Level.INFO.name());
#Bean
public IntegrationFlow testFlow() {
return IntegrationFlows.from("testChannel")
.transform("payload.toUpperCase()")
.channel("testChannel")
.transform("payload.concat(' Manoj')")
.channel("testChannel")
.handle(logger)
.get();
}
#Bean
public GatewayAdvice gtwyAdvice(){
return new GatewayAdvice();
}
}
#Retention(value = RetentionPolicy.RUNTIME)
#Target(value = ElementType.METHOD)
#Inherited
public #interface CustomAnnotation{
}
#Aspect
public static class GatewayAdvice {
#Before("execution(* advice.AdviceExample.TestGateway.testIt(*))")
public void beforeAdvice() {
System.out.println("Before advice called...");
}
#Before("#annotation(advice.AdviceExample.CustomAnnotation)")
public void beforeAnnotationAdvice() {
System.out.println("Before annotation advice called...");
}
}
}
Yes, you can do that. Take a look to the standard Spring AOP Framework. Since all those #Gateway are beans in the end you can add for them any Advice by their bean names and for the specific method, if that. For example we often suggest to use #Transactional on gateway's methods. And this is exactly a sample "how to use AOP on integration gateway".
My Spring Data Cassandra configuration looks like this:
#Configuration
#EnableCassandraRepositories(basePackages = {
"mypackage.repository.cassandra",
})
public class DistributedRepositoryConfiguration {
// ...
#Bean
public CassandraSessionFactoryBean session() throws Exception {
CassandraSessionFactoryBean session = new CassandraSessionFactoryBean();
session.setCluster(cluster().getObject());
session.setKeyspaceName(configuration.get().getKeyspace());
session.setConverter(converter());
session.setSchemaAction(SchemaAction.CREATE);
return session;
}
}
Generally, Spring Data Cassandra works in my project. However, when I start my application I have no tables created. Anyone who could tell me what I'm doing wrong?
It is not written well in documentation if you want to have automatic table creation you should tell cassandra where to look for entity classes:
<cassandra:mapping entity-base-packages="your.package" />
If you want to do the same using annotiation configuration you have to explicitly tell CassandraTemplate where to look for it. So
#Bean
public CassandraSessionFactoryBean session() throws Exception {
CassandraSessionFactoryBean session = new CassandraSessionFactoryBean();
session.setCluster(cluster().getObject());
session.setKeyspaceName(keyspaceName);
session.setConverter(converter());
session.setSchemaAction(SchemaAction.CREATE);
return session;
}
#Bean
public CassandraConverter converter() throws Exception {
return new MappingCassandraConverter(mappingContext());
}
#Bean
public CassandraMappingContext mappingContext() throws Exception {
BasicCassandraMappingContext bean = new BasicCassandraMappingContext();
bean.setInitialEntitySet(CassandraEntityClassScanner.scan(("package.with.your.entities")));
return bean;
}
To do it with ease I suggest using AbstractCassandraConfiguration and override methods which You need.
I checked the class AbstractCassandraConfiguration and found the following code:
public String[] getEntityBasePackages() {
return new String[] { getClass().getPackage().getName() };
}
Since my config class isn't in the main package, the component scan does not find my classes with the #Table annotation. So I override the method "getEntityBasePackages()" using my StartUp class and everything worked fine.
This is my config class:
#Configuration
public class CassandraConfig extends AbstractCassandraConfiguration {
#Value("${spring.data.cassandra.keyspace-name}")
private String keyspaceName;
#Override
protected String getKeyspaceName() {
return keyspaceName;
}
#Override
public String[] getEntityBasePackages() {
return new String[]{AppStartup.class.getPackage().getName()};
}
#Override
protected List<CreateKeyspaceSpecification> getKeyspaceCreations() {
return Collections.singletonList(CreateKeyspaceSpecification
.createKeyspace(keyspaceName)
.ifNotExists(true)
.with(KeyspaceOption.DURABLE_WRITES, true)
.withSimpleReplication());
}
#Override
public SchemaAction getSchemaAction() {
return SchemaAction.CREATE_IF_NOT_EXISTS;
}
}
Using this class, your application should create the required keyspace and tables to run.
If you using CassandraDataConfiguration just annotated you base class application with:
#EntityScan("mypackage.repository.cassandra")
The base package information will be used in CassandraDataAutoConfiguration.cassandraMapping method to add this package to cassandra mapping.