how to connect to Cassandra at application start up - cassandra

I have a Play application which need to connect to Cassandra. I am using Datastax's driver to connect to Cassandra.
I am able to connect to the db from a controller. The code snippet is (full code is from http://manuel.kiessling.net/setting-up-a-scala-sbt-multi-project-with-cassandra-connectivity-and-migrations
val cluster = new Cluster.Builder().
addContactPoints(uri.hosts.toArray: _*).
withPort(uri.port).
withQueryOptions(new QueryOptions().setConsistencyLevel(defaultConsistencyLevel)).build
val session = cluster.connect
session.execute(s"USE ${uri.keyspace}")
session
I am using the above code in a controller as follows:
class UserController #Inject()(cc: ControllerComponents)(implicit exec: ExecutionContext) extends AbstractController(cc){
def addUser = Action.async{ implicit request => {
println("addUser controller called")
println("testing database connection")
val uri = CassandraConnectionUri("cassandra://localhost:9042/killrvideo")
println(s"got uri object ${uri.host}, ${uri.hosts}, ${uri.port}, ${uri.keyspace}")
val session = Helper.createSessionAndInitKeyspace(uri)
val resultSet = session.execute(s"select * from users")
val row = resultSet.one()
println("got row ",row)
val user = User(UUID.randomUUID(),UserProfile(true,Some("m#m.com"),Some("m"),Some("c")))
...
}
Though the code works, I suppose I shouldn't be connecting to the database from within a controller. I should connect to the database when the play application starts and inject the connection in the controller. But I don't know how to do this. Is this the right way to create a database application in Play?

Short description:
It's not a good practice to connect C* from controller class. It is encouraged to have a separate repository/storage class while accessing DB. You will create a DB accessing class and inject that class to your controller class's constructor.
Here is an open-source sample application what I followed to create my own Cassandra application. Play-Framework-Cassandra-Example. You can follow this project.
Long description:
Here are some basic concepts how to do it:
Step 1:
Define DB configuration in application.conf file:
db {
keyspace = "persons"
table = "person_info"
preparedStatementCacheSize = 100
session {
contactPoints = ["127.0.0.1"]
queryOptions {
consistencyLevel = "LOCAL_QUORUM"
}
}
}
step 2: create a Singleton class to main the connection with Cassandra DB
class CassandraConnectionProvider #Inject()(config: Configuration) extends Provider[CassandraConnection] {
override def get(): CassandraConnection = {
val hosts = config.getStringList("db.session.contactPoints")
val keyspace = config.getString("db.keyspace")
// Use the Cluster Builder if you need to add username/password and handle SSL or tweak the connection
ContactPoints(hosts.asScala).keySpace(keyspace)
}
}
Step 3: Now create a repository class where you can operate CRUD operation into DB.
class PhantomPersonRepository #Inject()(config: Configuration, connection: CassandraConnection, ec: ExecutionContext)
extends CassandraTable[PhantomPersonRepository, Person] with PersonRepository[Future] {
// See https://github.com/outworkers/phantom/wiki/Using-the-Database-class-and-understanding-connectors
implicit val session: Session = connection.session
implicit val keySpace: KeySpace = connection.provider.space
override val tableName: String = config.getString("db.table").getOrElse("person_info")
implicit val executionContext: ExecutionContext = ec
object id extends UUIDColumn(this) with PartitionKey
object firstName extends StringColumn(this) {
override def name: String = "first_name"
}
object lastName extends StringColumn(this) {
override def name: String = "last_name"
}
object studentId extends StringColumn(this) {
override def name: String = "student_id"
}
object gender extends EnumColumn[Gender.Value](this)
override implicit val monad: Monad[Future] = cats.instances.future.catsStdInstancesForFuture
override def create(person: Person): Future[Person] =
insert.value(_.id, person.id)
.value(_.firstName, person.firstName)
.value(_.lastName, person.lastName)
.value(_.studentId, person.studentId)
.value(_.gender, person.gender)
.consistencyLevel_=(ConsistencyLevel.LOCAL_QUORUM)
.future()
.map(_ => person)
// https://github.com/outworkers/phantom/wiki/Querying#query-api
override def find(personId: UUID): Future[Option[Person]] =
select.where(_.id eqs personId)
.consistencyLevel_=(ConsistencyLevel.LOCAL_QUORUM)
.one()
override def update(person: Person): Future[Person] = create(person)
.....
Step 4: Now Inject this repository classes to your Controller class and access DB:
#Singleton
class PersonController #Inject()(personRepo: PersonRepository[Future])(implicit ec: ExecutionContext) extends Controller {
def create: Action[JsValue] = Action.async(parse.json) { request =>
onValidationSuccess[CreatePerson](request.body) { createPerson =>
val person = Person(UUID.nameUUIDFromBytes(createPerson.studentId.getBytes()), createPerson.firstName,
createPerson.lastName, createPerson.studentId, createPerson.gender.toModel)
personRepo.find(person.id).flatMap {
case None => personRepo.create(person).map(createdPerson => Created(createdPerson.toJson))
case Some(existing) => Future.successful(Conflict(existing.toJson))
}.recover { case _ => ServiceUnavailable }
}
}
.....
Hope this helps. All code credits to calvinlfer

Related

Update only fields that are not empty in Cassandra Sink

I'm trying to receive messages from Kafka to update a Cassandra Database using Flink
Messages are like
case class Message(userId: String, info: Info)
case class Info(property1: Option[Int], property2: Option[Int])
I'm using json4s to parse the Kafka message and extracting it to a DataStream[Message]
val kafkaSource: KafkaSource[String] = KafkaSource.builder()
.setBootstrapServers("localhost:29092")
.setTopics("my-topic")
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new SimpleStringSchema())
.build()
Now, I just want to update the fields that are not None, like
{
"user_id": "abc-123",
"info":
{
"property1": 1
}
}
Will generate a case class like:
Message(abc-123, Info(Some(1), None))
How can I make a CassandraSink to be able to update just this property for user abc-123?
I was trying to use something like:
CassandraSink
.addSink(dataStream)
.setClusterBuilder(new ClusterBuilder() {
override def buildCluster(builder: Cluster.Builder): Cluster = {
builder
.addContactPoint("127.0.0.1")
.withPort(29042)
.withCredentials("cassandra", "cassandra")
.build()
}
})
.setQuery("UPDATE user_info SET property1 = ?, property2 = ? WHERE id = ?")
.build()
and trying to manipulate the query outside CassandraSink builder, but it wasn't possible.
Is there any way to just update fields who aren't None?
May be you can implement RichMapFunction, and update fields who aren't None in the map interface
public class MyMapFunction extends RichMapFunction<T> {
#Override
public void open(Configuration parameters) throws Exception {
}
#Override
public T map(...) throws Exception {
return xxx;
}
}

Room cannot verify the data integrity and Crashes on Empty Data

I am working on Android Application in Which I am getting specific Data from Room Database by specific path in the Storage. My App Got Crashes as It does not have Any Data in the Storage and the Logcat gives me this..
java.lang.IllegalStateException: Room cannot verify the data integrity. Looks like you've changed schema but forgot to update the version number. You can simply fix this by increasing the version number.
at androidx.room.RoomOpenHelper.checkIdentity(RoomOpenHelper.java:154)
at androidx.room.RoomOpenHelper.onOpen(RoomOpenHelper.java:135)
at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper$OpenHelper.onOpen(FrameworkSQLiteOpenHelper.java:195)
at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked(SQLiteOpenHelper.java:428)
at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:317)
at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper$OpenHelper.getWritableSupportDatabase(FrameworkSQLiteOpenHelper.java:145)
at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper.getWritableDatabase(FrameworkSQLiteOpenHelper.java:106)
at androidx.room.RoomDatabase.inTransaction(RoomDatabase.java:476)
at androidx.room.RoomDatabase.assertNotSuspendingTransaction(RoomDatabase.java:281)
at com.maximus.technologies.views.activities.scanneddatabase.TodoDaoScanned_Impl.getAllScan(TodoDaoScanned_Impl.java:152)
at com.maximus.technologies.views.fragments.scanhistorypackage.QRRetrievingScanClassPresenter$getAllDatFromDatabase$1.invokeSuspend(QRRetrievingScanClassPresenter.kt:29)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:241)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594)
The Above Error or crash Only occurs as the app dont have any data in Storage. But as I put a Data the Crash Problem Get Resolved.
I am not able to Understand what the Problem actually is...
Here is My Room Database Class..
#Database(
entities = [TodoEntity::class,TodoEntityScanned::class],
version = 1)
abstract class AppDatabase : RoomDatabase() {
abstract fun TodoDao(): TodoDao
abstract fun TodoDaoScanned(): TodoDaoScanned
object DatabaseBuilder {
private var INSTANCE: AppDatabase? = null
fun getInstance(context: Context): AppDatabase {
if (INSTANCE == null) {
synchronized(AppDatabase::class) {
INSTANCE = buildRoomDB(context)
}
}
return INSTANCE!!
}
private fun buildRoomDB(context: Context) =
Room.databaseBuilder(
context.applicationContext,
AppDatabase::class.java,
"mindorks-example-coroutines"
).build()
}
}
Room Database Retrieving Interface where app Crashes on getall()
override fun getAllDatFromDatabase(appDatabasescanned: AppDatabase) {
var list = listOf<TodoEntityScanned>()
try {
GlobalScope.launch(Dispatchers.Default) {
list = appDatabasescanned.TodoDaoScanned().getAllScan()
Log.d("hello","hello")
mView.showAllData(list)
}
}
catch (e:Exception){
Log.d("get hello",e.toString())
}
}
The getAll lies in Dao Class
interface TodoDao {
#Query("SELECT * FROM tablefilepaths")
fun getAll(): List<TodoEntity>
#Query("SELECT * FROM tablefilepaths WHERE imagespath LIKE :title")
fun findByTitle(title: String): TodoEntity
#Insert
fun insertpaths(todo: TodoEntity)
#Delete
fun deletepaths(todo: TodoEntity)
#Query("DELETE FROM tablefilepaths WHERE id = :noteId")
fun deleteNoteById(noteId: Int)
#Update
fun updateTodo(vararg todos: TodoEntity)}
Here is My Fragment Class Where I am Setting data in RecyclerView
override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
super.onViewCreated(view, savedInstanceState)
recyclerviewcreatehistory?.layoutManager = LinearLayoutManager(context)
recyclerviewcreatehistory?.setHasFixedSize(true)
filefetch()
customAdaptercreatehistory = CustomAdapterCreateHistory(this.context ?: return, charItemcreate!!,this)
recyclerviewcreatehistory?.adapter = customAdaptercreatehistory
}
fun filefetch() {
val noteDatabase: AppDatabase = AppDatabase.DatabaseBuilder.getInstance(requireContext())
retrivingpresenter = QRRetrievingClassPresenter(this)
retrivingpresenter!!.getAllDatFromDatabase(noteDatabase)
}
override fun showAllData(note_list: List<TodoEntity>) {
if (note_list is ArrayList<*>) {
val arraylist = note_list as ArrayList<TodoEntity>
charItemcreate=arraylist
}
if (charItemcreate.isEmpty()){
}else{
customAdaptercreatehistory?.updateUsers(note_list as ArrayList<TodoEntity>)
customAdaptercreatehistory?.notifyDataSetChanged()
// Log.d("hello", note_list[0].imagesPathData)
}
}
You have to do some checks in your getAllDatFromDatabase() inside your coroutine. I guess list variable equals null or something like that. You should check if there are any files and if not you need to put there something else.

thread error while using realm in MVVM architecture

I'm using the Realm database in my kotlin project.
I use MVVM architecture so I created a repository class which contains codes bellow:
class DatabaseRepository {
private val database = Realm.getDefaultInstance()
fun addOrUpdateUser(user: JSONObject) {
database.executeTransactionAsync{
database.createOrUpdateObjectFromJson(UserModel::class.java, user)
}
}
}
also, I have created my ViewModel class like this:
class DatabaseViewModel(private val database:DatabaseRepository) : ViewModel() {
fun addUser(information: JSONObject) {
database.addOrUpdateUser(information)
}
}
and my View ModelFactory class is like this:
class ViewModelFactory(private val databaseRepository:DatabaseRepository):ViewModelProvider.Factory {
override fun <T : ViewModel?> create(modelClass: Class<T>): T {
return modelClass.getConstructor(DatabaseRepository::class.java).newInstance(databaseRepository)
}
}
so I create the instance of them in my MainActivity like this:
val databaseRepo=DatabaseRepository()
val factory=ViewModelFactory(databaseRepo)
database = ViewModelProviders.of(this,factory).get(DatabaseViewModel::class.java)
the problem is that while I'm trying to add some data to Database using 'addUser' function in ViewModel class I get this error:
Realm objects can only be accessed on the thread they were created.
what have I done wrong?
the only problem was because of executeTransactionAsync, so I changed from:
executeTransactionAsync
to
executeTransaction
and it worked!

Deadlocks with Play 2.5 scala, deadbolt 2.5 and MongoDB/Morphia

Basically all I want to do is getting all users from my database, which worked fine until the very moment i wanted to use deadbolt for it:
I think the 4 Threads(number of processors) of the fork-join-executor are already all used and then there is somekind of deadlock.
Things I tried:
Raise the number of threads the executor has, so however play/akka ignores my settings
Define another execution context for the futures in the controller, but this does not prevent deadlocks since more than four threads still wait at each other
use a thread-pool-executor, but my settings are ignored
A mixed scala/java code from here:
class UserController {
def getUserList = deadbolt.Restrict(List(Array("Admin")))(){ implicit request =>
Future {
val users = userModel.list
val json = Json.toJson(users)
Ok(json.toString)
}(
}
}
The User Model is essentially nothing more than:
public class UserModel {
private MongoClient client = new MongoClient();
private Morphia morphia = new Morphia();
protected Datastore datastore = morphia.createDatastore(client, "timetracking");
public List<User> list(){
return datastore.find(User.class).asList();
}
public User findUserByName(String name){
User found = datastore.createQuery(User.class).field("username").equal(name).get();
return found;
}
}
Authorization Handler:
class AuthorizationHandler extends DeadboltHandler {
val model = new UserModel
override def getSubject[A](request: AuthenticatedRequest[A]): Future[Option[Subject]] =
Future {
blocking {
request.subject match {
case Some(user) =>
request.subject
case None =>
val username = request.session.get("username")
if (username.isDefined) {
val user = model.findUserByName(username.get)
if (user == null) {
None
} else {
val subject = new ScalaSubject(user.getUsername, user.getRole)
Some(subject)
}
} else {
None
}
}
}
}
Defining a seperate deadbolt context does not help:
package deadbolt.scala
import be.objectify.deadbolt.scala.DeadboltExecutionContextProvider
import be.objectify.deadbolt.scala.cache.HandlerCache
import play.api.inject.{Binding, Module}
import play.api.{Configuration, Environment}
class DeadBoldModule extends Module {
override def bindings(environment: Environment,
configuration: Configuration): Seq[Binding[_]] = Seq(
bind[HandlerCache].to[TimeTrackerHandelCache],
bind[DeadboltExecutionContextProvider].to[ThreadPoolProvider]
)
}
Custom context provider:
package deadbolt.scala
import java.io.InvalidObjectException
import java.util.concurrent.Executors
import be.objectify.deadbolt.scala.DeadboltExecutionContextProvider
import scala.concurrent.ExecutionContext
class ThreadPoolProvider extends DeadboltExecutionContextProvider {
override def get(): ExecutionContext = ExecutionContext.fromExecutor(Executors.newFixedThreadPool(100))
}
When I try this, throwing some random exception, it is never thrown:
package deadbolt.scala
import java.io.InvalidObjectException
import java.util.concurrent.Executors
import be.objectify.deadbolt.scala.DeadboltExecutionContextProvider
import scala.concurrent.ExecutionContext
class ThreadPoolProvider extends DeadboltExecutionContextProvider {
override def get(): ExecutionContext = throw new IllegalAccessError("asd");ExecutionContext.fromExecutor(Executors.newFixedThreadPool(100))
}
It was not deadbolts fault, but however the MongoClient opened a new thread when it was instantiated.Which happened in our project quite often, but was not closed properly thereby blocking the threadpool. We used a Singleton and everything worked fine.

slick schema extension with activeslick

I have defined Person and Org (organization) schemas as
trait Schema { this: Tables with TableQueries with Profile =>
import jdbcDriver.simple._
class PersonsTable(tag: Tag) extends EntityTable[Player](tag, "PERSON") { def id = column[Int]("ID", O.PrimaryKey, O.AutoInc)
def dateCreated = column[Date]("DATE_CREATED")
def firstName = column[String]("FIRST_NAME")
def lastName = column[String]("LAST_NAME")}
class OrgTable(tag: Tag) extends EntityTable[Player](tag, "ORG") {
def id = column[Int]("ID", O.PrimaryKey, O.AutoInc)
def dateCreated = column[Date]("DATE_CREATED")
def = column[String]("NAME")
}
}
I soon realized that the schemas shared common fields like id (primary key) and dateCreated. So, I grouped them together so I don't have to repeat them,
trait Schema { this: Tables with TableQueries with Profile =>
import jdbcDriver.simple._
class PersonsTable(tag: Tag) extends EntityTable[Player](tag, "PERSON") with CommonSchema#CommonFields { def firstName = column[String]("FIRST_NAME")
def lastName = column[String]("LAST_NAME")}
class OrgTable(tag: Tag) extends EntityTable[Player](tag, "ORG") with CommonSchema#CommonFields {
def = column[String]("NAME")
}
}
trait CommonSchema { this: Tables with TableQueries with Profile =>
import jdbcDriver.simple._
trait CommonFields{this: Table[_] =>
def id = column[Int]("ID", O.PrimaryKey, O.AutoInc)
def dateCreated = column[Date]("DATE_CREATED")
}
}
I could not get this to compile. The compiler complained "CommonSchema is not a legal prefix for a constructor". How do I get it to work? I am using ActiveSlick with this setup and I am using a generic jdbcDriver for these classes as a place holder until the final moment when I run this classes with a specific Slick driver i.e. scala.slick.driver.H2Driver.simple._. Thanks

Resources