I am unable to figure out how to unit-test my class which access Cassandra. I might have to redesign!
I have created a Play Components class (https://www.playframework.com/documentation/2.6.x/api/scala/index.html#play.api.BuiltInComponentsFromContext) which creates a cassandra session at application start up.
trait CassandraRepositoryComponents {
def environment: Environment
def configuration: Configuration
def applicationLifecycle: ApplicationLifecycle
...
lazy private val cassandraSession: Session = {
val cluster = new Cluster.Builder().
addContactPoints(uri.hosts.toArray: _*).
withPort(uri.port).
withQueryOptions(new QueryOptions().setConsistencyLevel(defaultConsistencyLevel)).build
val session = cluster.connect
}
}
The session thus created is passed to my repo class when the repo class is instantiaited
class UsersRepository(utilities:HelperMethods, session: Session,tablename:String)
extends CassandraRepository[UserKeys,User](session, tablename, List("bucket","email")) {
//UsersRepository CLASS DOESN'T USE session DIRECTLY. IT PASSES THE session TO CassandraRepository WHICH EVENTUALLY CALLS session.execute TO RUN QUERIES...}
I want to unit test UsersRepository. I am using embedded-cassandra to test it but it seems embedded-cassandra doesn't provide a way to get instance of the session it creates.
question1 - Is there a way I could get the session of Cassandra started by TestCassandra?
question2 - is there a better way for me to organise the classes?
question1 - Is there a way I could get the session of Cassandra
started by TestCassandra?
There is no way to get the Session of Cassandra started by TestCassandra.
You can use either com.github.nosan.embedded.cassandra.test.ClusterFactory
or com.github.nosan.embedded.cassandra.test.CqlSessionFactory to create Cluster or CqlSession classes.
Related
I have a query listener where I would fetch data from the QueryListener object and send it to an API.
public class QueryListener implements QueryExecutionListener {
#Override
public void onSuccess(String funcName, QueryExecution qe, long durationNs) {
SparkSession sparkSession = qe.sparkSession();
//then fetch data from QueryExecution object
}
}
I want to have an identifier so that I can track which module/class the data belongs to. Say class X creates the spark session, and the listener gets called for it, I want a way to tell (inside the listener) that X was the class for which this listener was called and all the data inside the QueryExecution object belongs to that class.
I am not able to find this particular info from the spark session object. I tried checking methods like conf() but it doesn't have the info. Does anyone knows how to do it using the spark session object?
Also, I can't go with identifier like appName (sparkSession.sparkContext().appName()) because I don't have access to change the calling methods and I see that most of them don't have this parameter set. Hence, I would be getting some default value when using this. That's why I opted to go with class name for it.
Please let me know any solutions for this or any other alternatives which might work.
I used loopback 4 to bootstrap my API application and developed some parts of it.
Now I'm trying to access repositories outside the controllers and tried everything I could but didn't help. None of my searches did too. For example I have a controller in which I can access repo such way.
constructor(
#repository(UserRepository) public userRepository: UserRepository){}
But if it isn't a controller it won't work and I found out I had a wrong understanding about #repository and #inject decorators.
My Case:
I want to run a cron-job to perform an update operation on database every day at a specific time.
The thing is I think I should create a service or something like that to expose database operations so it can be accessible anywhere.
The issue you're trying to tackle comes down to dependency injection. It's described at https://loopback.io/doc/en/lb4/Dependency-injection.html but that article does not explain how to create a separate class that works with dependency injection.
In short, in order to perform the dependency injection through decorators, LoopBack needs to be the one to construct the class you're writing, so that it can pick what you need from its Context. So, instead of writing new ClassSomething, it has to be #inject(ClassSomething) or #service(ClassSomething), and those decorators only work in a class also instantiated by LoopBack. In other words, your class has to be in the "call tree" from the original application class (application.ts) for dependency injection to work.
The easiest way to do what you want to do is to use LoopBack's Cron component: see https://loopback.io/doc/en/lb4/Running-cron-jobs.html. You could convert your class to a CronJob if it has no other purpose, or let the CronJob create an instance of your class. In the latter case, you need to bind your class to the application Context so that the CronJob can access it through decorators. You can do so in two ways:
Convert your class to a Service (see https://loopback.io/doc/en/lb4/Service.html and https://loopback.io/doc/en/lb4/Service-generator.html). If what your class does can be thought of as a service, this is the way to go.
Inject your class with: #service(MyServiceClass) myService: MyServiceClass
Directly bind your class to the application context, in application.ts:
this.bind('MyClass').toClass(MyClass);
Inject your class with: #inject(MyClass) myClass: MyClass
The latter option is not best practice AFAIU as it does not adhere to IoC-principles (https://en.wikipedia.org/wiki/Inversion_of_control) - basically, by hard-coding the class binding in the application class, it is not exactly modular. When your class is converted to a service (and placed in src/services), it will automatically be added to the Context, meaning you can inject it everywhere using #service().
I also experience the same(cron word). I couldn't find any solution still in the documentation. But still, I have achieved this in this way.
just call repository as like class with dependency in it. do this in your index.ts file where you start the application.
async function startCronJobs(app: SawLoopbackApplication) {
const userRepo = app.repository(UserRepository);
const userRepoInstance = await userRepo.getValue(app);
const cron = new CronComponent(userInstance);
//---- Exicute your cron here
new CronJob('0 6 * * *', function () {
cron.sendMorningMail()
}).start();
//---- Exicute your Second Cron
new CronJob('0 8 * * 1', function () {
cron.weeklyAppraisalsToAgent()
}).start();
}
call your cron in the functional component and execute a raw query to get the details from DB etc...
import {MysqlDataSource} from '../../datasources'
const query = `select * from accounts where IP="${ipAddress}" ORDER By
createdAt DESC Limit 1`
const dataSource = new MysqlDataSource()
const accountDetails = await dataSource.execute(query)
console.log(accountDetails)
instead of a raw query. call your repository from the function component below
import {MysqlDataSource} from '../datasources'
import {ContactUsUserDetailsRepository} from '../repositories'
const contactUsUserDetailsRepository = new ContactUsUserDetailsRepository(new MysqlDataSource)
function saveContactDetails(){
const payload={UID:"Unique ID",name:"Durai"}
await contactUsUserDetailsRepository.create(payload)
}
I would prefer 2 because, if you have a lot of imports in your cron class constractor you need to pass all of them while calling a function like in the 3rd point.
Have fun:)
I've created a custom Catalog in Spark 3.0.0:
class ExCatalogPlugin extends SupportsNamespaces with TableCatalog
I've provided the configuration asking Spark to load the Catalog:
.config("spark.sql.catalog.ex", "com.test.ExCatalogPlugin")
But Spark never loads the plugin, during debug no breakpoints are ever hit inside the initialize method, and none of the namespaces it exposes are recognized. There are also no error messages logged. If I change the class name to an invalid class name no errors are thrown either.
I wrote a small TEST case similar to the test cases in the Spark code, and I am able to load the plugin if I call:
package org.apache.spark.sql.connector.catalog
....
class CatalogsTest extends FunSuite {
test("EX") {
val conf = new SQLConf()
conf.setConfString("spark.sql.catalog.ex", "com.test.ExCatalogPlugin")
val plugin:CatalogPlugin = Catalogs.load("ex", conf)
}
}
Spark is using it's normal Lazy loading techniques, and doesn't instantiate the custom Catalog Plugin until it's needed.
In my case referencing the plugin in one of two ways worked:
USE ex, this explicit USE statement caused Spark to lookup the catalog and instantiate it.
I have a companion TableProvider defined as class DefaultSource extends SupportsCatalogOptions. This class has a hard coded extractCatalog set to ex. If I create a reader for this source, it sees the name of the catalog provider and will instantiate it. It then uses the Catalog Provider to create the table.
I have a few unit tests that need to have their own sparkSession. I extended SQLTestUtils, and am overriding the beforeAll and afterAll functions that are used in many other Spark Unit tests (from the source). I have a few test suites that look something like this:
class MyTestSuite extends QueryTest with SQLTestUtils {
protected var spark: SparkSession = null
override def beforeAll(): Unit = {
super.beforeAll()
spark = // initialize sparkSession...
}
override def afterAll(): Unit = {
try {
spark.stop()
spark = null
} finally {
super.afterAll()
}
}
// ... my tests ...
}
If I run one of these, it's fine, but if I run two or more, I get this error:
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /home/jenkins/workspace/Query/apache-spark/sql/hive-thriftserver-cat-server/metastore_db.
But I thought that the afterAll() was supposed to properly shut spark down so that I could create a new one. Is this not right? How do I accomplish this?
One way to do it this is to disable parallel test execution for your Spark app project to make sure only one instance of Spark Session object is active at the time. In sbt syntax it would like this:
project.in(file("your_spark_app"))
.settings(parallelExecution in Test := false)
The downside is that this is a per project setting and it would also affect the tests that would benefit from parallelization. A workaround would be to create a separate project for Spark tests.
I'm looking into using the hazelcast distributed Executor functionality, however het
HazelcastClient.getExecutor(String name) confuses me.
if i run one 'server' instance like so:
Config config = new
ExecutorConfig executorConfig = new ExecutorConfig("name");
config.addExecutorConfig(executorConfig);
HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);
This create a hazelcast node with executorConfig named 'name'
Considering this is the only ExecutorConfig for this instance i would expect to be able to submit Callables on this node on the executorService named 'name'
However if run the following (in a different process or machine)
HazelcastInstance client = HazelcastClient.newHazelcastClient(config);
CallableTest test = new CallableTest(); //callable that does sleep and sysout
IExecutorService executorService = client.getExecutorService("wrong_name");
executorService.submit(test);
the callable job gets submitted to the 'server' process and gets excecuted.
This seems weird to. i would expect to be able to manage the constraints of its executors.
The fact that this job gets executed even though there is no executorService with the name 'wrong_name' seem strange.
This leaves me wondering what the executor name is used for and how i can properly configure these executors.
Hazelcast will automatically create an executor with the given name. There is no restriction on the name; you don't need to configure it.
I can imagine that it feels a bit strange.
In short, make sure that you configure the name(s) correctly.