ARGH!!!
There seems to be a little confusion surrounding the SharedDbConnectionScope and TransactionScope objects to enable wrapping your SubSonic queries within a transaction.
The docs suggest specifying the using SharedDbConnectionScope wrapped around the using TransactionScope...
using(SharedDbConnectionScope scope = new SharedDbConnectionScope())
{
using(TransactionScope ts = new TransactionScope())
{
// do something
ts.Complete();
}
}
Then other question here such as Subsonic: Using SharedDbConnectionScope together with TransactionScope seems to be broken suggest the docs are wrong and the two objects should be the other way around...
using(TransactionScope ts = new TransactionScope())
{
using(SharedDbConnectionScope scope = new SharedDbConnectionScope())
{
// do something
ts.Complete();
}
}
But looking into the source code I am even more confused.
In the SqlQuery.cs code file it has a number of ExecuteTransaction overloads. For example...
public static void ExecuteTransaction(List<SqlQuery> queries)
{
using(SharedDbConnectionScope scope = new SharedDbConnectionScope())
{
using(TransactionScope ts = new TransactionScope())
{
foreach(SqlQuery q in queries)
q.Execute();
}
}
}
Umm... Interesting... Matches the docs but... Where's the ts.Complete() call?
How is that supposed to commit the transaction? As far as I can see it will always rollback. And it is the same for all the ExecuteTransaction overloads!
But here is the real kicker...
In the TransactionWithDtcOffTests.cs code has some nice tests except they have set up the SharedDbConnectionScope and TransactionScope around the other way!
using(TransactionScope ts = new TransactionScope())
{
using(SharedDbConnectionScope connScope = new SharedDbConnectionScope())
{
// <snip />
}
}
I haven't had the opportunity to run the tests for SubSonic 2.2 but I assume someone has and they passed..
Finally...
Can someone give me the definitive answer to how Transactions in SubSonic2.2 should be set up? Are the docs indeed wrong? Does the source for the ExecuteTransaction overloads and tests be aligned to whichever way is actually correct?
The SharedConnectionScope (SCS) block must be inside a TransactionScope (TS).
The purpose of the SCS is to prevent escalating the transaction to the MSDTC if possible, so having the TS using block inside of a SCS using block makes little sense to me.
In any case, every TS block must have a Complete() call for the transaction to be committed.
Personally i found that when using SQL 2005, SCS must be inside TS
and when using SQL 2000 (with MSDTC), SCS must wrap TS.
I hope this helps...
Related
I'm playing around with reactive flows using RxJava2, Micronaut and Cassandra. I'm new to rxjava and not sure what is the correct way to return a of List Person in the best async manner?
data is coming from a Cassandra Dao interface
public interface PersonDAO {
#Query("SELECT * FROM cass_drop.person;")
CompletionStage<MappedAsyncPagingIterable<Person>> getAll();
}
that gets injected into a micronaut controller
return Single.just(personDAO.getAll().toCompletableFuture().get().currentPage())
.subscribeOn(Schedulers.io())
.map(people -> HttpResponse.ok(people));
OR
return Single.just(HttpResponse.ok())
.subscribeOn(Schedulers.io())
.map(it -> it.body(personDAO.getAll().toCompletableFuture().get().currentPage()));
OR switch to RxJava3
return Single.fromCompletionStage(personDAO.getAll())
.map(page -> HttpResponse.ok(page.currentPage()))
.onErrorReturn(throwable -> HttpResponse.ok(Collections.emptyList()));
Not a pro of RxJava nor Cassandra :
In your first and second example, you are blocking the thread executing the CompletionStage with get, even if you are doing it in the IO thread, I would not recommand doing so.
You are also using a Single wich can emit, only one value, or an error. Since you want to return a List, I would sugest to go for at least an Observable.
Third point, the result from Cassandra is paginated, I don't know if it's intentionnaly but you list only the first page, and miss the others.
I would try a solution like the one below, I kept using the IO thread (the operation may be costly in IO) and I iterate over the pages Cassandra fetch :
/* the main method of your controller */
#Get()
public Observable<Person> listPersons() {
return next(personDAO.getAll()).subscribeOn(Schedulers.io());
}
private Observable<Person> next(CompletionStage<MappedAsyncPagingIterable<Person>> pageStage) {
return Single.fromFuture(pageStage.toCompletableFuture())
.flatMapObservable(personsPage -> {
var o = Observable.fromIterable(personsPage.currentPage());
if (!personsPage.hasMorePages()) {
return o;
}
return o.concatWith(next(personsPage.fetchNextPage()));
});
}
If you ever plan to use reactor instead of RxJava, then you can give cassandra-java-driver-reactive-mapper a try.
The syntax is fairly simple and works in compile-time only.
I'm exploring Kotlin Native and have a program with a bunch of Workers doing concurrent stuff
(running on Windows, but this is a general question).
Now, I wanted to add simple logging. A component that simply logs strings by appending them as new lines to a file that is kept open in 'append' mode.
(Ideally, I'd just have a "global" function...
fun log(text:String) {...} ]
...that I would be able to call from anywhere, including from "inside" other workers and that would just work. The implication here is that it's not trivial to do this because of Kotlin Native's rules regarding passing objects between threads (TLDR: you shouldn't pass mutable objects around. See: https://github.com/JetBrains/kotlin-native/blob/master/CONCURRENCY.md#object-transfer-and-freezing ).
Also, my log function would ideally accept any frozen object. )
What I've come up with are solutions using DetachedObjectGraph:
First, I create a detached logger object
val loggerGraph = DetachedObjectGraph { FileLogger("/foo/mylogfile.txt")}
and then use loggerGraph.asCPointer() ( asCPointer() ) to get a COpaquePointer to the detached graph:
val myPointer = loggerGraph.asCPointer()
Now I can pass this pointer into the workers ( via the producer lambda of the Worker's execute function ), and use it there. Or I can store the pointer in a #ThreadLocal global var.
For the code that writes to the file, whenever I want to log a line, I have to create a DetachedObjectGraph object from the pointer again,
and attach() it in order to get a reference to my fileLogger object:
val fileLogger = DetachedObjectGraph(myPointer).attach()
Now I can call a log function on the logger:
fileLogger.log("My log message")
This is what I've come up with looking at the APIs that are available (as of Kotlin 1.3.61) for concurrency in Kotlin Native,
but I'm left wondering what a better approach would be ( using Kotlin, not resorting to C ). Clearly it's bad to create a DetachedObjectGraph object for every line written.
One could pose this question in a more general way: How to keep a mutable resource open in a separate thread ( or worker ), and send messages to it.
Side comment: Having Coroutines that truly use threads would solve this problem, but the question is about how to solve this task with the APIs currently ( Kotlin 1.3.61 ) available.
You definitely shouldn't use DetachedObjectGraph in the way presented in the question. There's nothing to prevent you from trying to attach on multiple threads, or if you pass the same pointer, trying to attach to an invalid one after another thread as attached to it.
As Dominic mentioned, you can keep the DetachedObjectGraph in an AtomicReference. However, if you're going to keep DetachedObjectGraph in an AtomicReference, make sure the type is AtomicRef<DetachedObjectGraph?> and busy-loop while the DetachedObjectGraph is null. That will prevent the same DetachedObjectGraph from being used by multiple threads. Make sure to set it to null, and repopulate it, in an atomic way.
However, does FileLogger need to be mutable at all? If you're writing to a file, it doesn't seem so. Even if so, I'd isolate the mutable object to a separate worker and send log messages to it rather than doing a DetachedObjectGraph inside an AtomicRef.
In my experience, DetachedObjectGraph is super uncommon in production code. We don't use it anywhere at the moment.
To isolate mutable state to a Worker, something like this:
class MutableThing<T:Any>(private val worker:Worker = Worker.start(), producer:()->T){
private val arStable = AtomicReference<StableRef<T>?>(null)
init {
worker.execute(TransferMode.SAFE, {Pair(arStable, producer).freeze()}){
it.first.value = StableRef.create(it.second()).freeze()
}
}
fun <R> access(block:(T)->R):R{
return worker.execute(TransferMode.SAFE, {Pair(arStable, block).freeze()}){
it.second(it.first.value!!.get())
}.result
}
}
object Log{
private val fileLogger = MutableThing { FileLogger() }
fun log(s:String){
fileLogger.access { fl -> fl.log(s) }
}
}
class FileLogger{
fun log(s:String){}
}
The MutableThing uses StableRef internally. producer makes the mutable state you want to isolate. To log something, call Log.log, which will wind up calling the mutable FileLogger.
To see a basic example of MutableThing, run the following test:
#Test
fun goIso(){
val mt = MutableThing { mutableListOf("a", "b")}
val workers = Array(4){Worker.start()}
val futures = mutableListOf<Future<*>>()
repeat(1000) { rcount ->
val future = workers[rcount % workers.size].execute(
TransferMode.SAFE,
{ Pair(mt, rcount).freeze() }
) { pair ->
pair.first.access {
val element = "ttt ${pair.second}"
println(element)
it.add(element)
}
}
futures.add(future)
}
futures.forEach { it.result }
workers.forEach { it.requestTermination() }
mt.access {
println("size: ${it.size}")
}
}
The approach you've taken is pretty much correct and the way it's supposed to be done.
The thing I would add is, instead of passing around a pointer around. You should pass around a frozen FileLogger, which will internally hold a reference to a AtomicRef<DetachedObjectGraph>, the the attaching and detaching should be done internally. Especially since DetachedObjectGraphs are invalid once attached.
I am using Spring Batch StoredProcedureItemReader to retrive the result set and insert it to another database using JpaItemWriter.
Below is my code configuration.
#Bean
public JdbcCursorItemReader jdbcCursorItemReader(){
JdbcCursorItemReader jdbcCursorItemReader = new JdbcCursorItemReader();
jdbcCursorItemReader.setSql("call myProcedure");
jdbcCursorItemReader.setRowMapper(new MyRowMapper());
jdbcCursorItemReader.setDataSource(myDataSource);
jdbcCursorItemReader.setFetchSize(50);
jdbcCursorItemReader.setVerifyCursorPosition(false);
jdbcCursorItemReader.setSaveState(false);
return jdbcCursorItemReader;
}
#Bean
public Step step() {
threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(50);
threadPoolTaskExecutor.setMaxPoolSize(100);
threadPoolTaskExecutor.setThreadNamePrefix("My-TaskExecutor ");
threadPoolTaskExecutor.setWaitForTasksToCompleteOnShutdown(Boolean.TRUE);
threadPoolTaskExecutor.initialize();
return stepBuilderFactory.get("myJob").transactionManager(secondaryTransactionManager)
.chunk(50).reader(jdbcCursorItemReader())
.writer(myJpaItemWriter())
.taskExecutor(threadPoolTaskExecutor)
.throttleLimit(100)
.build();
}
The code works fine without multithreading or threadpooltaskexecutor.However, when using them i encounter below error.
Caused by: java.sql.SQLDataException: Current position is after the last row
could not execute statement [n/a] com.microsoft.sqlserver.jdbc.SQLServerException: Violation of PRIMARY KEY constraint
I have tried using JdbcCursotItemReader, even then i am facing the same error.Any Suggestions on how to make this work
JdbcCursorItemReader is not thread safe because it is based on a ResultSet which is not thread safe. The StoredProcedureItemReader is also based on a ResultSet, hence it is not thread-safe neither. See https://stackoverflow.com/a/53964556/5019386
Try to use the JdbcPagingItemReader which is thread-safe or if you really have to use the StoredProcedureItemReader, then make it thread-safe by wrapping it in a SynchronizedItemStreamReader.
Hope this helps.
Wondering if there is a way I can use sql.eachRow like a generator, to use it in a DSL context where a Collection or Iterator is expected. The use case I'm trying to go for is streaming JSON generation - what I'm trying to do is something like:
def generator = { sql.eachRow { yield it } }
jsonBuilder.root {
status "OK"
rows generator()
}
You would need continuation support (or similiar) for this to work to some extend. Groovy does not have continuations, the JVM also not. Normally continuation passing style works, but then the method eachRow would have to support that, which it of course does not. So the only way I see is a makeshift solution using threads or something like that. So maybe something like that would work for you:
def sync = new java.util.concurrent.SynchronousQueue()
Thread.start { sql.eachRow { sync.put(it) } }
jsonBuilder.root {
status "OK"
rows sync.take()
}
I am not stating, that this is a good solution, just a random consumer-producer-work-around for your problem.
As new to groovy...
I'm trying to replace the java idiom for event listeners, filters, etc.
My working code in groovy is the following:
def find() {
ODB odb = ODBFactory.open(files.nodupes); // data nucleus object database
Objects<Prospect> src = odb.getObjects(new QProspect());
src.each { println it };
odb.close();
}
class QProspect extends SimpleNativeQuery {
public boolean match(Prospect p) {
if (p.url) {
return p.url.endsWith(".biz");
}
return false;
}
}
Now, this is far from what I'm used to in java, where the implementation of the Query interface is done right inside the odb.getObjects() method. If I where to code "java" I'd probably do something like the following, yet it's not working:
Objects<Prospect> src = odb.getObjects( {
boolean match(p) {
if (p.url) {
return p.url.endsWith(".biz");
}
return false;
}
} as SimpleNativeQuery);
Or better, I'd like it to be like this:
Objects<Prospect> src = odb.getObjects(
{ it.url.endsWith(".biz") } as SimpleNativeQuery
);
However, what groovy does it to associate the "match" method with the outer script context and fail me.
I find groovy... groovy anyways so I'll stick to learning more about it. Thanks.
What I should've asked was how do we do the "anonymous" class in groovy. Here's the java idiom:
void defReadAFile() {
File[] files = new File(".").listFiles(new FileFilter() {
public boolean accept(File file) {
return file.getPath().endsWith(".biz");
}
});
}
Can groovy be as concise with no additional class declaration?
I think it would have helped you to get answers if you'd abstracted the problem so that it didn't rely on the Neodatis DB interface -- that threw me for a loop, as I've never used it. What I've written below about it is based on a very cursory analysis.
For that matter, I've never used Groovy either, though I like what I've seen of it. But seeing as no one else has answered yet, you're stuck with me :-)
I think the problem (or at least part of it) may be that you're expecting too much of the SimpleNativeQuery class from Neodatis. It doesn't look like it even tries to filter the objects before it adds them to the returned collection. I think instead you want to use org.neodatis.odb.impl.core.query.criteria.CriteriaQuery. (Note the "impl" in the package path. This has me a bit nervous, as I don't know for sure if this class is meant to be used by callers. But I don't see any other classes in Neodatis that allow for query criteria to be specified.)
But instead of using CriteriaQuery directly, I think you'd rather wrap it inside of a Groovy class so that you can use it with closures. So, I think a Groovy version of your code with closures might look something like this:
// Create a class that wraps CriteriaQuery and allows you
// to pass closures. This is wordy too, but at least it's
// reusable.
import org.neodatis.odb.impl.core.query.criteria;
class GroovyCriteriaQuery extends CriteriaQuery {
private final c;
QProspect(theClosure) {
// I prefer to check for null here, instead of in match()
if (theClosure == null) {
throw new InvalidArgumentException("theClosure can't be null!");
}
c = theClosure;
}
public boolean match(AbstractObjectInfo aoi){
//!! I'm assuming here that 'aoi' can be used as the actual
//!! object instance (or at least as proxy for it.)
//!! (You may have to extract the actual object from aoi before calling c.)
return c(aoi);
}
}
// Now use the query class in some random code.
Objects<Prospect> src = odb.getObjects(
new GroovyCriteriaQuery(
{ it.url.endsWith(".biz") }
)
)
I hope this helps!
I believe your real question is "Can I use closures instead of anonymous classes when calling Java APIs that do not use closures". And the answer is a definite "yes". This:
Objects<Prospect> src = odb.getObjects(
{ it.url.endsWith(".biz") } as SimpleNativeQuery
);
should work. You write "However, what groovy does it to associate the "match" method with the outer script context and fail me". How exactly does it fail? It seems to me like you're having a simple technical problem to get the solution that is both "the groovy way" and exactly what you desire to work.
Yep, thanks y'all, it works.
I also found out why SimpleNativeQuery does not work (per Dan Breslau).
I tried the following and it worked wonderfully. So the idiom does work as expected.
new File("c:\\temp").listFiles({ it.path.endsWith(".html") } as FileFilter);
This next one does not work because of the neodatis interface. The interface does not enforce a match() method! It only mentions it in the documentation yet it's not present in the class file:
public class SimpleNativeQuery extends AbstactQuery{
}
Objects<Prospect> src = odb.getObjects(
{ it.url.endsWith(".biz") } as SimpleNativeQuery
);
In the above, as the SimpleNativeQuery does not have a match() method, it makes it impossible for the groovy compiler to identify which method in the SimpleNativeQuery should the closure be attached to; it then defaults to the outer groovy script.
It's my third day with groovy and I'm loving it.
Both books are great:
- Groovy Recipes (Scott Davis)
- Programming Groovy (Venkat Subramaniam)