I am using Spring Batch StoredProcedureItemReader to retrive the result set and insert it to another database using JpaItemWriter.
Below is my code configuration.
#Bean
public JdbcCursorItemReader jdbcCursorItemReader(){
JdbcCursorItemReader jdbcCursorItemReader = new JdbcCursorItemReader();
jdbcCursorItemReader.setSql("call myProcedure");
jdbcCursorItemReader.setRowMapper(new MyRowMapper());
jdbcCursorItemReader.setDataSource(myDataSource);
jdbcCursorItemReader.setFetchSize(50);
jdbcCursorItemReader.setVerifyCursorPosition(false);
jdbcCursorItemReader.setSaveState(false);
return jdbcCursorItemReader;
}
#Bean
public Step step() {
threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(50);
threadPoolTaskExecutor.setMaxPoolSize(100);
threadPoolTaskExecutor.setThreadNamePrefix("My-TaskExecutor ");
threadPoolTaskExecutor.setWaitForTasksToCompleteOnShutdown(Boolean.TRUE);
threadPoolTaskExecutor.initialize();
return stepBuilderFactory.get("myJob").transactionManager(secondaryTransactionManager)
.chunk(50).reader(jdbcCursorItemReader())
.writer(myJpaItemWriter())
.taskExecutor(threadPoolTaskExecutor)
.throttleLimit(100)
.build();
}
The code works fine without multithreading or threadpooltaskexecutor.However, when using them i encounter below error.
Caused by: java.sql.SQLDataException: Current position is after the last row
could not execute statement [n/a] com.microsoft.sqlserver.jdbc.SQLServerException: Violation of PRIMARY KEY constraint
I have tried using JdbcCursotItemReader, even then i am facing the same error.Any Suggestions on how to make this work
JdbcCursorItemReader is not thread safe because it is based on a ResultSet which is not thread safe. The StoredProcedureItemReader is also based on a ResultSet, hence it is not thread-safe neither. See https://stackoverflow.com/a/53964556/5019386
Try to use the JdbcPagingItemReader which is thread-safe or if you really have to use the StoredProcedureItemReader, then make it thread-safe by wrapping it in a SynchronizedItemStreamReader.
Hope this helps.
Related
Trying update Execution domain, update only values from table execution
import grails.gorm.annotation.Entity
#Entity
class Execution {
RatingItem price
Boolean isDone
}
#Entity
class RatingItem {
Boolean isDone
}
Saving in Transactional and Singleton Service called by a Controller
#Transactional
#Singleton
class ExecutionService{
boolean saveExecution(Execution execution){
if( execution.save(flush:true, failOnError:true) ){
return true
}
return false
}
}
Debugging..
execution.isDone is updated
execution.price.isDone not
After trasaction ends is_done in rating_item was not updated
This happens frequently. There is similar code that worked. In some cases I use DataSource to force update, but I need to use GORM this time.
As I mentioned before, there was similar code that worked. So I noticed that in this code I used the following:
Execution execution = Execution.findById(id)
And in the current problem:
Execution execution = Execution.get(id)
When I used findBy it worked.
I think it's because get doesn't fetch the domain correctly, even though all inner domains are lazy: false. I believe this does not happen in grails, but only in micronaut.
I'm playing around with reactive flows using RxJava2, Micronaut and Cassandra. I'm new to rxjava and not sure what is the correct way to return a of List Person in the best async manner?
data is coming from a Cassandra Dao interface
public interface PersonDAO {
#Query("SELECT * FROM cass_drop.person;")
CompletionStage<MappedAsyncPagingIterable<Person>> getAll();
}
that gets injected into a micronaut controller
return Single.just(personDAO.getAll().toCompletableFuture().get().currentPage())
.subscribeOn(Schedulers.io())
.map(people -> HttpResponse.ok(people));
OR
return Single.just(HttpResponse.ok())
.subscribeOn(Schedulers.io())
.map(it -> it.body(personDAO.getAll().toCompletableFuture().get().currentPage()));
OR switch to RxJava3
return Single.fromCompletionStage(personDAO.getAll())
.map(page -> HttpResponse.ok(page.currentPage()))
.onErrorReturn(throwable -> HttpResponse.ok(Collections.emptyList()));
Not a pro of RxJava nor Cassandra :
In your first and second example, you are blocking the thread executing the CompletionStage with get, even if you are doing it in the IO thread, I would not recommand doing so.
You are also using a Single wich can emit, only one value, or an error. Since you want to return a List, I would sugest to go for at least an Observable.
Third point, the result from Cassandra is paginated, I don't know if it's intentionnaly but you list only the first page, and miss the others.
I would try a solution like the one below, I kept using the IO thread (the operation may be costly in IO) and I iterate over the pages Cassandra fetch :
/* the main method of your controller */
#Get()
public Observable<Person> listPersons() {
return next(personDAO.getAll()).subscribeOn(Schedulers.io());
}
private Observable<Person> next(CompletionStage<MappedAsyncPagingIterable<Person>> pageStage) {
return Single.fromFuture(pageStage.toCompletableFuture())
.flatMapObservable(personsPage -> {
var o = Observable.fromIterable(personsPage.currentPage());
if (!personsPage.hasMorePages()) {
return o;
}
return o.concatWith(next(personsPage.fetchNextPage()));
});
}
If you ever plan to use reactor instead of RxJava, then you can give cassandra-java-driver-reactive-mapper a try.
The syntax is fairly simple and works in compile-time only.
In our poc, we have a cache in PARTIONED MODE, with 2 backups, and we started 3 nodes. 100 entries were loaded into cache and we did below steps to retrive it.
public void perform () throws GridException {
final GridCache<Long, Entity> cache= g.cache("cache");
GridProjection proj= g.forCache("cache");
Collection< Collection<Entity>> list= proj .compute().broadcast(
new GridCallable< Collection<Entity>>() {
#Override public Collection<Entity> call() throws Exception {
Collection<Entity> values= cache.primaryValues();
System.out.println("List size on each Node: "+ values.size());
// console from each node shows 28,38,34 respectively, which is correct
return values;
}
}).get();
for (Collection<Entity> e: list){
System.out.println("list size when arrives on main Node :"+ e.size());
//console shows 28 for three times, which is not correct
}
}
I assume that primaryValues() is to take value of each element returned by primaryEntrySet() out and put into a Collection. I also tried to use primaryEntrySet and it works without such problem.
The way GridGain serializes cache collections is by reference which may not be very intuitive. I have filed a Jira issue with Apache Ignite project (which is the next version of GridGain open source edition): https://issues.apache.org/jira/browse/IGNITE-38
In the mean time, please try the following from your GridCallable, which should work:
return new ArrayList(cache.primaryValues());
For a parameter class
class Criteria {
private Map params;
public getMap(){ return params; }
}
and a service method accept this criteria
class Service{
public List<Person> query(Criteria criteria){ ... }
}
A custom featureMatcher is used to match the criteria key
private Matcher<Criteria> hasCriteria(final String key, final Matcher<?> valueMatcher){
return new FeatureMatcher<Criteria, Object>((Matcher<? super Object>)valueMatcher, key, key){
#Override protected Object featureValueOf(Criteria actual){
return actual.getMap().get(key);
}
}
}
when using mockito to veryify the arguments:
verify(Service).query((Criteria) argThat("id", hasCriteria("id", equalTo(new Long(12)))));
The error message shows that:
Argument(s) are different! Wanted:
Service.query(
id <12L>
);
-> at app.TestTarget.test_id (TestTarget.java:134)
Actual invocation has different arguments:
Service.query(
app.Criteria#509f5011
);
If I use ArugmentCaptor,
ArgumentCaptor<Criteria> argument = ArgumentCaptor.forClass(Criteria.class);
verify(Service).query(argument.capture());
assertThat(argument.getValue(), hasCriteria("id", equalTo(new Long(12))));
The message is much better:
Expected: id <12L> but id was <2L>
How can I get such message, without using ArgumentCaptor?
The short answer is to adjust the Criteria code, if it's under your control, to write a better toString method. Otherwise, you may be better off using the ArgumentCaptor method.
Why is it hard to do without ArgumentCaptor? You know you're expecting one call, but Mockito was designed to handle it even if you have a dozen similar calls to evaluate. Even though you're using the same matcher implementation, with the same helpful describeMismatch implementation, assertThat inherently tries once to match where verify sees a mismatch and keeps trying to match any other call.
Consider this:
// in code:
dependency.call(true, false);
dependency.call(false, true);
dependency.call(false, false);
// in test:
verify(mockDependency).call(
argThat(is(equalTo(true))),
argThat(is(equalTo(true))));
Here, Mockito wouldn't know which of the calls was supposed to be call(true, true); any of the three might have been it. Instead, it only knows that there was a verification you were expecting that was never satisfied, and that one of three related calls might have been close. In your code with ArgumentCaptor, you can use your knowledge that there's only one call, and provide a more-sane error message; for Mockito, the best it can do is to output all the calls it DID receive, and without a helpful toString output for your Criteria, that's not very helpful at all.
ARGH!!!
There seems to be a little confusion surrounding the SharedDbConnectionScope and TransactionScope objects to enable wrapping your SubSonic queries within a transaction.
The docs suggest specifying the using SharedDbConnectionScope wrapped around the using TransactionScope...
using(SharedDbConnectionScope scope = new SharedDbConnectionScope())
{
using(TransactionScope ts = new TransactionScope())
{
// do something
ts.Complete();
}
}
Then other question here such as Subsonic: Using SharedDbConnectionScope together with TransactionScope seems to be broken suggest the docs are wrong and the two objects should be the other way around...
using(TransactionScope ts = new TransactionScope())
{
using(SharedDbConnectionScope scope = new SharedDbConnectionScope())
{
// do something
ts.Complete();
}
}
But looking into the source code I am even more confused.
In the SqlQuery.cs code file it has a number of ExecuteTransaction overloads. For example...
public static void ExecuteTransaction(List<SqlQuery> queries)
{
using(SharedDbConnectionScope scope = new SharedDbConnectionScope())
{
using(TransactionScope ts = new TransactionScope())
{
foreach(SqlQuery q in queries)
q.Execute();
}
}
}
Umm... Interesting... Matches the docs but... Where's the ts.Complete() call?
How is that supposed to commit the transaction? As far as I can see it will always rollback. And it is the same for all the ExecuteTransaction overloads!
But here is the real kicker...
In the TransactionWithDtcOffTests.cs code has some nice tests except they have set up the SharedDbConnectionScope and TransactionScope around the other way!
using(TransactionScope ts = new TransactionScope())
{
using(SharedDbConnectionScope connScope = new SharedDbConnectionScope())
{
// <snip />
}
}
I haven't had the opportunity to run the tests for SubSonic 2.2 but I assume someone has and they passed..
Finally...
Can someone give me the definitive answer to how Transactions in SubSonic2.2 should be set up? Are the docs indeed wrong? Does the source for the ExecuteTransaction overloads and tests be aligned to whichever way is actually correct?
The SharedConnectionScope (SCS) block must be inside a TransactionScope (TS).
The purpose of the SCS is to prevent escalating the transaction to the MSDTC if possible, so having the TS using block inside of a SCS using block makes little sense to me.
In any case, every TS block must have a Complete() call for the transaction to be committed.
Personally i found that when using SQL 2005, SCS must be inside TS
and when using SQL 2000 (with MSDTC), SCS must wrap TS.
I hope this helps...