I spotted problem with Quarkus CqlSession. When i calling method "prepare" its hang and timeout.
Code snippet:
#Inject
public NewsEngineRepository(QuarkusCqlSession quarkusCqlSession, NewsEngineConfig newsEngineConfig, Logger log) {
queryLimit = newsEngineConfig.dbQueryLimit();
this.quarkusCqlSession = quarkusCqlSession;
this.log = log;
SimpleStatement ss = SimpleStatement.newInstance("select " + ITEMID + ", " + PUBDATE + " from news_engine_items where defverid = ? AND group = ? Limit " + queryLimit);
preparedStatement= quarkusCqlSession.prepare(ss); // <---- hangs here
this.log.info("Prepared statement created successfully: " + ss.getQuery());
}
I found a two solutions:
Increasing property quarkus.vertx.eventLoopsPoolSize from 2 to 4
Setting properyty quarkus.cassandra.use-quarkus-event-loop to false
Can anyone explain why solution 1 works?
Related
I am having source and target csv files with 10 million records with 250 columns.
I am running an apache beam pipeline which joins all columns from source and target file.
When, I run this on spark cluster the pipeline executes correctly with no exceptions but,
The join beam metrics counter returns double count when the following spark property is used.
-- executor-memory "2g"
But, When I increase the excutor-memory to 11g then it returns the correct count.
I have tried following example,
Pipeline pipeline = Pipeline.create(options);
final TupleTag<String> eventInfoTag = new TupleTag<>();
final TupleTag<String> countryInfoTag = new TupleTag<>();
PCollection<KV<String, String>> eventInfo =
eventsTable.apply(ParDo.of(new ExtractEventDataFn()));
PCollection<KV<String, String>> countryInfo =
countryCodes.apply(ParDo.of(new ExtractCountryInfoFn()));
PCollection<KV<String, CoGbkResult>> kvpCollection =
KeyedPCollectionTuple.of(eventInfoTag, eventInfo)
.and(countryInfoTag, countryInfo)
.apply(CoGroupByKey.create());
PCollection<KV<String, String>> finalResultCollection =
kvpCollection.apply(
"Process",
ParDo.of(
new DoFn<KV<String, CoGbkResult>, KV<String, String>>() {
#ProcessElement
public void processElement(ProcessContext c) {
KV<String, CoGbkResult> e = c.element();
String countryCode = e.getKey();
String countryName = "none";
countryName = e.getValue().getOnly(countryInfoTag);
for (String eventInfo : c.element().getValue().getAll(eventInfoTag)) {
Metrics.counter("count", "errorcount").inc();
c.output(
KV.of(
countryCode,
"Country name: " + countryName + ", Event info: " + eventInfo));
}
}
}));
final PipelineResult result = pipeline.run();
MetricQueryResults metrics =
result
.metrics()
.queryMetrics(
MetricsFilter.builder()
.addNameFilter(MetricNameFilter.inNamespace("count"))
.build());
Iterable<MetricResult<Long>> counters = metrics.getCounters();
for (MetricResult<Long> counter : counters) {
System.out.println("Hi >> "+counter.getName().getName() + " : " + counter.getAttempted() + " " + counter.getCommittedOrNull());
}
I need help with this.
Thank you
public static void main(String[] args) {
Configuration hadoopConf = new Configuration();
hadoopConf.set("fs.defaultFS", args[13]);
hadoopConf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
hadoopConf.set("fs.file.impl",org.apache.hadoop.fs.LocalFileSystem.class.getName());
final TupleTag<Row> sourceDataInfoTag = new TupleTag<Row>(){};
final TupleTag<Row> targetDataInfoTag = new TupleTag<Row>(){};
HadoopFileSystemOptions options = PipelineOptionsFactory.as(HadoopFileSystemOptions.class);
options.setRunner(SparkRunner.class);
options.setHdfsConfiguration(Collections.singletonList(hadoopConf));
Pipeline pipeline = Pipeline.create(options);
PCollection<String> sourceData = pipeline.apply(TextIO.read().from(args[14]).withDelimiter("\n".getBytes()));
PCollection<KV<Row, Row>> sourceDataRows = sourceData.apply(ParDo.of(new ExtractFunction()));
PCollection<String> targetData = pipeline.apply(TextIO.read().from(args[23]).withDelimiter("\n".getBytes()));
PCollection<KV<Row, Row>> targetDataRows = targetData.apply(ParDo.of(new ExtractFunction()));
PCollection<KV<Row, CoGbkResult>> kvpCollection = KeyedPCollectionTuple
.of(sourceDataInfoTag, sourceDataRows.setCoder(KvCoder.of(RowCoder.of(SOURCE_JOIN_RECORD_TYPE),RowCoder.of(SOURCE_RECORD_TYPE))))
.and(targetDataInfoTag, targetDataRows.setCoder(KvCoder.of(RowCoder.of(TARGET_JOIN_RECORD_TYPE),RowCoder.of(TARGET_RECORD_TYPE))))
.apply(CoGroupByKey.<Row>create());
PCollection<GenericRecord> finalResultCollections = kvpCollection.apply("process",ParDo.of(new DoFn<KV<Row, CoGbkResult>, GenericRecord>() {
#ProcessElement
public void processElement(ProcessContext context) {
KV<Row, CoGbkResult> element = context.element();
Iterator<Row> srcIter = element.getValue().getAll(sourceDataInfoTag).iterator();
Iterator<Row> trgIter = element.getValue().getAll(targetDataInfoTag).iterator();
Metrics.counter("count", "count").inc();
GenericRecordBuilder builder = new GenericRecordBuilder(SCHEMA);
boolean done = false;
boolean captureError = false;
while (!done)
{
// Some iterator data here.
.
.
builder.set(colName, data);
if(captureError){
GenericRecord record = builder.build();
context.output(record);
}
}
}
})).setCoder(AvroCoder.of(GenericRecord.class, SCHEMA));
finalResultCollections.apply("writeText",FileIO.<GenericRecord>write()
.via(ParquetIO.sink(SCHEMA))
.withSuffix(".parquet")
.withPrefix("part")
.to("hdfs://temp/"));
final PipelineResult result = pipeline.run();
State state = result.waitUntilFinish();
MetricQueryResults metrics =
result
.metrics()
.queryMetrics(
MetricsFilter.builder()
.addNameFilter(MetricNameFilter.inNamespace("count"))
.build());
Iterable<MetricResult<Long>> counters = metrics.getCounters();
for (MetricResult<Long> counter : counters) {
System.out.println("Count >> "+counter.getName().getName() + " : " + counter.getAttempted() + " " + counter.getCommittedOrNull());
}
}
In your code, when you do Metrics.counter("count", "errorcount") you define the counter. But it is defined in a loop which is also in sort of a loop (processElement). You should define your counter as a field in the DoFn. Don't worry the DoFn is reused for processing the bundle. Such as: private final Counter counter = Metrics.counter(MyClass.class, COUNTER_NAME);
Also you showed only part of the code but I don't see done boolean set to true. But that is just out of curiosity.
And last but not least, you should try spark runner on master branch of Beam because there was a fix merged yesterday about metrics (metrics not reset when running multiple pipelines inside the same JVM). I don't know if it matches your use case but it's worth trying.
I'm stuck with the following problem within Processing for a school assignment.
I'm using a dataset in a tab seperated format, This is being read and parsed to my Activity class. The (menuAnswer, "Subagency") is used so I only get the data I need.
for(TableRow singleRow : trafficTable.findRows(menuAnswer, "SubAgency")){
Activity singleActivity = new Activity();
singleActivity.parseRow(singleRow);
activities.add(singleActivity);
}
The Activity class looks this:
class Activity{
String violationType;
String subAgency;
String race;
String gender;
Date readDate;
void parseRow(TableRow row){
this.subAgency = row.getString("SubAgency");
this.violationType = row.getString("Violation Type");
this.race = row.getString("Race");
this.gender = row.getString("Gender");
this.readDate = parseDate(row.getString("Time Of Stop") + " " + row.getString("Date"), "HH:mm:ss dd-MM-yyyy");
}
void printInfo(){
println(subAgency + " / " + race + " / " + readDate + " / " + violationType);
}
}
Every (usefull) piece of my dataset is inserted into an variable.
In my main class I want to search in the violationType String and count the ammount of "Warning"s within this String. I use the following code, which is not working:
for (Activity singleActivity : activities)
if(singleActivity.violationType == "Warning"){
warningCount++;
println("is it working?");
}
What am I doing wrong?
Mello
After searching I found my answer:
for (Activity singleActivity : activities)
if(singleActivity.violationType == "Warning"){
warningCount++;
println("is it working?");
}
My notation of my If statement is wrong and should be with .equals instead of == operator, like this:
void countWarning(){
for (Activity singleActivity : activities){
if(singleActivity.violationType.equals("Warning")){
warningCount++;
}
}
}
CRM 2011 on premise.
I have a plugin written in C#. It could throw exceptions or otherwise behave badly in production.
When this happens I want to capture information about the state and about recent code execution to help me analyse the problem.
Ideally I would like the following:
If the code decides I should know about a problem then I want it to be able to tell me that there was a problem asap, without me looking to see if a problem has occurred.
I want the information about the problem to be easily accessible to me. I don't want to have to RDP to another machine and search through files.
I don't want logging to have much impact on performance.
I guess I'm looking for something like this:
Retention of last n lines of log in memory.
A single function call to add a line to the log.
A single function call to cause an error to be recorded.
When there's an error, email to me saying there was a problem. This contains summary info.
The email contains a link to a web page showing me full error details. Stack trace, error message, date, time, user etc, last n lines of log.
It'd be nice if there was also a web page somewhere showing me all errors, with filtering, sorting and so on.
I'm fairly new to CRM but I've developed systems like this before. Since CRM has been around for many years I'd expect this to be available for it.
To support your wish list, I'd create these Entities:
PluginLog
Contains what ever information you'd like to retain for plugin calls that completed successfully
Is related to a Plugin Exception Class. This way you could look up what happened before
an exception occurred
PluginException
Contains any special information regarding the exception (user, context, stack trace)
Now let's go through your wish list:
Retention of last n lines of log in memory.
Not sure if you want to log the particular plugin class, or all plugins classes defined in the DLL, I'll assume a particular plugin class:
Create a static ConcurrentQueue for the plugin
A single function call to add a line to the log.
Create a single function that creates a PluginLog entity in memory (without creating it in the CRM database) and adds it to the queue.
If it's length > n, dequeue.
A single function call to cause an error to be recorded.
Again, this is something that you'll need to create. Basically I'd create a PLuginException Entity in CRM, then dequeue all the items off of the Queue, populating the Plugin Exception Id, and saving it to CRM
When there's an error, email to me saying there was a problem. This contains summary info.
As long as the App Domain context of the executing plugin has the rights needed (not sure if it does in CRM Online) this should be trivial.
The email contains a link to a web page showing me full error details. Stack trace, error message, date, time, user etc, last n lines of log.
You can create a link to the PluginException Entity Created, and include it in the e-mail along with all other pertinent information.
It'd be nice if there was also a web page somewhere showing me all errors, with filtering, sorting and so on.
Advanced Find to the rescue
Edit
To help get you started, this is what I currently use to retrieve all the information from the plugin context, and convert it into text that gets inserted into an Exception:
#region GetPluginInfo
private Exception GetPluginExecutionInfoForLog(IServiceProvider serviceProvider, Exception ex)
{
if(ex.GetType() == typeof(InvalidPluginExecutionException)){ return ex; }
try
{
var context = serviceProvider.GetContext();
ex = new InvalidPluginExecutionException(
String.Format("Error During Plugin Execution: {0}**** Context Values ****{0}{1}",
Environment.NewLine, GetPluginExecutionInfo(context)), ex);
}
catch (Exception childEx)
{
OnError(childEx);
}
return ex;
}
protected String GetPluginExecutionInfo(IPluginExecutionContext context)
{
var lines = new List<String>();
var target = GetTarget<Entity>(context);
lines.Add("MessageName: " + context.MessageName);
lines.Add("PrimaryEntityName: " + context.PrimaryEntityName);
lines.Add("PrimaryEntityId: " + context.PrimaryEntityId);
lines.Add("BusinessUnitId: " + context.BusinessUnitId);
lines.Add("CorrelationId: " + context.CorrelationId);
lines.Add("Depth: " + context.Depth);
lines.Add("Has Parent Context: " + (context.ParentContext != null));
lines.Add("InitiatingUserId: " + context.InitiatingUserId);
AddParameters(lines, context.InputParameters, "Input Parameters");
lines.Add("IsInTransaction: " + context.IsInTransaction);
lines.Add("IsolationMode: " + context.IsolationMode);
lines.Add("Mode: " + context.Mode);
lines.Add("OperationCreatedOn: " + context.OperationCreatedOn);
lines.Add("OperationId: " + context.OperationId);
lines.Add("Organization: " + context.OrganizationName + "(" + context.OrganizationId + ")");
AddParameters(lines, context.OutputParameters, "Output Parameters");
AddEntityReference(lines, context.OwningExtension, "OwningExtension");
AddEntityImages(lines, context.PostEntityImages, "Post Entity Images");
AddEntityImages(lines, context.PreEntityImages, "Pre Entity Images");
lines.Add("SecondaryEntityName: " + context.SecondaryEntityName);
AddParameters(lines, context.SharedVariables, "Shared Variables");
lines.Add("Stage: " + context.Stage);
lines.Add("UserId: " + context.UserId);
if (target == null || target.Attributes.Count == 0)
{
lines.Add("Target: Empty ");
}
else
{
lines.Add("* Target " + target.ToEntityReference().GetNameId() + " *");
foreach (var att in target.Attributes)
{
lines.Add(" Entity[" + att.Key + "]: " + GetAttributeValue(att.Value));
}
}
lines.Add("* App Config Values *");
foreach (var key in ConfigurationManager.AppSettings.AllKeys)
{
lines.Add(" [" + key + "]: " + ConfigurationManager.AppSettings[key]);
}
return String.Join(Environment.NewLine, lines);
}
private static string GetAttributeValue(object value)
{
if(value == null){
return "Null";
}
var type = value.GetType();
if (type == typeof(OptionSetValue))
{
return ((OptionSetValue)value).Value.ToString();
}
else if (type == typeof(EntityReference))
{
return ((EntityReference)value).GetNameId();
}
else
{
return value.ToString();
}
}
private static void AddEntityReference(List<string> nameValuePairs, EntityReference entity, string name)
{
if (entity != null)
{
nameValuePairs.Add(name + ": " + entity.GetNameId());
}
}
private static void AddEntityImages(List<string> nameValuePairs, EntityImageCollection images, string name)
{
if (images != null && images.Count > 0)
{
nameValuePairs.Add("** " + name + " **");
foreach (var image in images)
{
if (image.Value == null || image.Value.Attributes.Count == 0)
{
nameValuePairs.Add(" Image[" + image.Key + "] " + image.Value.ToEntityReference().GetNameId() + ": Empty");
}
else
{
nameValuePairs.Add("* Image[" + image.Key + "] " + image.Value.ToEntityReference().GetNameId() + " *");
foreach (var att in image.Value.Attributes)
{
nameValuePairs.Add(" Entity[" + att.Key + "]: " + GetAttributeValue(att.Value));
}
}
}
}
else
{
nameValuePairs.Add(name + ": Empty");
}
}
private static void AddParameters(List<string> nameValuePairs, ParameterCollection parameters, string name)
{
if (parameters != null && parameters.Count > 0)
{
nameValuePairs.Add("* " + name + " *");
foreach (var param in parameters)
{
nameValuePairs.Add(" Param[" + param.Key + "]: " + param.Value);
}
}
else
{
nameValuePairs.Add(name + ": Empty");
}
}
#endregion // GetPluginInfo
I've found a very quick way to do this at http://mscrmtech.com/mscrm-2011-useful-itracingservice-addiion-when-creating-plugin-assemblies/
It is documented (barely) here: http://msdn.microsoft.com/en-us/library/microsoft.xrm.sdk.itracingservice.aspx
So I can do
public void Execute(IServiceProvider serviceProvider)
{
ITracingService tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));
if (tracingService == null)
throw new InvalidPluginExecutionException("Failed to retrieve the tracing service.");
tracingService.Trace("CRMSTClasses.Main_Code.Execute - testing tracing");
throw new InvalidPluginExecutionException("Test exception");
And then the log file I can download from the error dialogue includes:
[CRMSTClasses: CRMSTClasses.Main_Code.Receiver]
[cb42fcb0-0717-e311-9097-0050569274a2: CRMSTClasses.Main_Code.Receiver: Create of incident]
CRMSTClasses.Main_Code.Execute - testing tracing
In the latest versions of Dynamics, the Plugin Trace Log captures a lot of this information simply by a user calling the pre-built Tracing Service. It doesn't enable proactive notifications as requested, but it has lots of other nice features. When you're ready to upgrade (if that's in the cards), I would recommend taking a look at it.
i am using EF 4.1 and when I use the following code i get an error:
'id' could not be resolved in the current scope or context. Make sure that all referenced variables are in scope, that required schemas are loaded, and that namespaces are referenced correctly. Near simple identifier, line 6, column 1.
c#
foreach (Person Profile in _ProfilesRepository.GetProfiles(filterExpression.ToString(), sortingName, request.SortingOrder.ToString(), request.PageIndex, request.RecordsCount, request.PagesCount.HasValue ? request.PagesCount.Value : 1))
{
//the error happens here when it comes back from the GetProfiles call : (
response.Records.Add(new JqGridRecord(Convert.ToString(Profile.ID), new List<object>()
{
Profile.Name,
Profile.JobTitle,
Profile.Type
}));
}
public IQueryable<Person> GetProfiles(string filterExpression, string sortExpression, string sortDirection, int pageIndex, int pageSize, int pagesCount)
{
if (!String.IsNullOrWhiteSpace(filterExpression))
return db.persons.Where(filterExpression).OrderBy(sortExpression + " " + sortDirection).Skip(pageIndex * pageSize).Take(pageSize);
else
return db.persons.OrderBy(sortExpression + " " + sortDirection).Skip(pageIndex * pageSize).Take(pagesCount * pageSize);
//return db.persons;
}
The message appears when using orderBy("fieldname asc/desc")
db.persons.OrderBy(sortExpression + " " + sortDirection)
When I try to use orderBy using lambda expression it's working normally, no error message.
db.persons.OrderBy(x=>x.FieldName)
Try This, add "it" before your short expression/fieldname :
OrderBy("it."+sortExpression + " " + sortDirection).Skip(pageIndex *
PageSize).Take(pageSize); OrderBy("it."+sortExpression + " " +
sortDirection).Skip(pageIndex * pageSize).Take(pagesCount * pageSize);
I use log4j for the logging.
I could see the SQLs through Log4j like below.
Here's my java source which access data base with jdbcTemplate.
public QnaDTO selectBoard(int articleID) {
String SQL =
"SELECT " +
" QA.ARTICLE_ID, " +
" QA.EMAIL, " +
" QA.TEL, " +
" QA.CATEGORY_ID, " +
" CG.CATEGORY_NAME, " +
" QA.SUBJECT, " +
" QA.CONTESTS, " +
" QA.WRITER_NAME, " +
" QA.WRITER_ID, " +
" QA.READCOUNT, " +
" QA.ANSWER, " +
" QA.FILE_NAME, " +
" QA.OPEN_FLG, " +
" QA.KTOPEN_FLG, " +
" TO_CHAR(QA.WRITE_DAY, 'YYYY.MM.DD') WRITE_DAY, " +
" QA.DISPOSAL_FLG " +
"FROM QNA QA JOIN QNA_CATEGORY_GROUP CG " +
"ON QA.CATEGORY_ID = CG.CATEGORY_ID " +
"WHERE QA.ARTICLE_ID = ? ";
QnaDTO qnaDTO = (QnaDTO) jdbcTemplate.queryForObject(
SQL,
new Object[]{articleID},
new RowMapper() {
public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
QnaDTO qnaDTO = new QnaDTO();
qnaDTO.setArticleID(rs.getInt("ARTICLE_ID"));
qnaDTO.setCategoryID(rs.getInt("CATEGORY_ID"));
qnaDTO.setCategoryName(rs.getString("CATEGORY_NAME"));
qnaDTO.setEmail1(rs.getString("EMAIL"));
qnaDTO.setTel1(rs.getString("TEL"));
qnaDTO.setSubject(rs.getString("SUBJECT"));
qnaDTO.setContests(rs.getString("CONTESTS"));
qnaDTO.setName(rs.getString("WRITER_NAME"));
qnaDTO.setUserID(rs.getString("WRITER_ID"));
//
qnaDTO.setReadcount(rs.getString("READCOUNT"));
qnaDTO.setAnswer(rs.getString("ANSWER"));
qnaDTO.setFileName(rs.getString("FILE_NAME"));
qnaDTO.setOpenFlg(rs.getString("OPEN_FLG"));
qnaDTO.setKtOpenFlg(rs.getString("KTOPEN_FLG"));
//
qnaDTO.setWriteDay(rs.getString("WRITE_DAY"));
qnaDTO.setDisposalFlg(rs.getString("DISPOSAL_FLG"));
return qnaDTO;
}
}
);
return qnaDTO;
}
As you can see above.
jdbcTemplate.queryForObject(...) is the method which really send Query And Get some result.
Inside of jdbcTemplate.queryForObject, finally logger used
public Object query(final String sql, final ResultSetExtractor rse)
throws DataAccessException
{
Assert.notNull(sql, "SQL must not be null");
Assert.notNull(rse, "ResultSetExtractor must not be null");
if(logger.isDebugEnabled())
logger.debug("Executing SQL query [" + sql + "]");
class _cls1QueryStatementCallback
implements StatementCallback, SqlProvider
{
public Object doInStatement(Statement stmt)
throws SQLException
{
ResultSet rs = null;
Object obj;
try
{
rs = stmt.executeQuery(sql);
ResultSet rsToUse = rs;
if(nativeJdbcExtractor != null)
rsToUse = nativeJdbcExtractor.getNativeResultSet(rs);
obj = rse.extractData(rsToUse);
}
finally
{
JdbcUtils.closeResultSet(rs);
}
return obj;
}
public String getSql()
{
return sql;
}
_cls1QueryStatementCallback()
{
super();
}
}
return execute(new _cls1QueryStatementCallback());
}
But with above sources, I could only get the SQL with ?.
What I want is that my result doesn't have question mark ?
It means filling ? with real data.
Is there any way to do this?
thanks
Jeon, sorry, was occupied with work. :-) Anyway, I have looked into your code and replicate here using spring 2.5. I've also google and I think you want to read this and this further to understand.
From the official documentation,
Finally, all of the SQL issued by this class is logged at the 'DEBUG'
level under the category corresponding to the fully qualified class
name of the template instance (typically JdbcTemplate, but it may be
different if a custom subclass of the JdbcTemplate class is being
used).
So you need to figure out how to enable logging with debug level.
Not sure exactly how you trace, but with my trace, I end up to below. So if you enable debug level, you should be able to see the output, maybe not exactly like QA.ARTICLE_ID = 123; but you should probably get the value printed in the next line something like in that example. Anyway, I don't have the exact setup like in your environment but this I think should you give a clue.
public Object execute(PreparedStatementCreator psc, PreparedStatementCallback action)
throws DataAccessException {
Assert.notNull(psc, "PreparedStatementCreator must not be null");
Assert.notNull(action, "Callback object must not be null");
if (logger.isDebugEnabled()) {
String sql = getSql(psc);
logger.debug("Executing prepared SQL statement" + (sql != null ? " [" + sql + "]" : ""));
}