Below are my rule configs.
The problem is I am unable to receive emails after the first one or two emails.
Elastalert2 is running and its prints match found but don't send an alert except first alert when I started running the rule.
Last email I received was 19 hours ago and no emails after that although matches are found and elasrtalert2 rule is running constantly.
es_host: localhost
es_port: 9200
# Rule name, must be unique
name: Platform aggrigation rule prod Oct13
type: any
index: new-logstash*
aggregation:
schedule: '0 */12 * * *'
filter:
- term:
loglevel.keyword: "ERROR"
- terms:
servicename.keyword: ["postoffice", "pqrs"]
# - query:
# query_string:
# query: "message: enrolled"
# (Required)
# The alert is use when a match is found
alert:
- "email"
from_addr: "devtest#abc.com"
# (required, email specific)
# a list of email addresses to send alerts to
email:
#- "aamir.xyz#abc.net"
- "team#abc.net"
email_format: html
smtp_host: "mail.abc.com"
#smtp_host: "smtp.gmail.com"
smtp_port: 587 #for google email addresses
smtp_ssl: false
smtp_auth_file: "smtp_auth_file.yaml"
alert_subject: "Error Alert"
alert_text_type: alert_text_only
alert_text_args: ["loglevel","host.hostname","logtime","messageDetails", "servicename"]
alert_text: <html>
I have following 3 lines in a log that need to be grok'd for ElasticSearch through logstash.
2020-01-27 13:30:43,536 INFO com.test.bestmatch.streamer.function.BestMatchProcessor - Best match for ID: COi0620200110450BAD5CB723457A9B4747F1727 Total Batch Processing time: 3942
2020-01-27 13:30:43,581 INFO HTTPConnection - COi0620200110450BAD5CB723457A9B4747F1727 | People: 51 | Addresses: 5935 | HTTP Query Time: 24
2020-01-27 13:30:43,698 INFO bestRoute - COi0620200110450BAD5CB723457A9B4747F1727 | Touch Points: 117 | Best Match Time 3943
I tried various grok patterns but couldn't get to any concrete one.
Edited as per request
I need the following in ES in the context of the specific log entry
1st line
ID: COi0620200110450BAD5CB723457A9B4747F1727
Total Batch Processing time: 3942
2nd Line
ID: COi0620200110450BAD5CB723457A9B4747F1727
People: 51
Addresses: 5935
HTTP Query Time: 24
3rd Line
Touch Points 117
Best Match Time: 3943.
The output is from a Flink log. If there are flink patterns out there then please let me know.
1st line:
^%{TIMESTAMP_ISO8601:time}\s*%{LOGLEVEL:loglevel}.*ID: (?<ID>[\w\d]*).*time: (?<total_time>[\d]*)$
2nd line:
^%{TIMESTAMP_ISO8601:time}\s*%{LOGLEVEL:loglevel}.* - (?<ID>[\w]*).*People: (?<people>[\w]*).*Addresses: (?<addresses>[\d]*).*HTTP Query Time: (?<query_time>[\d]*)$
3rd line:
^%{TIMESTAMP_ISO8601:time}\s*%{LOGLEVEL:loglevel}.* - (?<ID>[\w]*).*Touch Points: (?<touch_points>[\d]*).*Best Match Time (?<best_match_time>[\d]*)$
There are many ways to parse this, this is only one approach. I would reccomend to adjust the field names I used to the new ECS. https://www.elastic.co/guide/en/ecs/current/index.html
I'm doing something with Spark-SQL and got error below:
YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to
remove executor 1 for reason Container marked as failed:
container_1568946404896_0002_02_000002 on host: worker1. Exit status:
-1000. Diagnostics: [2019-09-20 10:43:11.474]Task java.util.concurrent.ExecutorCompletionService$QueueingFuture#76430b7c
rejected from
org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor#16970b[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks =
1]
I'm trying to figure it out by checking the meaning of Exit status: 1000, however, no valuable info returned by googling.
According to this thread, the -1000 is not even mentioned.
Any comment is welcomed, thanks.
Hi I am parsing a complex json in ASA, My input is blob and output is SQL DB, When i run Azure stream analytics query and test it it runs fine and gives the result(i am manually giving input data from file to test the query), But when i start the stream analytics job it gives me the following error
"JobFailedMessage": "The streaming job failed: Stream Analytics job has validation errors: Query compilation error: Expression is not supported: 'GetArrayElement ( message . Body . NewRx . MedicationPrescribed , 0 ) . CompoundInformation . FinalCompoundPharmaceuticalDosageForm'..",
The complete query is
SELECT
message.Header.MessageID src_msg_id
,prescriberSPI as msg_pbr_spi
,eventProducedTime as evt_produce_dttm
,correlationId as correlation_id
,transactionType as msg_txn_type
,message.Header.RelatesToMessageID as relate_to_msg_id
,message.Header.SentTime as msg_sent_dttm
,message.Header.SenderSoftware.SenderSoftwareDeveloper as sndr_software_developer_name
,message.Header.SenderSoftware.SenderSoftwareProduct as sndr_software_prod__name
,message.Header.SenderSoftware.SenderSoftwareVersionRelease as sndr_software_vers_release
,message.Header.RxReferenceNumber.encValue as rx_ref_nbr
,message.Header.PrescriberOrderNumber as pbr_ord_nbr
,message.Header.DigitalSignature.DigitalSignatureIndicator as dgtl_signature_ind
,message.Header.DigitalSignature.DigestMethod as dgtl_signature_digest_method
,message.Header.PrescriberOrderGroup.OrderGroupNumber as ord_group_nbr
,message.Header.PrescriberOrderGroup.ItemCountInOrderGroup as item_cnt_in_ord_group
,message.Header.PrescriberOrderGroup.TotalCountForOrderGroup as tot_cnt_for_ord_group
,message.Header.PrescriberOrderGroup.OrderGroupReason as ord_group_reason
,message.Body.NewRx.UrgencyIndicatorCode as urgency_cd
,message.Body.NewRx.ChangeOfPrescriptionStatusFlag as rx_stat_chng_cd
,message.Body.NewRx.FollowUpRequest as followup_reqst_nbr
,message.Body.NewRx.AllergyOrAdverseEvent.NoKnownAllergies as allergy_no_known_ind
,GetArrayElement(message.Body.NewRx.MedicationPrescribed,0).CompoundInformation.FinalCompoundPharmaceuticalDosageForm as fnl_cmpnd_dosage_form_cd
,GetArrayElement(message.Body.NewRx.Observation,0).ObservationNotes as observation_note
from blobinput
This query works perfect when i run it in query tab and select test and it produces the output, but when i start the job it errors out saying
The streaming job failed: Stream Analytics job has validation errors: Query compilation error: Expression is not supported: 'GetArrayElement ( message . Body . NewRx . MedicationPrescribed , 0 )
Please try splitting the query into 2 steps, so there is no immediate referencing of the GetArrayElement() result.
Something like:
WITH Step1 AS (
SELECT
message.Header.MessageID src_msg_id
,prescriberSPI as msg_pbr_spi
,eventProducedTime as evt_produce_dttm
,correlationId as correlation_id
,transactionType as msg_txn_type
,message.Header.RelatesToMessageID as relate_to_msg_id
,message.Header.SentTime as msg_sent_dttm
,message.Header.SenderSoftware.SenderSoftwareDeveloper as sndr_software_developer_name
,message.Header.SenderSoftware.SenderSoftwareProduct as sndr_software_prod__name
,message.Header.SenderSoftware.SenderSoftwareVersionRelease as sndr_software_vers_release
,message.Header.RxReferenceNumber.encValue as rx_ref_nbr
,message.Header.PrescriberOrderNumber as pbr_ord_nbr
,message.Header.DigitalSignature.DigitalSignatureIndicator as dgtl_signature_ind
,message.Header.DigitalSignature.DigestMethod as dgtl_signature_digest_method
,message.Header.PrescriberOrderGroup.OrderGroupNumber as ord_group_nbr
,message.Header.PrescriberOrderGroup.ItemCountInOrderGroup as item_cnt_in_ord_group
,message.Header.PrescriberOrderGroup.TotalCountForOrderGroup as tot_cnt_for_ord_group
,message.Header.PrescriberOrderGroup.OrderGroupReason as ord_group_reason
,message.Body.NewRx.UrgencyIndicatorCode as urgency_cd
,message.Body.NewRx.ChangeOfPrescriptionStatusFlag as rx_stat_chng_cd
,message.Body.NewRx.FollowUpRequest as followup_reqst_nbr
,message.Body.NewRx.AllergyOrAdverseEvent.NoKnownAllergies as allergy_no_known_ind
,GetArrayElement(message.Body.NewRx.MedicationPrescribed,0) as MedicationPrescribed
,GetArrayElement(message.Body.NewRx.Observation,0) as Observation
from blobinput
)
Select
src_msg_id
,msg_pbr_spi
,evt_produce_dttm
,correlation_id
,msg_txn_type
,relate_to_msg_id
,msg_sent_dttm
,sndr_software_developer_name
,sndr_software_prod__name
,sndr_software_vers_release
,rx_ref_nbr
,pbr_ord_nbr
,dgtl_signature_ind
,dgtl_signature_digest_method
,ord_group_nbr
,item_cnt_in_ord_group
,tot_cnt_for_ord_group
,ord_group_reason
,urgency_cd
,rx_stat_chng_cd
,followup_reqst_nbr
,allergy_no_known_ind
,MedicationPrescribed.CompoundInformation.FinalCompoundPharmaceuticalDosageForm as fnl_cmpnd_dosage_form_cd
,Observation.ObservationNotes as observation_note
INTO output
from Step1
This is solved in the compatibility level 1.2 , From the Azure Streaming Analytics portal select the compatibility level 1.2(default is 1.1).
Artillery: How to run the scenarios sequentially and also display the results of each scenario in the same file?
I'm currently writing nodejs test with artillery.io to compare performance between two endpoints that I implemented. I defined two scenarios and I would like to get the result of each in a same report file.
The execution of the tests is not sequential, it means that at the end of the test I have a result already combined and impossible to know the performance of each one but for all.
config:
target: "http://localhost:8080/api/v1"
plugins:
expect: {}
metrics-by-endpoint: {}
phases:
- duration: 60
arrivalRate: 2
environments:
dev:
target: "https://backend.com/api/v1"
phases:
- duration: 60
arrivalRate: 2
scenarios:
- name: "Nashhorn"
flow:
- post:
url: "/casting/nashhorn"
auth:
user: user1
pass: user1
json:
body:
fromFile: "./casting-dataset-01-as-input.json"
options:
filename: "casting_dataset"
conentType: "application/json"
expect:
statusCode: 200
capture:
regexp: '[^]*'
as: 'result'
- log: 'result= {{result}}'
- name: "Nodejs"
flow:
- post:
url: "/casting/nodejs"
auth:
user: user1
pass: user1
json:
body:
fromFile: "./casting-dataset-01-as-input.json"
options:
filename: "casting_dataset"
conentType: "application/json"
expect:
statusCode: 200
capture:
regexp: '[^]*'
as: 'result'
- log: 'result= {{result}}'
How to run the scenarios sequentially and also display the results of each scenario in the same file?
Thank you in advance for your answers
I think you miss the param weight, this param defines de probability to execute the scenario. if in you first scenario put a weight of 1 and in the second put the same value, both will have the same probability to been execute (50%).
If you put in the first scenario a weight of 3 and in the second one a weight of 1, the second scenario will have a 25% probability of execution while the first one will have a 75% probability of being executed.
This combined with the arrivalRate parameter and setting the value of rampTo to 2, will cause 2 scenarios to be executed every second, in which if you set a weight of 1 to the two scenarios, they will be executed at the same time.
Look down for scenario weights in the documentation
scenarios:
- flow:
- log: Scenario for GET requests
- get:
url: /v1/url_test_1
name: Scenario for GET requests
weight: 1
- flow:
- log: Scenario for POST requets
- post:
json: {}
url: /v1/url_test_2
name: Scenario for POST
weight: 1
I hope this helps you.
To my knowledge, there isn't a good way to do this with the existing the artillery logic.
using this test script:
scenarios:
- name: "test 1"
flow:
- post:
url: "/postman-echo.com/get?test=123"
weight: 1
- name: "test 2"
flow:
- post:
url: "/postman-echo.com/get?test=123"
weight: 1
... etc...
Started phase 0 (equal weight), duration: 1s # 13:21:54(-0500) 2021-01-06
Report # 13:21:55(-0500) 2021-01-06
Elapsed time: 1 second
Scenarios launched: 20
Scenarios completed: 20
Requests completed: 20
Mean response/sec: 14.18
Response time (msec):
min: 117.2
max: 146.1
median: 128.6
p95: 144.5
p99: 146.1
Codes:
404: 20
All virtual users finished
Summary report # 13:21:55(-0500) 2021-01-06
Scenarios launched: 20
Scenarios completed: 20
Requests completed: 20
Mean response/sec: 14.18
Response time (msec):
min: 117.2
max: 146.1
median: 128.6
p95: 144.5
p99: 146.1
Scenario counts:
test 7: 4 (20%)
test 5: 2 (10%)
test 3: 1 (5%)
test 1: 4 (20%)
test 9: 2 (10%)
test 8: 3 (15%)
test 10: 2 (10%)
test 4: 1 (5%)
test 6: 1 (5%)
Codes:
404: 20
So basically you can see that they are weighted equally, but are not running equally. So I think there needs to be something added to the code itself for artillery. Happy to be wrong here.
You can use the per endpoint metrics plugin to give you the results per endpoint instead of aggregated.
https://artillery.io/docs/guides/plugins/plugin-metrics-by-endpoint.html
I see you already have this in your config, but it cannot be working if it is not giving you what you need. Did you install it as well as add to config?
npm install artillery-plugin-metrics-by-endpoint
In terms of running sequentially, I'm not sure why you would want to, but assuming you do, you just need to define each POST as part of the same Scenario instead of 2 different scenarios. That way the second step will only execute after the first step has responded. I believe the plugin is per endpoint, not per scenario so will still give you the report you want.