The bull dashboard doesnt display the failed jobs ,but just displays count with some pagination
Related
I have a video uploading work that is handled by the work manager. When the user clicks the upload button, I want to show the notification with indeterminate progress bar and when the task starts, show the progress in the same notification.
I passed the notificationID to the work through workData and the progress is shown properly.
The issue is: If there is no internet connection, the indeterminate progress bar is shown and the work is not started (It is enqueued). When I close the app, the indeterminate progress bar is also dismissed and the enqueued task is not shown to the user.
How to make sure the notification persist even on app close or when the work manager has work enqueued to it?
We are trying to test Azure's "Healthcheck status" to be used in an auto-scaled app service.
We have an API endpoint which return status code 200 when it's OK and error 500 when it fails.
The problem is that we don't see any indication on the metrics blade. We are forcing the endpoint to return error 500 during a 10 minutes period every 30 minutes and the metrics doesn't show any difference. We only see "count" aggregation values.
Anyone knows what's is the meaning of "avg", "min", "max" in that blade ?
Additionally, is there any place where to validate the endpoint is being called every minute and the status code received? We are assuming it's working but we have a tracelog -using log4net- that only show messages if we call the endpoint manually but there are no records every minute as it should be if the endpoint is being called automatically.
Help is appreciated
Requirement:
Trigger a spark job from UI by user action (say submit button click).
Once the spark job is finished, the summary of the status has to be displayed in UI.
Design approach:
1. Once the user initiates a job run by clicking the submit button from UI, we will make an insert into a Impala queue table using Impala JDBC.
The simplified structure of the queue table is as follows:
JOB_RUN_QUEUE (REQUEST_ID, STATUS, INPUT_PARAM_1, INPUT_PARAM_2, SUMMARY)
The initial request will have STATUS='SUBMIT'
Oozie will be configured to orchestrate the request handling and job spark execution.
Once Oozie finds entry into the queue table JOB_RUN_QUEUE with status='SUBMIT' it will pull the arguments from the queue table and trigger the spark job.
It will update the status in the queue table to 'IN PROGRESS'. Upon successfull completion it will update the summary and status in the queue table.
On failure it will update the status to FAILURE.
UI will read the data from the queue table and display on UI.
Questions:
1. Is there any alternative ad better design approach.
2. D I need to have a queue mechanism to the initial request or can I leverage some inbuilt functionality?
I was trying to get the jobs count in the first time,
and I try to get it from JobprogressListener. But only stages and tasks information, not jobs. We know Spark application generate the jobs "as it goes".
But if there is a component or a class recording the job information UP FRONT?
It's possible, but I would recommend Spark RESTful API
Steps:
Get applicationId from SparkContext.applicationId property
Query http://context-url:port/applications/api/v1/[app-id]/jobs Where context-url is address of your Spark Driver and port is port with Web UI, it's 4040 normally. Here is documentation
Count jobs that are returned in response from RESTful API
I am wondering if there is a way to hook in some notifications for jobs submitted in Dataproc. We are planning to use Dataproc to run a streaming application 24/7. But Dataproc doesnt seem to have a way to notify for failed jobs.
Just wondering if Google StackDriver can be used by any means.
Thanks
Suren
Sure, StackDriver can be used to set an alert policy on a defined log-metric.
For example, you can set a Metric Absence policy which will monitor for successful job completion and alert if it's missing for a defined period of time.
Go to Logging in your console and set a filter:
resource.type="cloud_dataproc_cluster"
jsonPayload.message:"completed with exit code 0"
Click on Create Metric, after filling the details you'll be redirected to the log-metrics page where you'll be able to create an alert from the metric
As noted in the answer above, log-based metrics can be coerced to provide the OP required functionality. But, metric absence for long-running jobs would imply you would have to wait for longer than a guess at the longest job running time (and you still might get an alert if the job takes a bit longer but is not failing). What 'we' really want is a way of monitoring and alerting on job status failed, or, service completion message indicating failure (like your example), such that we are alerted immediately. Yes, you can define a Stackdriver log-based metric, looking for specific strings or values indicating failure, and this 'works', but metrics are measures that are counted, for example 'how many jobs failed', and require inconvenient workarounds to turn alert-from-metric into a simple 'this job failed' alert. To make this work, for example, the alert filters on a metric and also needs to specify a mean aggregator over an interval to fire an alert. Nasty :(