I am using spark 1.6.2, and I am using the hidden REST API (http://arturmkrtchyan.com/apache-spark-hidden-rest-api).
How can I to get all jobs status in one rest call instead of using it for each - http://spark-cluster-ip:6066/v1/submissions/status/driver-20151008145126-0000 ?
Depending on exactly what you need, you can use :8080/json to get a json representing all the applications. You should see an activeapps array which has a short info on each application (including its status e.g. Running):
For example, if I open up spark-shell I get the following field in the json:
"memoryused" : 82944,
"activeapps" : [ {
"starttime" : 1484638046648,
"id" : "app-20170117022726-0113",
"name" : "Spark shell",
"cores" : 60,
"user" : "assaf",
"memoryperslave" : 27648,
"submitdate" : "Tue Jan 17 02:27:26 EST 2017",
"state" : "RUNNING",
"duration" : 26954
} ],
Note that this is basically adding /json to the UI port rather than going to the submission port.
Related
When I submit a Spark job through API /v1/submissions/create on port 6066 and check the status of it by /v1/submissions/status/{driver-id}, I only get something like this
{
"action" : "SubmissionStatusResponse",
"driverState" : "FINISHED",
"serverSparkVersion" : "3.1.2",
"submissionId" : "driver-20211226153006-0000",
"success" : true,
"workerHostPort" : "xx.xx.xx.xx:xx",
"workerId" : "worker-20211226152727-xx.xx.xx.xx-xx"
}
Is there any way to get more information about this job, such as execution time, memory used or the output of this job, etc?
I am running a Spark application on a Stand alone Spark cluster. If I check the status of the applications using the Spark REST API, I get something like the following.
curl localhost:4040/api/v1/applications
[ {
"id" : "app-20191107150648-0000",
"name" : "MyAppName",
"attempts" : [ {
"startTime" : "2019-11-07T14:06:47.372GMT",
"endTime" : "1969-12-31T23:59:59.999GMT",
"lastUpdated" : "2019-11-07T14:06:47.372GMT",
"duration" : 0,
"sparkUser" : "Username",
"completed" : false,
"appSparkVersion" : "2.4.4",
"startTimeEpoch" : 1573135607372,
"lastUpdatedEpoch" : 1573135607372,
"endTimeEpoch" : -1
} ]
}
My question is, is there any API which also shows how much the application has progressed in %. For example, if its half done, it shoud show 50%. I know its impossible to get an exact figure, but even an estimate would be fine.
You could get a list of all stages for a given app from .../applications/[app_id]/stages, and then see how many have been finished via .../applications/[app_id]/stages?status=complete (see https://spark.apache.org/docs/latest/monitoring.html#rest-api). After making a daring assumption that all stages are created equal, you can divide the size of list #2 by #1, and get a percentage.
I am setting up Spark 2.2.0 in standalone mode (https://spark.apache.org/docs/latest/spark-standalone.html) and submitting spark jobs programatically using
SparkLauncher sparkAppLauncher = new SparkLauncher(userNameMap).setMaster(sparkMaster).setAppName(appName).....;
SparkAppHandle sparkAppHandle = sparkAppLauncher.startApplication();
I do have java client program that polls Job status for the jobs submitted programatically, for which i am using following REST endpoint.
curl http://192.168.1.139:8080/json/ which provide JSON response as following,
{
"url" : "spark://192.168.1.139:7077",
"workers" : [ { "id" : "x", "host" : "x", "port" : x, "webuiaddress" : "x",
"cores" : x, "coresused" : x, "coresfree" : x, "memory" : xx,
"memoryused" : xx, "memoryfree" : xx, "state" : "x", "lastheartbeat" : x
}, { ...}, ],
"cores" : x,
"coresused" : x,
"memory" : x,
"memoryused" : x,
"activeapps" : [ ],
"completedapps" : [ { "starttime" : x, "id" : "app-xx-xxxx", "name" : "abc", "user" : "xx",
"memoryperslave" : x, "submitdate" : "x","state" : "FINISHED OR RUNNING", "duration" : x
}, {...}],
"activedrivers" : [ ],
"status" : "x"
}
In above response, I have observed state for completedapps is always FINISHED even if application fails, while on UI (http://master:8080), associated driver shows a failed state, as captured below.
Completed Applications
Application ID - app-20190925115750-0003
Name - EXPORT_TABLE%1707
Cores -
Memory per Executor
Submitted Time
User
State - FINISHED
Duration
Completed Drivers
Submission ID - driver-20190925115748-0003
Submitted Time-
Worker - worker-20190925112049-192.168.1.110-46224
State - FAILED
Cores
Memory
Referring to above example, Currently, My java client gets status for application (app-20190925115750-0003) FINISHED, even though it got failed (encountered exception) and associated driver shows "FAILED" state. I intend to show the final status in this case as FAILED.
It seems if i can co-relate, an application-id (app-20190925115750-0003) to driver-id (driver-20190925115748-0003), I can report a "FAILED" (final) status. I could not find any co-relation between them (appID --> driver ID).
Looking forward to your suggestions to resolving this or any possible approaches to achieve this.
I have also come across some hidden REST APIs like http://xx.xx.xx.xx:6066/v1/submissions/status/driver-20190925115748-0003, which seems have a limited info returned in response.
We upgraded from graylog 2.1.3 to 2.3.2 and now receive this message repeatedly. Some parts of the UI load but not Search or Streams. Alerts are still going out. Anyone now how I can fix this? Rollback seems to not work at all.
Could not apply filter [StreamMatcher] on message <d8fa4293-dc7a-11e7-bc81-0a206782e8c1>:
java.lang.IllegalStateException: index set must not be null! (stream id=5a00a043a9b2c72984c581b6 title="My Streams")
What seems to have happened is that some streams did not get the "index_set_id" added in their definition in the Streams Collection in mongo. Here is an example of a bad one:
{
"_id" : ObjectId("5a1d6bb2a9b2c72984e24dc0"),
"creator_user_id" : "admin",
"matching_type" : "AND",
"description" : "EU2 Queue Prod",
"created_at" : ISODate("2017-11-28T13:59:14.546Z"),
"disabled" : false,
"title" : "EU2 Queue Prod",
"content_pack" : null
}
I was able to add the "index_set_id" : "59bb08b469d42f3bcfa6f18e" value in and restore the streams:
{
"_id" : ObjectId("5a1d6bb2a9b2c72984e24dc0"),
"creator_user_id" : "admin",
"index_set_id" : "59bb08b469d42f3bcfa6f18e",
"matching_type" : "AND",
"description" : "EU2 Queue Prod",
"created_at" : ISODate("2017-11-28T13:59:14.546Z"),
"disabled" : false,
"title" : "EU2 Queue Prod",
"content_pack" : null
}
I faced this issue too with other version of Graylog in kubernetes environment.
I took below actions to fix this issue:
From Graylog UI under Stream menu, select more actions next to your stream, in your case its : My stream click > edit stream > select "Default index set" from drop down list.
Do it for all the available streams.
I am trying to solve a problem that occurs when inserting related nodes in Neo4j. Nodes are inserted by several threads using the standard save method of org.springframework.data.neo4j.repository.GraphRepository.
Sometimes the insertion fails when fetching a related node in order to define a relationship. The exception messages are like this: org.neo4j.graphdb.NotFoundException: '__type__' on http://neo4j:7474/db/data/relationship/105550.
Calling this URL from curl returns a JSON object which appears to have __type__ correctly defined, which suggests that the exception is caused by a race between inserting threads.
The method that originates the calls to the repository is annotated #Neo4jTransactional. What atomicity and transaction isolation does #Neo4jTransactional guarantee? And how should I use it for multi-threaded insertions?
Update:
I have now been able to see this happening in the debugger. The code is trying to fetch the node at one end of this relationship, together with all its relationships. It throws an exception because the type attribute is missing. This is the JSON initially returned.
{
"extensions" : { },
"start" : "http://localhost:7474/db/data/node/617",
"property" : "http://localhost:7474/db/data/relationship/533/properties/{key}",
"self" : "http://localhost:7474/db/data/relationship/533",
"properties" : "http://localhost:7474/db/data/relationship/533/properties",
"type" : "CONTAINS",
"end" : "http://localhost:7474/db/data/node/650",
"metadata" : {
"id" : 533,
"type" : "CONTAINS"
},
"data" : { }
}
A few seconds later, the same REST call returns this JSON:
{
"extensions" : { },
"start" : "http://localhost:7474/db/data/node/617",
"property" : "http://localhost:7474/db/data/relationship/533/properties/{key}",
"self" : "http://localhost:7474/db/data/relationship/533",
"properties" : "http://localhost:7474/db/data/relationship/533/properties",
"type" : "CONTAINS",
"end" : "http://localhost:7474/db/data/node/650",
"metadata" : {
"id" : 533,
"type" : "CONTAINS"
},
"data" : {
"__type__" : "ProductRelationship"
}
}
I can't understand why there is such a long delay between inserting the relationship and specifying the type. Why doesn't it all happen at once?