I have been trying to create a hub and register nodes using selenium grid on Jenkins CI.
I have tried creating "execute shell" and perform this process first running the selenium hub and tried registering the nodes in further steps but nothing worked. If I do it in this way. It only runs Selenium Hub but unable to register the nodes to it.
I have tried installing the selenium grid plugin for jenkins but nothing works.
Finally, I tried creating three different jobs to start and hub and register nodes to it.
Is there anyway I can do this process in a single job or Is there anyway if I run the first job which is starting the hub and then automatically the other two jobs should start the process.
Starting hub and registering node on Jenkins server is one-time process which you can do from terminal.
Or
In Jenkins execute shell section try below commands:
To start grid hub
java -jar selenium-server-standalone-2.53.0.jar -role hub -timeout 300000 &
// do not forget to add "&" at the end to run this process in the background.
To register node
java -jar selenium-server-standalone-2.53.0.jar -role node -hub http://localhost:4444/grid/register &
I don't think you can run a Selenium Grid from Jenkins UNLESS the grid is ran in the foreground of a user session so that there is a "space" to run the browsers within. It probably wont work if you run the grid as a background process. You didn't say if your using Linux or Windows, but in either case, you'll have the same problem I think.
Related
Killing Spark job using command Prompt
This is the thread that I hoped would answer my question. But all four answers explain how to kill the entire application.
How can I stop a job? Like a count for example?
I can do it from the Spark Web UI by clicking "kill" on the respective job. I suppose it must be possible to list running jobs and interact with them also directly via CLI.
Practically speaking I am working in a Notebook with PySpark on a Glue endpoint. If I kill the application the entire endpoint dies and I have to spin up a new cluster. I just want to stop a job. Cancelling it within the Notebook will just detach synchronization and the job keeps running, blocking any further commands from being executed.
Spark History Server provides REST API interface. Unfortunately, it only exposes monitoring capabilities for applications, jobs, stages, etc.
There is also a REST Submission interface that provides capabilities to submit, kill and check up on status of the applications. It is undocumented AFAIK, and is only supported on Spark standalone and Mesos clusters, no YARN. (Thats why there is no "kill" link in Jobs UI screen for Spark on YARN, I guess.)
So you can try using that "hidden" API, but if you know your application's Spark UI URL and job id of a job you want to kill, the easier way is something like:
$ curl -G http://<Spark-Application-UI-host:port>/jobs/job/kill/?id=<job_id>
Since I don't work with Glue I'd be interested to find out myself how its going to react, because the kill normally results in org.apache.spark.SparkException: Job <job_id> cancelled.
building on the answer by mazaneicha it appears that, for Spark 2.4.6 in standalone mode for jobs submitted in client mode, the curl request to kill an app with a known applicationID is
curl -d "id=<your_appID>&terminate=true" -X POST <your_spark_master_url>/app/kill/
We had a similar problem with people not disconnecting their notebooks from the cluster and hence hogging resources.
We get the list of running applications by parsing the webUI. I'm pretty sure there's less painful ways to manage a Spark cluster..
list the job in linux and kill it.
I would do
ps -ef |grep spark-submit
if it was started using spark-submit. Get the PID from the output and then
kill -9
Kill running job by:
open Spark application UI.
Go to jobs tab.
Find job among running jobs.
Click kill link and confirm.
For example if a user submits a job, and the workers are running as some app acount, the jobs run as the app. Often it seems the cleanup step (removing tmp files) then tries to run locally as the user and you see errors. This isn't the only problem.
Is there some way of running as another other user in Standalone?
I'm trying to use pm2 to deploy my nodejs application.
I have some cron jobs that should get executed only once. With PM2 cluster mode, I'm able to scale my application across multiple CPUs, but all of them will execute my cron jobs.
Just like the nodejs API has cluster.isMaster or cluster.isWorker, is there a way I can ask only the PM2 master to execute it?
I'm setting up Titan graph database for the first time in a production environment on Debian virtual machines, and I am utilising Rexster to provide the interface into Titan. However after googling around I cannot find any scripts to allow rexster to run as a daemon in the background. As per titan rexster with external cassandra instance I have split off Cassandra, Elasticsearch, and Rexster to start as their own processes. Cassandra and Elasticsearch conveniently have Debian packages that deploy the daemon scripts out of the box, however there is nothing for Rexster. Has anyone made a script that allows Rexster to run as a daemon?
Looking at the rexster.sh script in titan download zip ../$titan_base/bin/ it calls java to start Rexster up, so I'm thinking that some kind of wrapper like JSVC could be used to start it up, unless there is an easier way?
A simple, generic tool to handle this is Daemonize. More details in this post.
If your Debian is new enough to be using Systemd, look into creating a service script. The key commands for using your script would be:
systemctl start rexster.service
systemctl enable rexster.service
I have a job running Integration Test on Slaves with label "IntegrationTest".
When the Job started, I use Node.setLabelString(String) to modify the Slave Label from IntegrationTest to "_out_IntegrationTest", in order to block this slave from running next round integration test, because we need to revert this slave to clean environment before integration test.
The problem is that the following Integration Test job in queue still can take this slave to run even though its label was set to "_out_IntegrationTest" by Node.setLabelString(String).
I am not sure the Label is modified, but queue is not aware of this.
When I modify slave Label to "_out_IntegrationTest" manually from Web UI, the Integration Test Job will not run on this slave.
When Slave Label modified by Node.setLabelString(String), the Job still can run on it.
Note:
Queue.maintain() is called after Node.setLabelString(String) invoked.
I have a similar situation in my workplace where I need to reboot a slave after running a job. So I have a separate job that reboots the slave it runs on. As the first build step in the main job, I request to schedule a parameterized build of the reboot job, using the parameter "build on the same node". The reboot job is configured with a very high priority (via Priority Sorter Plugin) so that it will be guaranteed to be the next job run on that node. Then I have the other build steps of the main job to perform its required tasks.
When the main job finishes, the reboot job comes in and reboots the slave - here you could perform the reversion of your slave to the clean environment.