I want to be able to select a file or dir based on the TASK_ID.
Ideally like this
sbatch -o %a.log --array=1-10 script.sh data/%a
But %a is only meant to be used for log files, and can't be passed to the script at runtime.
In the docs $SLURM_ARRAY_TASK_ID is brought up. But that requires modifying the script because only then will that variable be set (not when I submit the job with sbatch)!
Is there really no nice way to do this?
Related
I have many JMeter scripts that are testing different scenarios and I want to create a configuration file for some common values across my scripts, so I will not have to edit each one to change let's say the Number of Threads.
I tried the following method but without success: I created a CSV file to contain the number of threads and passed this variable inside the Thread Group.
There is no error, but the script is not starting, always showing: "Starting 0 threads for group Thread Group."
Is there a way to set the Number of Threads from an external file?
It is, you need to create a .properties file like settings.properties and define the values there for example:
number.of.threads=100
number.of.loops=10
etc.
And refer the properties in the Thread Group via __P() function:
- ${__P(number.of.threads,)}
- ${__P(number.of.loops,)}
- etc.
Once done you can pass the file to JMeter using -q command line argument:
jmeter -q settings.properties -n -t test.jmx -l result.jtl
You can also override the values from the command-line using -J command-line argument like:
jmeter -Jnumber.of.threads=10 -n -t test.jmx -l result.jtl
More information: Apache JMeter Properties Customization Guide
For me that solution it returns error when using the properties file, using the args in command line it is ok.
How does jmeter knows that the values come from that properties file?
I am working on a mac.
enter image description here
I'm trying to have data (generated by an application only after its launch) persisted across multiple invocations of an application, but only when they're started from the same shell session.
One possible way to do that would be to pass the data back from the application to the calling shell, but since environment variable changes are only passed from parent to child, I don't know how to implement that.
Practical example:
There is job command that create subdirectory with current datetime and does work inside. Sometimes job needs to be killed and restarted, so it need directory where if finished, like job --resume 21Fri_1849/data. I would like to save 21Jan_1849/data so I don't have to check and type it each time I need to resume job. If I created something like .last_job, and wanted to restart job in another session, it could resume wrong (last) job, so files are not solution (AFAIK).
How can this be done?
Since you're only trying to target Linux, there are a fair number of tricks available here. Consider this one:
#!/usr/bin/env bash
current_boot_id=$(</proc/sys/kernel/random/boot_id)
# honor myprog_shell_pid if set and valid, fall back to PPID otherwise
if [[ $myprog_shell_pid ]] && [[ -e /proc/$myprog_shell_pid/stat ]]; then
parent_pid=$myprog_shell_pid
else
parent_pid=$PPID
fi
parent_start_time=$(awk '{print $22}' "/proc/$parent_pid/stat")
mkdir -p "$HOME/.cache/myscript-sessions"
data=$HOME/.cache/myscript-sessions/${current_boot_id}:${parent_pid}:${parent_start_time}
Now, we have a data file name that changes:
When we're rebooted (because current_boot_id is updated)
If we're run from a different shell (because our PPID changes).
If we're run from a different shell with the same PID (because the start time for the parent PID will be different).
...and you can easily delete files with the wrong boot id (because the system rebooted), or with names that refer to PID/start-time combinations that don't exist.
One caveat is that by default, this is sensitive to being called by subshells (output=$(./yourprog) will have a different PPID than ./yourprog will), but if the parent shell runs export myprog_shell_pid=$$, that issue goes away.
You're crossing over to where you need a simple job management engine instead of just shell. Using 'make' and writing Makefiles is the probably the simplest way to set this up. You can write a rule that tells how to turn a stage 1 file into a stage 2 file based on file extension, and then make will know how far things got and how to resume next time you run it.
I really like being able to run Groovy scripts in Hudson (or Jenkins, but I use Hudson).
For example, see my question In Groovy, how do I get the list of parameter names for a given job? Hudson parameter names question][1]
The thing is, now I'd like use these Groovy scripts like a keyboard macro or utility. I want to be visiting one of my jobs, hit the special keystroke, and voila, the Groovy script is run. I leave it as an exercise for myself to pick up parameters from environment of current job.
Does anybody out there do this sort of thing, and if so, what strategy has been useful. So far, all I know how to do is bring up the script console, pasted in my code, edit the variable containing the name of the Hudson job, and hit "run". Kinda klunky. Suggestions appreciated.
You can use jenkins Remote access method to do this. The Jenkins wiki describes how to use Remote access:
User can execute groovy scripts remotely sending post request to
/script/ url or /scriptText/ to have response returned without the
html wrapping.
$ curl -d "script=<your_script_here>" http://jenkins/script
$ # or
$ curl -d "script=<your_script_here>" http://jenkins/scriptText
Also, Jenkins CLI offers the possibility to execute groovy
scripts remotely using groovy command or execute groovy interactivelly
via groovysh.
When I run my shell script so much of java error handling messages get displayed every single time I ran it.
I want an option of not display it to the screen for now since I am testing.
Is there a way to do this?
the way I have it right now is:
./load_data.sh TCOMPASS/Protocol >> load_semantic.log
Redirect standard error:
./load_data.sh TCOMPASS/Protocol >> load_semantic.log 2>/dev/null
My goal is to run a bash script automatically whenever any new file is added to a particular directory or any subdirectory of that particular directory.
Detail Scenario:
I am creating an automated process for file submission from teachers to students and vice versa. Sender will upload file and it will be stored inside the Uploads directory in the LAMP server in the format, ex. "name_course-name_filename.pdf". I want some method so that when any file stored inside the Uploads folder, the same time a script will be called and send that file to the list of receives.
From the database I can find the list of receiver for that particular course and student.
The only concern of mine is, how to call a script automatically and make it work on individual file whenever the content of the directory changes. Cron will do in intervals but not a real time work.
Linux provides a nice mechanism for that purpose which is called inotify. inotify is mostly available as a C API. But there have been developed shell utilities as well. You should use inotifywait from inotifytools (pkg name in debian) for this. Here comes a basic example:
#!/bin/bash
directory="/tmp" # or whatever you are interested in
inotifywait -m -e create "$directory" |
while read folder eventlist eventfile
do
echo "the following events happened in folder $folder:"
echo "$eventlist $eventfile"
done
Update:
If the problem goes complicated, for example you'll have to monitor recursive, dynamic directory structures, you should have a look at incron It's a cron like daemon which executes scripts on certain events. But the events are file system events rather than timer events.
There is another option to 'inotifywait':
-d --daemon
Same as --monitor, except run in the background logging events to a file
that must be specified by --outfile. Implies --syslog.
For completeness:
-m --monitor
Instead of exiting after receiving a single event, execute indefinitely.
The default behaviour is to exit after the first event occurs.
Within the do-done block of your 'while' statement, you might parse each event report for interesting details then use 'case-esac' to take action based on each event that you care about.
For something that you plan to rely on for your operations, you might also consider replacing the hard-coded '$directory' with some sort of configuration file. Such a file might include the path and filename, the interesting events for that path and file, and a script to run when those events happened.
The script might take the list of events as parameters and then 'case-esac' again.
Just one man's ramblins,
~~~ 8d;-Dan