I have a flow with the logic as below:
class MyFlow(Flow):
start = flow.Start(StartProcess).Next(this.if_A_B)
if_A_B = flow.If(cond=lambda act: act.process.if_A_B). \
Then(this.A).
Else(this.B)
A = flow.View(HandleA). \
Next(this.B)
B = flow.View(HandleB). \
Next(this.if_A_C)
if_A_C = flow.If(cond=lambda act: act.process.if_A_C). \
Then(this.A). \
Else(this.C)
C = flow.View(HandleC). \
Next(this.end)
I expect that in some case: task A done -> task B done -> if_A_C is True then activate a new task A_NEW -> task A_NEW done -> task B_NEW done.
But I find after task A is done and activate task B, the flow_task & process of B are empty.
I try set process.if_A_B to False so that skip task A, then task B had flow_task & process.
Another try is comment if_A_C and C, so task B's next is this.end, then after task A is done and activate task B, the flow_task & process are there.
I wonder does viewflow support directly jumping to previous task? Or I should cancel B then undo A to go from B back to A?
Related
I have a question in multithreading in python (Python 3.7).
I have 2 sets of processes, 12 each in each set. Let's name the processes in set 1 as {P1, P2, P3,...,P12}. Similarly in set 2, we have processes {X1, X2, X3,...,X12}.
Now, I want to run 4 jobs at max in set 1 jobs. So at any given point of time, only a maximum of 4 jobs should be running.
In set 2, I want jobs to be dependent on set 1 jobs. So for example X1 should run only after P1 is completed, but I don't want to explicitly cap number of processes in set 2 (unlike set 1 restriction).
So let's say P jobs are data formatting jobs, and X are jobs for sending the data to some system. So I don't want to run more than 4 data formatting jobs at a time, but no such restriction for processes sending the data.
I tried ThreadPool in the below way:
with ft.ThreadPoolExecutor(max_workers=4) as executor:
future1 = executor.submit(P1) #P1
future2 = executor.submit(P2) #P2
future3 = executor.submit(P3) #P3
future4 = executor.submit(P4) #P4
future5 = executor.submit(P5) #P5
future6 = executor.submit(P6) #P6
future7 = executor.submit(P7) #P7
future8 = executor.submit(P8) #P8
future9 = executor.submit(P9) #P9
future10 = executor.submit(P10) #P10
future11 = executor.submit(P11) #P11
future12 = executor.submit(P12) #P12
future1.add_done_callback(X1)
future2.add_done_callback(X2)
future3.add_done_callback(X3)
future4.add_done_callback(X4)
future5.add_done_callback(X5)
future6.add_done_callback(X6)
future7.add_done_callback(X7)
future8.add_done_callback(X8)
future9.add_done_callback(X9)
future10.add_done_callback(X10)
future11.add_done_callback(X11)
future12.add_done_callback(X12)
But I guess what the above code does is, it triggers P1 to P4 in the first shot, and then at one point of time we have {P2,P3,P4,X1} running, which I don't want. I want if there are any P jobs in the threadpool, they should run 4 at a time, and in addition to that any number of X jobs can run.
Any help please?
I am writing a job scheduler where I schedule M jobs across N co-routines (N < M). As soon as one job finishes, I add a new job so that it can start immediately and run in parallel with the other jobs. Additionally, I would like to ensure that no single job takes more than a certain fixed amount of time. Any jobs that take too long should be cancelled. I have something pretty close, like this:
def update_run_set(waiting, running, max_concurrency):
number_to_add = min(len(waiting), max_concurrency - len(running))
for i in range(0, number_to_add):
next_one = waiting.pop()
running.add(next_one)
async def _run_test_invocations_asynchronously(jobs:List[MyJob], max_concurrency:int, timeout_seconds:int):
running = set() # These tasks are actively being run
waiting = set() # These tasks have not yet started
waiting = {_run_job_coroutine(job) for job in jobs}
update_run_set(waiting, running, max_concurrency)
while len(running) > 0:
done, running = await asyncio.wait(running, timeout=timeout_seconds,
return_when=asyncio.FIRST_COMPLETED)
if not done:
timeout_count = len(running)
[r.cancel() for r in running] # Start cancelling the timed out jobs
done, running = await asyncio.wait(running) # Wait for cancellation to finish
assert(len(done) == timeout_count)
assert(len(running) == 0)
else:
for d in done:
job_return_code = await d
if len(waiting) > 0:
update_run_set(waiting, running, max_concurrency)
assert(len(running) > 0)
The problem here is that say my timeout is 5 seconds, and I'm scheduling 3 jobs across 4 cores. Job A takes 2 seconds, Job B takes 6 seconds and job C takes 7 seconds.
We have something like this:
t=0 t=1 t=2 t=3 t=4 t=5 t=6 t=7
-------|-------|-------|-------|-------|-------|-------|-------|
AAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
However, at t=2 the asyncio.await() call returns because A completed. It then loops back up to the top and runs again. At this point B has already been running for 2 seconds, but since it starts the countdown over, and only has 4 seconds remaining until it completes, B will appear to be successful. So after 4 seconds we return again, B is successful, then we start the loop over and now C completes.
How do I make it so that B and C both fail? I somehow need the time to be preserved across calls to asyncio.wait().
One idea that I had is to do my own bookkeeping of how much time each job is allowed to continue running, and pass the minimum of these into asyncio.wait(). Then when something times out, I can cancel only those jobs whose time remaining was equal to the value I passed in for timeout_seconds.
This requires a lot of manual bookkeeping on my part though, and I can't help but wonder about floating point problems which cause me to decide that it's not time to cancel a job even though it really is). So I can't help but think that there's something easier. Would appreciate any ideas.
You can wrap each job into a coroutine that checks its timeout, e.g. using asyncio.wait_for. Limiting the number of parallel invocations could be done in the same coroutine using an asyncio.Semaphore. With those two combined, you only need one call to wait() or even just gather(). For example (untested):
# Run the job, limiting concurrency and time. This code could likely
# be part of _run_job_coroutine, omitted from the question.
async def _run_job_with_limits(job, sem, timeout):
async with sem:
try:
await asyncio.wait_for(_run_job_coroutine(job), timeout)
except asyncio.TimeoutError:
# timed out and canceled, decide what you want to return
pass
async def _run_test_invocations_async(jobs, max_concurrency, timeout):
sem = asyncio.Semaphore(max_concurrency)
return await asyncio.gather(
*(_run_job_with_limits(job, sem, timeout) for job in jobs)
)
I would like to trigger several downstream builds in groovy and then make the upstream job finish without waiting for the results of the downstream jobs.
With the following code:
hudson.model.queue.QueueTaskFuture build(String fullName) {
def p = jenkins.model.Jenkins.instance.getItemByFullName(fullName)
def thisR = Thread.currentThread().executable
def f = p.scheduleBuild2(p.quietPeriod, new hudson.model.Cause.UpstreamCause(thisR))
return f
}
def f1 = build('job1')
def f2 = build('job2')
// wait for both builds to finish
def b1 = f1.get()
def b2 = f2.get()
The downstream builds must finish before the upstream job can finish. Can I force the upstream job to finish with a build status of SUCCESS while the downstream jobs continue to run?
I think you could call the URL to start the job using the remote access API
You merely need to perform an HTTP POST on
JENKINS_URL/job/JOBNAME/build?token=TOKEN where TOKEN is set up in
the job configuration.
Which can be done in groovy
def data = new URL(feedUrl).getText()
I have 3 jobs:
Debug version (after commits).
QA version (manual, timer).
Release version (triggered by 2, manual).
Builds from second job should have BUILD_NUMBER larger by one than newest build from first job, and when second job triggered third job then he should have the same BUILD_NUMBER as second. When third job will be triggered manual, then he also should have BUILD_NUMBER larger by one than newest build from first job.
I hope that what I wrote is understandable ;P
I wrote script in groovy:
import jenkins.model.*
jenkinsInstance = jenkins.model.Jenkins.instance
searchedJobName = "^synchronize_build_numbers.*"
jobName = "synchronize_build_numbers_qa"
allItems = jenkinsInstance.items
if ("${JOB_NAME}" == jobName) {
chosenJob = allItems.findAll{ job -> job.name.matches("synchronize_build_numbers_release") }
chosenJob.each{ job -> job.nextBuildNumber = "${BUILD_NUMBER}".toInteger() }
} else {
chosenJobs = allItems.findAll{ job -> job.name.matches(searchedJobName) }
buildNumber = chosenJobs.collect{ job -> job.nextBuildNumber }.max()
chosenJobs.each{ job -> job.nextBuildNumber = buildNumber }
}
return [:]
And it works, but I would like to make this script was more universal.
When working with embedded SQL in RPG, you often end up with a cursor and a dow-loop for processing all rows in your result. The condition in the loop is somehow dependent on SQLCOD and/or SQLSTT, some globally available variables in an SQLRPGLE-program?
But what is the correct way of checking these values? Some suggest SQLCOD = 0 others not (SQLCOD = +100 or SQLSTT = '02000'). One fails on all warnings, the other does not fail on some errors, so I'm not content.
To illustrate what I do with some code:
Pmain B
D PI
Dmy_ds E DS extname(SOME_TABLE)
D qualified
/free
exec sql
DECLARE cur CURSOR FOR
SELECT *
FROM some_table;
exec sql
OPEN cur;
exec sql
FETCH cur
INTO :my_ds;
dow sql_found();
exec sql
FETCH cur
INTO :my_ds;
enddo;
exec sql
CLOSE cur;
/end-free
Pmain E
Psql_found B
D PI N
/free
// insert return statement here...
/end-free
Psql_found E
I'm looking for the correct return statement here, that will make me go through all rows if no error occurs and lets me leave when an error occurs. Bonus points for some decent way to check for errors.
SQLSTATE is better, and recommended by IBM.
From IBM's InfoCenter SQL Messages and Codes Reference: SQLCODE and SQLSTATE concepts
SQLSTATE is the preferred standard return code.
SQLSTATE is 5 characters, with the first two bytes identifying a class of conditions.
'00' = Unqualified Successful Completion
'01' = Warning
'02' = No Data
Anything else is an error. I generally only check for '00'.
Simple. Easy. More portable.
Using SQLCODE often involves lists of codes which are, IMHO, less than developer friendly.
Example:
Personally, I generally include definitions and code like this:
D xSQLState# s * inz( %addr(SQLState) )
D xSQLState ds 5 based(xSQLState#)
D xSQLState2 2a
D
D Success_On_SQL C const('00')
D Warning_On_SQL C const('01')
D NoData_On_SQL C const('02')
Then after any SQL operation, I generally check
if xSQLState2 <> Success_On_Sql;
someflag = true;
endif;
The best practice is to process the SQLCODEs you expect (as part of the expected processing) and to add exception code to handle the ones you don't. One implementation:
dow 1=1; // forever
exec sql
FETCH cur
INTO :my_ds;
// normal exit
if sqlstt = SQL_NODATA;
SFLEND = *on;
leave;
endif;
// can't CAST a value
if sqlstt = SQL_CAST; // CAST error
... tell user there's an error and read another
iter;
endif;
// decimal data error
if sqlstt = SQL_DDE;
tell user to call IT and stop reading
leave;
endif;
// whoops! not expected at all. Dump for post-mortem
if sqlstt <> SQL_NORMAL;
... tell user to call IT and stop reading
dump(a);
leave;
endif;
// test for end of loop
// filled subfile page?
enddo; // forever
With this type of implementation you have to intentionally leave the loop; whether you've filled a subfile page, loaded the highest element in an array or hit an error. I'm not sure there is a single, generic implementation that will handle all circumstances. Sometimes you might want to leave the read loop if you have a record lock and sometimes you want to issue a message and try again (for example).
I did some more searching on the topic and found something on IBM's site (quote):
The SQLCODE is also set by the database manager after each SQL
statement is executed as follows:
- If SQLCODE = 0 and SQLWARN0 is blank, execution was successful.
- If SQLCODE = 100, no data was found. For example, a FETCH
statement returned no data, because the cursor was positioned
after the last row of the result table.
- If SQLCODE > 0 and not = 100, execution was successful with a
warning.
- If SQLCODE = 0 and SQLWARN0 = 'W', execution was successful
with a warning.
- If SQLCODE < 0, execution was not successful.
Which would lead me to an sql_found() like this:
Pfound_sql B
D PI N
/free
return (SQLCOD >= 0) and (SQLCOD<>100);
/end-free
Pfound_sql E
That should take care of the End of Data condition and fail on all errors. I'm not sure if there are some warnings that I should take care of (don't want to get trapped in an endless loop, if there is a warning that leads to not reading).