I use a for loop to exec() a file 300 times in pthreads. Sometimes the full 300 exec() calls are successful, but most times a few of the exec() fail and I get between 295 or 299 successful executes of the file.
The error code from the exec() always comes back at #127,
I checked file_exists on failures and it always says "file exists", but it also says "file is not executable" on failures. Which is strange because it was executable the other 295 times within the loop.
My pthread version is only 2 months old. I am very confident that I don't have any pthread workers sharing files or writing to same spots. Scratching my head with what else I can do.
class WorkerThreads extends Thread
{
private $workerId;
public function __construct($id)
{
$this->workerId = $id;
}
public function run()
{
$mainfile="/pastlotto_1/".$this->workerId."/preset_pthread.php";
for($x=0; $x<100; $x++)
{
for($o=0; $o<3; $o++)
{
$command="php $mainfile";
$findE=exec($command,$output,$return_var);
if($return_var !== 0){
echo " file not executed at worker id ".$this->workerId."\n";
echo $return_var."\n";
if (file_exists($mainfile)) {
echo "The file $mainfile exists\n";
} else {
echo "The file $mainfile does not exist\n";
}
if(is_executable($mainfile))
{
echo ("file is executable\n");
}
else
{
echo ("file is not executable\n");
}
}
else{
echo "File executed Successfully";
}
}
}
}//END OF RUN
}
for($r=0; $r<1; $r++)
{
$workers = [];
// Initialize and start the threads
foreach (range(0,2) as $j) {
$workers[$j] = new WorkerThreads($j);
$workers[$j]->start();
//echo $i." worker started\n";
}
// Let the threads come back
foreach (range(0,2) as $j) {
$workers[$j]->join();
}
unset($workers);
}
I was able to solve the problem by changing the code that fires up the pthreads from:
for($r=0; $r<1; $r++)
{
$workers = [];
// Initialize and start the threads
foreach (range(0,2) as $j) {
$workers[$j] = new WorkerThreads($j);
$workers[$j]->start();
//echo $i." worker started\n";
}
// Let the threads come back
foreach (range(0,2) as $j) {
$workers[$j]->join();
}
unset($workers);
}
TO REPLACEMENT CODE:
$p = new Pool(3);
// Initialize and start the threads
for($b=0; $b<3; $b++) {
$tasks[$b]= new WorkerThreads($b);
}
// Add tasks to pool queue
foreach ($tasks as $task) {
$p->submit($task);
}
// shutdown will wait for current queue to be completed
$p->shutdown();
Related
I have a call() method in my code, which based on certain conditions calls specific methods :
call(){
if(a){
methodA();
}
if(b){
methodB();
}
if(c){
methodC();
}
}
In the above scenario, I want to limit concurrent executions for methodC.
How can this be achieved?
What you need here is a Semaphore construct (check the bouncer/night club specification in the example).
// Create the semaphore with 3 slots, where 3 are available.
var bouncer = new Semaphore(3, 3);
call(){
if(a){
methodA();
}
if(b){
methodB();
}
if(c){
// Let a thread execute only after acquiring access (a semaphore to be released).
Bouncer.WaitOne();
methodC();
// This thread is done. Let someone else go for it
Bouncer.Release(1);
}
}
If you want to limit the number of concurrent executions to at most one at a time, then you should use a Lock. In Java it should look like:
final Lock lock = new ReentrantLock();
call() {
if(a) {
methodA();
}
if(b) {
methodB();
}
if(c) {
lock.lock();
try {
methodC();
} finally {
lock.unlock();
}
}
}
If you want to limit the number of concurrent executions to more than one at a time, you can use a Semaphore; here CONCURRENT_CALLS_ALLOWED is an int.
final Semaphore semaphore = new Semaphore(CONCURRENT_CALLS_ALLOWED);
call() {
if(a) {
methodA();
}
if(b) {
methodB();
}
if(c) {
semaphore.aquire();//throws checked exception
try {
methodC();
} finally {
semaphore.release();
}
}
}
I have a requirement where I need to spawn multiple threads to execute a process. here the process is an exe. I'm facing a problem while multiple threads try to access the exe which is in a shared path.
I tried using a lock .even that din't help. Any help appreciated.Please find the code below.
Task t;
t = Task.Factory.StartNew(() =>
{
//lock (thisLock)
//{
// process = ProcessHelper.StartNew(objPSI.FileName, objPSI.Arguments, true, 5000);
// process.BeginOutputReadLine();
// process.OutputDataReceived += process_OutputDataReceived;
//}
process = ProcessHelper.StartNew(objPSI);
process.BeginOutputReadLine();
process.OutputDataReceived += process_OutputDataReceived;
//}
//var token = Task.Factory.CancellationToken;
//Task.Factory.StartNew(() =>
//{
// //this.label1.Text = "Task past first work section...";
//}, token, TaskCreationOptions.None, context);
}
)
.ContinueWith(_ =>
{
if (process.ExitCode != 0)
{
ErrorTexts.Add(process.StandardOutput.ReadToEnd());
}
else
{
ProgressText = "::: Migration Successful";
}
process.Close();
});
I know there are other classes I can use, however I want to use LWP. The accepted answer here (How do I make parallel HTTP requests in Perl, and receive them back in order?) does what I want however I need to "tag" the threads so that I can match the request with the response. I have modified the code to the following:
#Setup and query the DB:
#hold all response\id pairs
my #response_bundles;
while (my #row = $query->fetchrow_array()) {
my $id = $row[0];
my $url = $row[1];
push #threads, async {
my #container_hash;
$container_hash[0] = #row;
$container_hash[1] = $ua->get($url);
push #response_bundles, #container_hash;
};
}
#Once this loop terminates all threads are done
for my $thread (#threads) {
$thread->join;
}
#Since all threads are completed we can now sort through the response bundles
for my $response_bundle (#response_bundles) {
print "test\n";
}
My goal is to kick off a bunch of HTTP requests and store their ($id, $response) pair in an array. I push them all into an async process and the sub in async{} SHOULD be doing this (but it's not). I loop through the thread array and once that is complete all threads should be done. I then go through my bundle and do things however the "print test" is never firing. Am I thinking about this wrong?
EDIT:
Per the comment I tried returning the value but this does not work either
while (my #row = $query->fetchrow_array()) {
my $id = $row[0];
my $url = $row[1];
push #threads, async {
my #container_hash;
$container_hash[0] = #row;
$container_hash[1] = $ua->get($url);
return #container_hash;
};
}
#Once this loop terminates all threads are done
for my $thread (#threads) {
my #container;
#container = $thread->join;
print $container[1];
}
You need to return the data you need from your thread so main program can process it:
while (my #row = $query->fetchrow_array()) {
my $id = $row[0];
my $url = $row[1];
push #threads, async {
my #container_hash;
$container_hash[0] = \#row; #I think you need this
$container_hash[1] = $ua->get($url);
return \#container_hash; #and here we return reference to our data
};
}
#Once this loop terminates all threads are done
for my $thread (#threads) {
my $container = $thread->join;
print $container->[1], "\n"; #this is a reference that's why we use ->
}
I have an array which contains a list of file #arr=(a.txt,b.txt,c.txt);
I am iterating the array and processing the files with foreach loop; each line of the file will generate a sql and will run on the DB server.
I want to create one thread with each line of the file and query the DB. I also want to control the max no of threads at a time running simultaneously.
You can use a Thread::Pool based system. Or any Boss/Worker model based system.
That's just a simple worker model, an ideal scenario. No problem.
use threads;
use Thread::Queue qw( );
use constant NUM_WORKERS => 5;
sub work {
my ($dbh, $job) = #_;
...
}
{
my $q = Thread::Queue->new();
my #threads;
for (1..NUM_WORKERS) {
push #threads, async {
my $dbh = ...;
while (my $job = $q->dequeue())
work($dbh, $job);
}
};
}
while (<>) {
chomp;
$q->enqueue($_);
}
$q->enqueue(undef) for 1..#threads;
$_->join() for #threads;
}
Pass the file names to the script as arguments, or assign them to #ARGV within the script.
local #ARGV = qw( a.txt b.txt c.txt );
Interesting I manually control how many threads to run. I use Hash of the thread id
[code snip]
my %thr; #my hashes for threads
$count=1;
$maxthreads=5;
while (shift (#data) {
$syncthr = threads->create(sub {callfunction here}, {pass variables});
$tid = $syncthr->tid; #get the thread ID
$thr{$tid} = $syncthr;
if ($count >= $maxthreads) {
threads->yield();
while (1) { # loop until threads are completed
$num_run_threads = keys (%thr);
foreach $one_thread ( keys %thr ) {
if ($thr{$one_thread}->is_running() ) { # if thread running check for error state
if ($err = $thr{$one_thread}->error() } {
[ do stuff here]
}
# otherwise move on to next thread check
} else { # thread is either completed or has error
if ($err = $thr{$one_thread}->error()) {
[ check for error again cann't hurt to double check ]
}
if ($err = $thr{$one_thread}->join()) {
print "Thread completed id: [$one_thread]\n";
}
delete $thr{$one_thread}; # delete the hash since the thread is no more
$num_run_threads = $num_run_threads - 1; # reduce the number of running threads
}
} # close foreach loop
#threads = threads->list(threads::running); # get threads
if ($num_run_threads < $maxthreads ) {
$count = $num_run_threads; # reset the counter to number of threads running
if ( $#data != -1 ) { # check to make sure we still have data
last; # exit the infinite while loop
} else {
if (#threads) {
next; # we still have threads continue with processing
} else {
{ no more threads to process exit program or do something else }
}
} # end else
} # end threads running
} # end the while statement
#Check the threads to see if they are joinable
undef #threads;
#threads = threads->joinable()
if (#threads) {
foreach $mthread(#threads) {
if ($mthreads != 0) {
$thr->join();
}
} #end foreach
} #end #threads
} #end the if statement
$count++; Increment the counter to get to number of max threads to spawn
}
This is by no means a complete program. Furthermore, I have changed it to be very bland. However, I've been using this for a while with success. Especially in the OO Perl. This works for me and have quite a lot of uses. I maybe missing a few more error checking especially with timeout but I do that in the thread itself. Which by the way the thread is actually a sub routine that I am calling.
I wanted to call my custom method in the Thread.
public void LoopOverAllLists(String _webAndSiteXml)
{
try
{
XmlNode _nodelist = SharePoint.ListsGetListCollection();
foreach (System.Xml.XmlNode _item in _nodelist.ChildNodes)
{
string title = _item.Attributes["Title"].Value;
//check for hidden list
if (_item.Attributes["Hidden"].Value.ToLower() == "false")
{
switch (_item.Attributes["ServerTemplate"].Value)
{
//Check whether list is document library
case SharePoint.LIST_ID_DOCUMENT_LIBRARY:
case SharePoint.LIST_ID_XML_FORMS:
case SharePoint.Publishing_ID_Pages:
{
//Get all documents info
try
{
GetAllDocumentsInfo(_item, _webAndSiteXml);
}
catch
{
}
break;
}
//Check whether list is having attachment
case SharePoint.LIST_ID_GENERIC:
case SharePoint.LIST_ID_ANNOUNCEMENTS:
case SharePoint.LIST_ID_CONTACTS:
case SharePoint.LIST_ID_TASKS:
case SharePoint.LIST_ID_EVENTS:
case SharePoint.LIST_ID_CUSTOM_GRID:
case SharePoint.LIST_ID_MEETING_SERIES:
case SharePoint.LIST_ID_MEETING_AGENDA:
case SharePoint.LIST_ID_MEETING_ATTENDEES:
case SharePoint.LIST_ID_MEETING_DECISIONS:
case SharePoint.LIST_ID_MEETING_OBJECTIVES:
case SharePoint.LIST_ID_MEETING_TTB:
case SharePoint.LIST_ID_MEETING_WS_PAGES:
case SharePoint.LIST_ID_PORTAL_SITE_LIST:
{
//Get all list items info having attachment
try
{
GetAllListItemsInfoOnlyAttachments(_item, _webAndSiteXml);
}
catch
{
}
break;
}
default:
GetAllListItemsInfoOnlyAttachments(_item, _webAndSiteXml);
break;
}
// Get All the List Forms
try
{
GetAllListForms(title, _webAndSiteXml);
}
catch
{
}
}
}
}
catch (Exception ex)
{
throw ex;
}
}
in above method three methods which is " GetAllDocumentsInfo , GetAllListItemsInfoOnlyAttachments and GetAllListForms " I wanted to call these function using thread in C#.
Thanks
Here is how I would approach the problem. Notice that I encapsulated the contents of the foreach loop into a separate method and then queued the execution of that method into the ThreadPool so that each iteration of the loop occurs in parallel. I also use a well established pattern to wait for all pending work items to complete. This code is compatible with .NET 3.5.
public void LoopOverAllLists(String _webAndSiteXml)
{
int pending = 1; // Used to track the number of pending work items.
var finished = new ManualResetEvent(false); // Used to wait for all work items to complete.
XmlNode nodes = SharePoint.ListsGetListCollection();
foreach (XmlNode item in nodes)
{
XmlNode capture = item; // This is required to capture the loop variable correctly.
Interlocked.Increment(ref pending); // There is another work item in progress.
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
ProcessNode(capture);
}
finally
{
// Signal the event if this is the last work item to complete.
if (Interlocked.Decrement(ref pending) == 0) finished.Set();
}
}, null);
}
// Signal the event if the for loop was last work item to complete.
if (Interlocked.Decrement(ref pending) == 0) finished.Set();
// Wait for all work items to complete.
finished.WaitOne();
}
private void ProcessNode(XmlNode item)
{
// Put the contents of your loop here.
}
instead of calling
GetAllDocumentsInfo(_item, _webAndSiteXml);
use
Task.Factory.StartNew(() => GetAllDocumentsInfo(_item, _webAndSiteXml));
repeat this pattern for the other method calls as well