I have been having trouble for a while with two recurring jobs (the ones at the top of this list) that don't get run even though they are scheduled.
I can trigger them just fine, and they get rescheduled, but when the schedule time comes around they don't run, and the "Next Execution" time just slips into the past.
Now there are a bunch of other jobs that are having the same problem. These are supposed to run hourly, but if they get past the schedule time they just don't run.
Visiting the dashboard makes no difference. The web app is always on. Hangfire will never run these jobs unless I trigger them manually. Jobs that AREN'T in this state still run just fine as scheduled every day or every hour.
What would cause this?
My hangfire instance (version 1.7.6) is in an Azure WebApp that is set to be always running. It uses an Azure-sql database for its data store.
Here's my Bootstrapper.cs code:
using System.Configuration;
using System.Web.Hosting;
using Hangfire;
namespace MyApi
{
public class HangfireBootstrapper : IRegisteredObject
{
public static readonly HangfireBootstrapper Instance = new HangfireBootstrapper();
private readonly object _lockObject = new object();
private bool _started;
private BackgroundJobServer _backgroundJobServer;
private HangfireBootstrapper()
{
}
public void Start()
{
lock (_lockObject)
{
if (_started) return;
_started = true;
HostingEnvironment.RegisterObject(this);
var jobOptions = new BackgroundJobServerOptions();
jobOptions.ServerName = ConfigurationManager.AppSettings.Get("hangfire:servername");
jobOptions.Queues = new[] {"k1"};
GlobalConfiguration.Configuration
.UseSqlServerStorage("Kdb");
_backgroundJobServer = new BackgroundJobServer(jobOptions);
}
}
public void Stop()
{
lock (_lockObject)
{
if (_backgroundJobServer != null)
{
_backgroundJobServer.Dispose();
}
HostingEnvironment.UnregisterObject(this);
}
}
void IRegisteredObject.Stop(bool immediate)
{
Stop();
}
}
}
Here's code that is used to queue the majority of the jobs:
jobMgr.AddOrUpdate($"Script.{i1}.{name}", Job.FromExpression(() => HangfireJobs.ReplayQueue.EnqueueScript(scriptId, i1, null)),cronExpression);
Here's how my EnqueueScript method is defined:
[Queue("k1")]
public static void EnqueueScript(Guid scriptId, int env, PerformContext context)
{
try
{
...
This issue was resolved by upgrading Hangfire to version 1.7.8.
The Hangfire bug report is viewable at https://github.com/HangfireIO/Hangfire/issues/1459
It appears to have been introduced around version 1.7.4 (maybe earlier, but certainly by then) and fixed with version 1.7.8.
Related
i am making cron job like loop to do something using new thread.
when module stop, this thread keeps running, so when i deployed updated module, i'm afraid it will make duplicate thread doing similar task
#Component(immediate = true, service = ExportImportLifecycleListener.class)
public class StaticUtils extends Utils{
private StaticUtils() {}
private static class SingletonHelper{
private static final StaticUtils INSTANCE = new StaticUtils();
}
public static StaticUtils getInstance() {
return SingletonHelper.INSTANCE;
}
}
public class Utils extends BaseExportImportLifecycleListener{
public Utils() {
startTask();
}
protected Boolean CRON_START = true;
private void startTask() {
new Thread(new Runnable() {
public void run() {
while (CRON_START) {
System.out.println("test naon bae lah ");
}
}
}).start();
}
#Deactivate
protected void deactivate() {
CRON_START = false;
System.out.println(
"cron stop lah woooooooooooooooooy");
}
}
i'm using liferay 7
I have populated task that i store from db, so this thread is checking is there a task that it must do, then if it exist execute it.
I'm quite new in osgi and liferay. i've try to use scheduler and failed and also exportimportlifecycle listener but dont really get it yet
think again: Do you really need something to run all the time in the background, or do you just need some asynchronous processing in the background, when triggered? It might be better to start a background task as a one-off, that automatically terminates
Liferay provides an internal MessageBus, that you can utilize to listen to events and implement background processing, without the need for a custom thread
You're in the OSGi world, so you can utilize #Activate, #Modified, #Deactivate (from org.osgi.service.component.annotations) or use a org.osgi.framework.BundleActivator.
But, in general, it's preferable if you don't start your own thread
I am maintaining a legacy Cloud Services application hosted on Azure targeting .net 4.6.1. Inside the Application_Start method of the Global.asax on the Web Role we are registering an event handler for RoleEnvironment.StatusCheck however our logs are demonstrating that this event call back is never being called or triggered.
According to this blog: https://convective.wordpress.com/2010/03/18/service-runtime-in-windows-azure/ we were expecting this event to be triggered every 15 seconds and we believe this was happening however has since stopped. We expect that the stopped working around the time we installed some new DLLs into the solution (some of these dlls include: Microsoft.Rest.ClientRuntime.dll, Microsoft.Azure.Storage.Common.dll, Microsoft.Azure.Storage.Blob.dll, Microsoft.Azure.KeyVault.dll)
We've tried RDP-ing onto the VM to check the event logs but nothing obvious is there. Any suggestions on where we may be able to search for clues?
It seems your event handler is not registered. Try below code with a different approach:
public class WorkerRole : RoleEntryPoint
{
public override bool OnStart()
{
RoleEnvironment.StatusCheck += RoleEnvironmentStatusCheck;
return base.OnStart();
}
// Use the busy object to indicate that the status of the role instance must be Busy
private volatile bool busy = true;
private void RoleEnvironmentStatusCheck(object sender, RoleInstanceStatusCheckEventArgs e)
{
if (this.busy)
{
// Sets the status of the role instance to Busy for a short interval.
// If you want the role instance to remain busy, add code to
// continue to call the SetBusy method
e.SetBusy();
}
}
public override void Run()
{
Trace.TraceInformation("Worker entry point called", "Information");
while (true)
{
Thread.Sleep(10000);
}
}
public override void OnStop()
{
base.OnStop();
}
}
I am working with an Azure Service Bus Queue (or potentially a topic if required), and would like to know how a Web Job can be used with the Queue.
When a message comes onto the queue it represents a process that will run within the web job (or be started from the webjob). This process might be quick, 30 seconds, or it might be slow, 1 hour etc.
Can I use a single Web Job for this and somehow say that it should be running no more than 10 of these processes at a time?
Yes you can use a WebJob. I have created a simple WebJob with Storage Queue to just guide how it can be done. The below workflow will run only ten process at a time and keep all the other requests in memory of ConcurrentQueue. You will have to implement the logic to dequeue it and consume it
public class Functions
{
public delegate void CompletedProcessHandler(object sender, CompletedProcessHandlerArgs args);
static readonly Dictionary<int, CustomProcess> _dictionary =
new Dictionary<int, CustomProcess>();
static readonly ConcurrentQueue<ProcessEntity> _remaining =
new ConcurrentQueue<ProcessEntity>();
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public static void ProcessQueueMessage([QueueTrigger("testqueue")] ProcessEntity msg,
TextWriter log)
{
if (_dictionary.Count <= 10)
{
var newProcess = new CustomProcess((_dictionary.Last().Key) + 1,
msg.Duration);
}
else
{
_remaining.Enqueue(msg);
}
}
public static void CompletedProcess(object sender, CompletedProcessHandlerArgs args)
{
_dictionary[Int32.Parse(args.ProcessID)].Dispose();
_dictionary.Remove(Int32.Parse(args.ProcessID));
}
}
public class CustomProcess : IDisposable
{
public event Functions.CompletedProcessHandler OnProcessCompleted;
private CancellationTokenSource _token;
private string _id;
private Timer _timer;
public CustomProcess(int i, int duration)
{
_timer = new Timer { Enabled = true, Interval = duration * 1000 };
_timer.Elapsed += Timer_Elapsed;
_id = i.ToString();
_token = new CancellationTokenSource();
Task.Factory.StartNew(() => WriteMessages());
_timer.Start();
OnProcessCompleted += Functions.CompletedProcess;
}
private void Timer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
_token.Cancel();
OnProcessCompleted?.Invoke(this, new CompletedProcessHandlerArgs(_id));
}
private void WriteMessages()
{
while (!_token.Token.IsCancellationRequested)
{
Console.WriteLine("Test Message from process " + _id);
}
}
public void Dispose()
{
_token.Dispose();
_timer.Dispose();
}
}
public class CompletedProcessHandlerArgs : EventArgs
{
public string ProcessID { get; set; }
public CompletedProcessHandlerArgs(string ID)
{
ProcessID = ID;
}
}
public class ProcessEntity
{
public int Duration { get; set; }
}
In the app.config of the web job you need to provide the two app settings
<add name="AzureWebJobsDashboard"
connectionString="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" />
<add name="AzureWebJobsStorage"
connectionString="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" />
The Program file is the default one from the Visual Studio template
public class Program
{
// Please set the following connection strings in app.config for this WebJob to run:
// AzureWebJobsDashboard and AzureWebJobsStorage
static void Main()
{
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
}
WebJob will keep dequeue the message the moment it comes. Since you want only 10 to run at a time you will have to enqueue the message in memory and wait for running process to complete before you start a new one
As #Rick has mentioned you can set the is_Singleton property to true in settings.job file of the web job
Yes, you can trigger a web job with an Azure Service Bus Queue or Topic. A good example to look at to get you going would be the Service Bus quick start project template in Visual Studio.
In particular, you want to look at the ServiceBusTrigger attribute that the Web Jobs SDK provides.
As for the scalability of the web job, this will scale according to your web app instances. So, if you had say 5 instances of your web app with always on enabled, then you would have 5 instances of your web job. As an additional comment on this, if you wanted just one instance of the web job in an environment of 5 web app instances, then you could set the is_singleton property to true in the settings.job file.
I'm new to threading; in fact I'm not even trying to multi- thread the Windows Forms app I'm working on, but all of my searches on this issue lead me to the topic of multithreading. When debugging in Visual Studio 2010 Express, it seems to "jump around" to use the term I've seen others use to describe the same problem. When I let it run, sometimes it runs as expected, other times it just seems to keep running, getting hung up.
In trying to hone my question, I think I need to figure out:
If the timer class calls a method on a different thread, and there isn't an obvious danger of unpredictable instance values/ state corruption in the executing code (there aren't any conditional checks of instance variables etc), why would that method called by the timer appear to behave unpredictably? To me it seems that the code should run synchronously, and if a different thread is used for part of the process, so be it. I can't see where there is opportunity for thread corruption.
When the program starts, it prompts for the timer to be set to run a data download process. After the procedure runs, the timer is set again to a default time, at the end of the procedure. Consistently, the initial timer setting works, and fires as expected, running the data download process... it's that data download method, somewhere within it it goes awry. The last line of code is what sets the timer again, but I can't tell if it's getting hit while debugging it. (jumping around)..
I've added relevant code below... and I stepped into every procedure in my code from the beginning... they all show current thread id 10. This is up to an including the timer firing off, and stopping at a breakpoint at the very next line to execute, which is the data download process. The current thread at that point: 14. I've built the solution before running it/ trying to debug btw. Any ideas?
public partial class frmTradingAppMain : Form
{
private TradingAppDataRunManager drm;
private void frmTradingAppMain_Shown(object sender, EventArgs e)
{
drm = new TradingAppDataRunManager();
drm.StatusChanged += new DataRunManager.DRMStatusChangeHandler(UpdateFormData);
drm.InitializeOrScheduleDataRun();
}
private void UpdateFormData()
{
this.Invoke(new DataRunManager.DRMStatusChangeHandler(UpdateFormDataImpl));
}
private void UpdateFormDataImpl()
{
lblDataDwnLoadManagerStatus.Text = Convert.ToString(drm.Status);
if (drm.Status == DataRunManager.DRMStatus.Inactive)
{
lblNextScheduledDataDownloadDate.Text = "Date not set.";
lblNextScheduledDataDownloadTime.Text = "Time not set.";
}
else
{
lblNextScheduledDataDownloadDate.Text = drm.DateTimeOfNextScheduledDataRun.ToShortDateString();
lblNextScheduledDataDownloadTime.Text = drm.DateTimeOfNextScheduledDataRun.ToShortTimeString();
}
}
}
public abstract class DataRunManager
{
protected DataRunTimer dataRuntimer;
public delegate void DRMStatusChangeHandler();
public event DRMStatusChangeHandler StatusChanged;
public DRMStatusChangeHandler statusChanged;
public void InitializeOrScheduleDataRun()
{
if (DataRunIsAvailable() && UserWouldLikeToPerformDataRun())
RunMainDataProcedure(null);
else
ScheduleDataRun();
}
public void RunMainDataProcedure(object state)
{
start = DateTime.Now;
Status = DRMStatus.Running;
StatusChanged();
GetDataCollections();
foreach (DataCollection dcl in dataCollectionList)
{
dcl.RunDataCollection();
dcl.WriteCollectionToDatabase();
}
PerformDBServerSideProcs();
stop = DateTime.Now;
WriteDataRunStartStopTimesToDB(start, stop);
SetDataRunTimer(DateTimeOfNextAvailableDR());
}
public void ScheduleDataRun()
{
FrmSetTimer frmSetTimer = new FrmSetTimer(DateTimeOfNextAvailableDataRun);
DateTime currentScheduledTimeOfNextDataRun = DateTimeOfNextScheduledDataRun;
DRMStatus currentStatus= Status;
try
{
frmSetTimer.ShowDialog();
DateTimeOfNextScheduledDataRun = (DateTime)frmSetTimer.Tag;
SetDataRunTimer(DateTimeOfNextScheduledDataRun);
}
catch
{
Status = currentStatus;
DateTimeOfNextScheduledDataRun = currentScheduledTimeOfNextDataRun;
}
}
}
public class DataRunTimer
{
System.Threading.Timer timer;
public DataRunTimer(){}
public void SetNextDataRunTime(TimerCallback timerCallback, DateTime timeToSet)
{
if (timer == null)
timer = new System.Threading.Timer(timerCallback);
TimeSpan delayTime = new TimeSpan(timeToSet.Day - DateTime.Now.Day, timeToSet.Hour - DateTime.Now.Hour, timeToSet.Minute - DateTime.Now.Minute,
timeToSet.Second - DateTime.Now.Second);
TimeSpan intervalTime = new TimeSpan(0, 0, 10);
timer.Change(delayTime, intervalTime);
}
public void DataRunTimerCancel()
{
if (timer != null)
timer.Dispose();
}
}
I wrote about this topic in another question.
However, I've since refactored my code to get rid of configuration access, thus allowing the specs to pass. Or so I thought. They run fine from within Visual Studio using TestDriven.Net. However, when I run them during rake using the mspec.exe tool, they still fail with a serialization exception. So I've created a completely self-contained example that does basically nothing except setup fake security credentials on the thread. This test passes just fine in TD.Net, but blows up in mspec.exe. Does anybody have any suggestions?
Update: I've discovered a work-around. After researching the issue, it seems the cause is that the assembly containing my principal object is not in the same folder as the mspec.exe. When mspec creates a new AppDomain to run my specs, that new AppDomain has to load the assembly with the principal object in order to deserialize it. That assembly is not in the same folder as the mspec EXE, so it fails. If I copied my assembly into the same folder as mspec, it works fine.
What I still don't understand is why ReSharper and TD.Net can run the test just fine? Do they not use mspec.exe to actually run the tests?
using System;
using System.Security.Principal;
using System.Threading;
using Machine.Specifications;
namespace MSpecTest
{
[Subject(typeof(MyViewModel))]
public class When_security_credentials_are_faked
{
static MyViewModel SUT;
Establish context = SetupFakeSecurityCredentials;
Because of = () =>
SUT = new MyViewModel();
It should_be_initialized = () =>
SUT.Initialized.ShouldBeTrue();
static void SetupFakeSecurityCredentials()
{
Thread.CurrentPrincipal = CreatePrincipal(CreateIdentity());
}
static MyIdentity CreateIdentity()
{
return new MyIdentity(Environment.UserName, "None", true);
}
static MyPrincipal CreatePrincipal(MyIdentity identity)
{
return new MyPrincipal(identity);
}
}
public class MyViewModel
{
public MyViewModel()
{
Initialized = true;
}
public bool Initialized { get; set; }
}
[Serializable]
public class MyPrincipal : IPrincipal
{
private readonly MyIdentity _identity;
public MyPrincipal(MyIdentity identity)
{
_identity = identity;
}
public bool IsInRole(string role)
{
return true;
}
public IIdentity Identity
{
get { return _identity; }
}
}
[Serializable]
public class MyIdentity : IIdentity
{
private readonly string _name;
private readonly string _authenticationType;
private readonly bool _isAuthenticated;
public MyIdentity(string name, string authenticationType, bool isAuthenticated)
{
_name = name;
_isAuthenticated = isAuthenticated;
_authenticationType = authenticationType;
}
public string Name
{
get { return _name; }
}
public string AuthenticationType
{
get { return _authenticationType; }
}
public bool IsAuthenticated
{
get { return _isAuthenticated; }
}
}
}
Dan,
thank you for providing a reproduction.
First off, the console runner works differently than the TestDriven.NET and ReSharper runners. Basically, the console runner has to perform a lot more setup work in that it creates a new AppDomain (plus configuration) for every assembly that is run. This is required to load the .dll.config file for your spec assembly.
Per spec assembly, two AppDomains are created:
The first AppDomain (Console) is created
implicitly when mspec.exe is
executed,
a second AppDomain is created by mspec.exe for the assembly containing the specs (Spec).
Both AppDomains communicate with each other through .NET Remoting: For example, when a spec is executed in the Spec AppDomain, it notifies the Console AppDomain of that fact. When Console receives the notification it acts accordingly by writing the spec information to the console.
This communiciation between Spec and Console is realized transparently through .NET Remoting. One property of .NET Remoting is that some properties of the calling AppDomain (Spec) are automatically included when sending notifications to the target AppDomain (Console). Thread.CurrentPrincipal is such a property. You can read more about that here: http://sontek.vox.com/library/post/re-iprincipal-iidentity-ihttpmodule-serializable.html
The context you provide will run in the Spec AppDomain. You set Thread.CurrentPrincipal in the Because. After Because ran, a notification will be issued to the Console AppDomain. The notification will include your custom MyPrincipal that the receiving Console AppDomain tries to deserialize. It cannot do that since it doesn't know about your spec assembly (as it is not included in its private bin path).
This is why you had to put your spec assembly in the same folder as mspec.exe.
There are two possible workarounds:
Derive MyPrincipal and MyIdentity from MarshalByRefObject so that they can take part in cross-AppDomain communication through a proxy (instead of being serialized)
Set Thread.CurrentPrincipal transiently in the Because
(Text is required for formatting to work -- please ignore)
Because of = () =>
{
var previousPrincipal = Thread.CurrentPrincipal;
try
{
Thread.CurrentPrincipal = new MyPrincipal(...);
SUT = new MyViewModel();
}
finally
{
Thread.CurrentPrincipal = previousPrincipal;
}
}
ReSharper, for example, handles all the communication work for us. MSpec's ReSharper Runner can hook into the existing infrastructure (that, AFAIK, does not use .NET Remoting).