Azure Diagnostics wrt Custom Logs and honoring scheduledTransferPeriod - azure

Question

I have implemented my own TraceListener similar to http://blogs.technet.com/b/meamcs/archive/2013/05/23/diagnostics-of-cloud-services-custom-trace-listener.aspx .
One thing I noticed is that that logs show up immediately in My Azure Table Storage. I wonder if this is expected with Custom Trace Listeners or because I am in a development environment.
My diagnosics.wadcfg
<?xml version="1.0" encoding="utf-8"?>
<DiagnosticMonitorConfiguration configurationChangePollInterval="PT1M""overallQuotaInMB="4096" xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration">
<DiagnosticInfrastructureLogs scheduledTransferLogLevelFilter="Information" />
<Directories scheduledTransferPeriod="PT1M">
<IISLogs container="wad-iis-logfiles" />
<CrashDumps container="wad-crash-dumps" />
</Directories>
<Logs bufferQuotaInMB="0" scheduledTransferPeriod="PT30M" scheduledTransferLogLevelFilter="Information" />
</DiagnosticMonitorConfiguration>
I have changed my approach a bit. Now I am defining in the web config of my webrole. I notice when I set autoflush to true in the webconfig, every thing works but scheduledTransferPeriod is not honored because the flush method pushes to the table storage. I would like to have scheduleTransferPeriod trigger the flush or trigger flush after a certain number of log entries like the buffer is full. Then I can also flush on server shutdown. Is there any method or event on the CustomTraceListener where I can listen to the scheduleTransferPeriod?
<system.diagnostics>
<!--http://msdn.microsoft.com/en-us/library/sk36c28t(v=vs.110).aspx
By default autoflush is false.
By default useGlobalLock is true. While we try to be threadsafe, we keep this default for now. Later if we would like to increase performance we can remove this. see http://msdn.microsoft.com/en-us/library/system.diagnostics.trace.usegloballock(v=vs.110).aspx -->
<trace>
<listeners>
<add name="TableTraceListener"
type="Pos.Services.Implementation.TableTraceListener, Pos.Services.Implementation"
/>
<remove name="Default" />
</listeners>
</trace>
</system.diagnostics>
I have modified the custom trace listener to the following:
namespace Pos.Services.Implementation
{
class TableTraceListener : TraceListener
{
#region Fields
//connection string for azure storage
readonly string _connectionString;
//Custom sql storage table for logs.
//TODO put in config
readonly string _diagnosticsTable;
[ThreadStatic]
static StringBuilder _messageBuffer;
readonly object _initializationSection = new object();
bool _isInitialized;
CloudTableClient _tableStorage;
readonly object _traceLogAccess = new object();
readonly List<LogEntry> _traceLog = new List<LogEntry>();
#endregion
#region Constructors
public TableTraceListener() : base("TableTraceListener")
{
_connectionString = RoleEnvironment.GetConfigurationSettingValue("DiagConnection");
_diagnosticsTable = RoleEnvironment.GetConfigurationSettingValue("DiagTableName");
}
#endregion
#region Methods
/// <summary>
/// Flushes the entries to the storage table
/// </summary>
public override void Flush()
{
if (!_isInitialized)
{
lock (_initializationSection)
{
if (!_isInitialized)
{
Initialize();
}
}
}
var context = _tableStorage.GetTableServiceContext();
context.MergeOption = MergeOption.AppendOnly;
lock (_traceLogAccess)
{
_traceLog.ForEach(entry => context.AddObject(_diagnosticsTable, entry));
_traceLog.Clear();
}
if (context.Entities.Count > 0)
{
context.BeginSaveChangesWithRetries(SaveChangesOptions.None, (ar) => context.EndSaveChangesWithRetries(ar), null);
}
}
/// <summary>
/// Creates the storage table object. This class does not need to be locked because the caller is locked.
/// </summary>
private void Initialize()
{
var account = CloudStorageAccount.Parse(_connectionString);
_tableStorage = account.CreateCloudTableClient();
_tableStorage.GetTableReference(_diagnosticsTable).CreateIfNotExists();
_isInitialized = true;
}
public override bool IsThreadSafe
{
get
{
return true;
}
}
#region Trace and Write Methods
/// <summary>
/// Writes the message to a string buffer
/// </summary>
/// <param name="message">the Message</param>
public override void Write(string message)
{
if (_messageBuffer == null)
_messageBuffer = new StringBuilder();
_messageBuffer.Append(message);
}
/// <summary>
/// Writes the message with a line breaker to a string buffer
/// </summary>
/// <param name="message"></param>
public override void WriteLine(string message)
{
if (_messageBuffer == null)
_messageBuffer = new StringBuilder();
_messageBuffer.AppendLine(message);
}
/// <summary>
/// Appends the trace information and message
/// </summary>
/// <param name="eventCache">the Event Cache</param>
/// <param name="source">the Source</param>
/// <param name="eventType">the Event Type</param>
/// <param name="id">the Id</param>
/// <param name="message">the Message</param>
public override void TraceEvent(TraceEventCache eventCache, string source, TraceEventType eventType, int id, string message)
{
base.TraceEvent(eventCache, source, eventType, id, message);
AppendEntry(id, eventType, eventCache);
}
/// <summary>
/// Adds the trace information to a collection of LogEntry objects
/// </summary>
/// <param name="id">the Id</param>
/// <param name="eventType">the Event Type</param>
/// <param name="eventCache">the EventCache</param>
private void AppendEntry(int id, TraceEventType eventType, TraceEventCache eventCache)
{
if (_messageBuffer == null)
_messageBuffer = new StringBuilder();
var message = _messageBuffer.ToString();
_messageBuffer.Length = 0;
if (message.EndsWith(Environment.NewLine))
message = message.Substring(0, message.Length - Environment.NewLine.Length);
if (message.Length == 0)
return;
var entry = new LogEntry()
{
PartitionKey = string.Format("{0:D10}", eventCache.Timestamp >> 30),
RowKey = string.Format("{0:D19}", eventCache.Timestamp),
EventTickCount = eventCache.Timestamp,
Level = (int)eventType,
EventId = id,
Pid = eventCache.ProcessId,
Tid = eventCache.ThreadId,
Message = message
};
lock (_traceLogAccess)
_traceLog.Add(entry);
}
#endregion
#endregion
}
}

Answer 1

I looked at the source code in the blog post you referred. If you notice in the code for Details method, it is calling Trace.Flush() method which is essentially writing the trace log data collected so far in table storage. In other words, the custom trace listener is not picking up the scheduled transfer period from diagnostics.wadcfg file at all.

Answer 2

At this point, I do not think there is a solution for leveraging the scheduledTransferPeriod and custom logs. I ended up living with the immediate transfer as I wanted my own table schema. At some point I may write my own transfer interval.