We are using an EvenProcessor which are using the iterface IEventProcessor.
When I connect to our Azure Eventhub I always get "all" data, but I only want "live" data. Was reading about setting the Offset, using a DateTime but I dont know where. Have been trying around without and success. Is it possible? Attached some of the code below:
Greetings
public class EventProcessor : IEventProcessor
{
IDictionary<string, int> map;
PartitionContext partitionContext;
Stopwatch checkpointStopWatch;
public EventProcessor()
{
this.map = new Dictionary<string, int>();
}
public Task OpenAsync(PartitionContext context)
{
context.Lease.Offset = DateTime.Now.ToString(); // not working - still gives all data
this.partitionContext = context;
this.checkpointStopWatch = new Stopwatch();
this.checkpointStopWatch.Start();
return Task.FromResult<object>(null);
}
Related
I would like to see if/how it would be possible to plug into the deserialization process for a parameter that's decorated with the ServiceBusTrigger?
Say I have a function that looks like:
public static void HandleMessage([ServiceBusTrigger("myqueue")] MyCustomType myCustomType) { }
How would I go about taking over the deserialization? I know that there is a notion of an IArgumentBindingProvider and IArgumentBinding but it does not look like ServiceBusTrigger supports these concepts.
I know I can use GetBody<Stream>() and deserialize that way but I'd like to know if I can plug into the ServiceBusTrigger's pipeline. By the looks at the SDK, the ServiceBusTrigger has a hard coded list of IQueueArgumentBindingProviders and so I can't add my own.
If you have a look at the Azure WebJobs SDK Extensions, there is an overview on how to create your own bindings :
Binding Extensions Overview
Otherwise the ServiceBusConfiguration exposes a MessagingProvider property that allows you to intercept the ServiceBusTrigger pipeline:
private static void Main()
{
var sbConfig = new ServiceBusConfiguration()
{
MessagingProvider = // you implemetation of the MessagingProvider class goes here !!!
};
var config = new JobHostConfiguration();
config.UseServiceBus(sbConfig);
new JobHost(config).RunAndBlock();
}
Here is a simple skeleton of a MessagingProvider implementation:
public sealed class MyMessagingProvider : MessagingProvider
{
private readonly ServiceBusConfiguration _config;
public MyMessagingProvider(ServiceBusConfiguration config)
: base(config)
{
_config = config;
}
public override MessageProcessor CreateMessageProcessor(string entityPath)
{
return new MyMessageProcessor(_config.MessageOptions);
}
private class MyMessageProcessor : MessageProcessor
{
public MyMessageProcessor(OnMessageOptions messageOptions)
: base(messageOptions)
{
}
public override Task<bool> BeginProcessingMessageAsync(BrokeredMessage message, CancellationToken cancellationToken)
{
// Intercept the message before the execution of the triggerred function
return base.BeginProcessingMessageAsync(message, cancellationToken);
}
public override Task CompleteProcessingMessageAsync(BrokeredMessage message, FunctionResult result, CancellationToken cancellationToken)
{
// Intercept the message after the execution of the triggerred function and before being completed
return base.CompleteProcessingMessageAsync(message, result, cancellationToken);
}
}
}
So you're main function now looks like that:
private static void Main()
{
var sbConfig = new ServiceBusConfiguration();
sbConfig.MessagingProvider = new MyMessagingProvider(sbConfig);
var config = new JobHostConfiguration();
config.UseServiceBus(sbConfig);
new JobHost(config).RunAndBlock();
}
in previous si versions (si 2.11 to be specific and spring 3.1.1) getStandardRequestHeaderNames could be overrided to include Additional Application specific objects in the si message header. Our application relied on this ability (may be wrongfully so) to override this method and supply a custom POJO to be carried downstream consisting of many splitters, aggregators etc. The app used an ws inbound gateway and used the header-mapper attribute to specify the custom soap header mapper.
Any clues on the reasoning behind why getStandardRequestHeaderNames cannot be overriden?
Need some advise on how I can migrate this to the current spring release.
The requirement is to extract elements from soapHeader and map them to an SI message headers as an POJO and send it down stream.
All help appreciated.
Code Snippet: Works with older versions of spring
<int-ws:inbound-gateway id="webservice-inbound-gateway"
request-channel="input-request-channel"
reply-channel="output-response-channel"
header-mapper="CustomSoapHeaderMapper"
marshaller="marshaller"
unmarshaller="marshaller" />
#Component("CustomSoapHeaderMapper")
public class CustomSoapHeaderMapper extends DefaultSoapHeaderMapper {
private static final Logger logger = Logger.getLogger("CustomSoapHeaderMapper");
public static final String HEADER_SEARCH_METADATA = SearchMetadata.HEADER_ATTRIBUTE_NAME;
public static final String HEADER_SERVICE_AUDIT = "XXXXXXXX";
// Use simulation if security token is set to this value
public static final String SECURITY_TOKEN_SIMULATION = "XXXX";
private static final List<String> CUSTOM_HEADER_NAMES = new ArrayList<String>();
static {
CUSTOM_HEADER_NAMES.add(WebServiceHeaders.SOAP_ACTION);
CUSTOM_HEADER_NAMES.add(HEADER_SEARCH_METADATA);
}
private int version =SearchMetadata.VERSION_CURRENT;
public void setVersion(int version) {
this.version = version;
}
#Override
protected List<String> getStandardRequestHeaderNames() {
return CUSTOM_HEADER_NAMES;
}
#Override
protected Map<String, Object> extractUserDefinedHeaders(SoapMessage source) {
// logger.log(Level.INFO,"extractUserDefinedHeaders");
// call base class to extract header
Map<String, Object> map = super.extractUserDefinedHeaders(source);
Document doc = source.getDocument();
SearchMetadata searchMetadata = new SearchMetadata();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
try {
source.writeTo(baos);
baos.flush();
searchMetadata.setRequestXML(baos.toString());
baos.close();
} catch (IOException e1) {
}
//logger.log(Level.WARNING, "Incoming Message " + baos.toString());
SOAPMessage soapMessage = ((SaajSoapMessage) source).getSaajMessage();
// generate TransactionID with UUID value
String transactionID = UUID.randomUUID().toString();
// logger.log(Level.WARNING, "TransactionID=" + transactionID);
Date now = new Date();
searchMetadata.setTransactionID(transactionID);
searchMetadata.setRequestType(SearchMetadata.REQUEST_TYPE_SYNCHRONOUS);
searchMetadata.setRequestTime(now);// initialize the request time
searchMetadata.setReceivedTime(now);// mark time system receives request
searchMetadata.setVersion(version);
Map<String, Object> finalHeaders = new HashMap<String, Object>();
finalHeaders.put(HEADER_SEARCH_METADATA, searchMetadata);
if (!CollectionUtils.isEmpty(map)) {
// copy from other map
finalHeaders.putAll(map);
// check if ServiceAudit is available
SoapHeaderElement serviceAuditElement = null;
for (String key : map.keySet()) {
// logger.log(Level.WARNING, "SoapHeader.{0}", key);
if (StringUtils.contains(key, HEADER_SERVICE_AUDIT)) {
serviceAuditElement = (SoapHeaderElement) map.get(key);
break;
}
}
}
return finalHeaders;
}
// GK Key Thing here for performance improvement is avoiding marshalling
public gov.dhs.ice.ess.schema.ServiceAudit ExtractAuditHeader(Document doc) {
....
}
return serviceAudit;
}
}
Share, please, some code how would you like to see that.
Maybe you can just implement your own SoapHeaderMapper and inject it into WS Inbound Gateway?
You can still reuse your logic and copy/paste the standard behavior from the DefaultSoapHeaderMapper.
UPDATE
The test-case to demonstrate how to add user-defined header manually:
#Test
public void testCustomSoapHeaderMapper() {
DefaultSoapHeaderMapper mapper = new DefaultSoapHeaderMapper() {
#Override
protected Map<String, Object> extractUserDefinedHeaders(SoapMessage source) {
Map<String, Object> headers = super.extractUserDefinedHeaders(source);
headers.put("foo", "bar");
return headers;
}
};
mapper.setRequestHeaderNames("*");
SoapMessage soapMessage = mock(SoapMessage.class);
Map<String, Object> headers = mapper.toHeadersFromRequest(soapMessage);
assertTrue(headers.containsKey("foo"));
assertEquals("bar", headers.get("foo"));
}
I was hoping for some guidance on how to use the EventProcessorHost with a worker role. Basically I am hoping to have the EventProcessorHost process the partitions in parallel and I'm wondering where I should go about placing this type of code within the worker role and if I'm missing anything key.
var manager = NamespaceManager.CreateFromConnectionString(connectionString);
var desc = manager.CreateEventHubIfNotExistsAsync(path).Result;
var client = Microsoft.ServiceBus.Messaging.EventHubClient.CreateFromConnectionString(connectionString, path);
var host = new EventProcessorHost(hostname, path, consumerGroup, connectionString, blobStorageConnectionString);
EventHubProcessorFactory<EventData> factory = new EventHubProcessorFactory<EventData>();
host.RegisterEventProcessorFactoryAsync(factory);
Everything I've read says the EventProcessorHost will divide up the partitions on its own, but is the above code sufficient to process all the partitions asynchronously?
Here's a simplified version of how we process our event hub from an Worker Role. We keep the instance in the mainWorker role and call the IEventProcessor to start processing it.
This way we can call it and close it down when the Worker Responds to shutdown events etc.
EDIT:
As for the processing it in parallel, the IEventProcessor class will just grab 10 more events from the event hub when it's finished processing the current one. Handling all the fancy partition leasing for you.
It's a synchronous workflow, When I scale to multiple worker roles I start to see the partitions get split between instances and it gets faster etc. You'd have to roll your own solution if you wanted it to process the event hub in a different way.
public class WorkerRole : RoleEntryPoint
{
private readonly CancellationTokenSource _cancellationTokenSource = new CancellationTokenSource();
private readonly ManualResetEvent _runCompleteEvent = new ManualResetEvent(false);
private EventProcessorHost _eventProcessorHost;
public override bool OnStart()
{
ThreadPool.SetMaxThreads(4096, 2048);
ServicePointManager.DefaultConnectionLimit = 500;
ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;
var eventClient = EventHubClient.CreateFromConnectionString("consumersConnectionString",
"eventHubName");
_eventProcessorHost = new EventProcessorHost(Dns.GetHostName(), eventClient.Path,
eventClient.GetDefaultConsumerGroup().GroupName,
"consumersConnectionString", "blobLeaseConnectionString");
return base.OnStart();
}
public override void Run()
{
try
{
RunAsync(this._cancellationTokenSource.Token).Wait();
}
finally
{
_runCompleteEvent.Set();
}
}
private async Task RunAsync(CancellationToken cancellationToken)
{
// starts processing here
await _eventProcessorHost.RegisterEventProcessorAsync<EventProcessor>();
while (!cancellationToken.IsCancellationRequested)
{
await Task.Delay(TimeSpan.FromMinutes(1));
}
}
public override void OnStop()
{
_eventProcessorHost.UnregisterEventProcessorAsync().Wait();
_cancellationTokenSource.Cancel();
_runCompleteEvent.WaitOne();
base.OnStop();
}
}
I have multiple processors for the specific partitions (you can guarantee FIFO this way), but you can implement you're own logic easily i.e. skip the use of a EventDataProcessor class and Dictionary lookup in my example and just implement some logic within the ProcessEventsAsync method.
public class EventProcessor : IEventProcessor
{
private readonly Dictionary<string, IEventDataProcessor> _eventDataProcessors;
public EventProcessor()
{
_eventDataProcessors = new Dictionary<string, IEventDataProcessor>
{
{"A", new EventDataProcessorA()},
{"B", new EventDataProcessorB()},
{"C", new EventDataProcessorC()}
}
}
public Task OpenAsync(PartitionContext context)
{
return Task.FromResult<object>(null);
}
public async Task ProcessEventsAsync(PartitionContext context, IEnumerable<EventData> messages)
{
foreach(EventData eventData in messages)
{
// implement your own logic here, you could just process the data here, just remember that they will all be from the same partition in this block
try
{
IEventDataProcessor eventDataProcessor;
if(_eventDataProcessors.TryGetValue(eventData.PartitionKey, out eventDataProcessor))
{
await eventDataProcessor.ProcessMessage(eventData);
}
}
catch (Exception ex)
{
_//log exception
}
}
await context.CheckpointAsync();
}
public async Task CloseAsync(PartitionContext context, CloseReason reason)
{
if (reason == CloseReason.Shutdown)
await context.CheckpointAsync();
}
}
Example of one of our EventDataProcessors
public interface IEventDataProcessor
{
Task ProcessMessage(EventData eventData);
}
public class EventDataProcessorA : IEventDataProcessor
{
public async Task ProcessMessage(EventData eventData)
{
// Do Something specific with data from Partition "A"
}
}
public class EventDataProcessorB : IEventDataProcessor
{
public async Task ProcessMessage(EventData eventData)
{
// Do Something specific with data from Partition "B"
}
}
Hope this helps, it's been rock solid for us so far and scales easily to multiple instances
I'm trying to write a target for NLog to send messages out to connected clients using SignalR.
Here's what I have now. What I'm wondering is should I be using resolving the ConnectionManager like this -or- somehow obtain a reference to the hub (SignalrTargetHub) and call a SendMessage method on it?
Are there performance ramifications for either?
[Target("Signalr")]
public class SignalrTarget:TargetWithLayout
{
public SignalR.IConnectionManager ConnectionManager { get; set; }
public SignalrTarget()
{
ConnectionManager = AspNetHost.DependencyResolver.Resolve<IConnectionManager>();
}
protected override void Write(NLog.LogEventInfo logEvent)
{
dynamic clients = GetClients();
var logEventObject = new
{
Message = this.Layout.Render(logEvent),
Level = logEvent.Level.Name,
TimeStamp = logEvent.TimeStamp.ToString("yyyy-MM-dd HH:mm:ss.fff")
};
clients.onLoggedEvent(logEventObject);
}
private dynamic GetClients()
{
return ConnectionManager.GetClients<SignalrTargetHub>();
}
}
I ended up with the basic the same basic structure that I started with. Just a few tweaks to get the information I needed.
Added exception details.
Html encoded the final message.
[Target("Signalr")]
public class SignalrTarget:TargetWithLayout
{
protected override void Write(NLog.LogEventInfo logEvent)
{
var sb = new System.Text.StringBuilder();
sb.Append(this.Layout.Render(logEvent));
if (logEvent.Exception != null)
sb.AppendLine().Append(logEvent.Exception.ToString());
var message = HttpUtility.HtmlEncode(sb.ToString());
var logEventObject = new
{
Message = message,
Logger = logEvent.LoggerName,
Level = logEvent.Level.Name,
TimeStamp = logEvent.TimeStamp.ToString("HH:mm:ss.fff")
};
GetClients().onLoggedEvent(logEventObject);
}
private dynamic GetClients()
{
return AspNetHost.DependencyResolver.Resolve<IConnectionManager>().GetClients<SignalrTargetHub>();
}
}
In my simple testing it's working well. Still remains to be seen if this adds any significant load when under stress.
I'm trying to get windows workflows working, and I've become a little stumped.
I've gotten a single workflow working, but now I am trying to do something a little more complex: start a workflow, where each activity itself contains a workflow. (Picture something like the main program starts the activities "Input, logic, and output", and then each of those have additional activities like "prompt user, get input, etc.")
I've had it working fine, with the example from here (http://msdn.microsoft.com/en-us/magazine/gg535667.aspx), when I am not passing any parameters from the main program to the activites. My question is, how exactly does the 'Variables' and 'metadata.SetVariablesCollection' work in the NativeActivity, and how to I get the parameters to the low level activities?
This is what I am currently trying:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Activities;
using System.Collections.ObjectModel;
using System.Activities.Statements;
namespace Project1
{
internal class MainProgram
{
internal static void Main(string[] args)
{
try
{
var act = new SimpleSequence();
act.Activities.Add((Activity)(new WriteSomeText()));
act.Activities.Add((Activity)(new WriteSomeText()));
act.Activities.Add((Activity)(new WriteSomeText()));
act.Variables.Add(new Variable<string> ("stringArg", "TEXT"));
WorkflowInvoker.Invoke(act);
}
catch (Exception ex)
{
System.Console.WriteLine("EXCEPTION: {0}", ex);
}
}
public class WriteSomeText : CodeActivity
{
[RequiredArgument]
public InArgument<string> stringArg { get; set; }
protected override void Execute(CodeActivityContext context)
{
string output = context.GetValue(stringArg);
System.Console.WriteLine(output);
}
}
public class SimpleSequence : NativeActivity
{
Collection<Activity> activities;
Collection<Variable> variables;
Variable<int> current = new Variable<int> { Default = 0 };
public Collection<Activity> Activities
{
get
{
if (this.activities == null)
this.activities = new Collection<Activity>();
return this.activities;
}
set
{
this.activities = value;
}
}
public Collection<Variable> Variables
{
get
{
if (this.variables == null)
this.variables = new Collection<Variable>();
return this.variables;
}
set
{
this.variables = value;
}
}
protected override void CacheMetadata(NativeActivityMetadata metadata)
{
metadata.SetChildrenCollection(this.activities);
metadata.SetVariablesCollection(this.variables);
metadata.AddImplementationVariable(this.current);
}
protected override void Execute(NativeActivityContext context)
{
if (this.Activities.Count > 0)
context.ScheduleActivity(this.Activities[0], onChildComplete);
}
void onChildComplete(NativeActivityContext context, ActivityInstance completed)
{
int currentExecutingActivity = this.current.Get(context);
int next = currentExecutingActivity + 1;
if (next < this.Activities.Count)
{
context.ScheduleActivity(this.Activities[next], this.onChildComplete);
this.current.Set(context, next);
}
}
}
}
}
This ends up throwing the following exception:
EXCEPTION: System.Activities.InvalidWorkflowException: The following errors were encountered while processing the workflow tree:
'WriteSomeText': Value for a required activity argument 'stringArg' was not supplied.
'WriteSomeText': Value for a required activity argument 'stringArg' was not supplied.
'WriteSomeText': Value for a required activity argument 'stringArg' was not supplied.
at System.Activities.Validation.ActivityValidationServices.ThrowIfViolationsExist(IList`1 validationErrors)
at System.Activities.Hosting.WorkflowInstance.ValidateWorkflow(WorkflowInstanceExtensionManager extensionManager)
at System.Activities.Hosting.WorkflowInstance.RegisterExtensionManager(WorkflowInstanceExtensionManager extensionManager)
at System.Activities.WorkflowApplication.EnsureInitialized()
at System.Activities.WorkflowApplication.RunInstance(WorkflowApplication instance)
at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout)
at System.Activities.WorkflowInvoker.Invoke(Activity workflow, TimeSpan timeout, WorkflowInstanceExtensionManager extensions)
at System.Activities.WorkflowInvoker.Invoke(Activity workflow)
at Project1.MainProgram.Main(String[] args) in c:\users\user\documents\visual studio 2010\Projects\ModelingProject1\Project1\MainProgram.cs:line 25
I know, I only pass 1 parameter, but the exception still says that I am missing 3 parameters. I am missing something as to how to do this properly.
You're correctly declaring stringArg as an InArgument but you're not passing any value to it when calling it inside SimpleSequence.
You can pass something using the constructor, while constructing the all activity itself, like this:
public class WriteSomeText : CodeActivity
{
[RequiredArgument]
public InArgument<string> stringArg { get; set; }
public WriteSomeText(string stringArg)
{
this.stringArg = stringArg;
}
protected override void Execute(CodeActivityContext context
{
string output = context.GetValue(stringArg);
System.Console.WriteLine(output);
}
}
// Calling the activity like this:
internal static void Main(string[] args)
{
var act = new SimpleSequence()
{
Activities =
{
new WriteSomeText("hello"),
new WriteSomeText("world"),
new WriteSomeText("!")
}
};
WorkflowInvoker.Invoke(act);
Console.WriteLine("Press any key to exit");
Console.ReadKey();
}
Also notice that is a best practice to use the constructor to initialize collections:
public SimpleSequence()
{
activities = new Collection<Activity>();
variables = new Collection<Variable>();
}
This way is even more intuitive to initialize the activity:
var act = new SimpleSequence()
{
Activities =
{
new WriteSomeText("hello"),
new WriteSomeText("world"),
new WriteSomeText("!")
},
Variables =
{
new Variable<int>("myNewIntVar", 10),
// ....
}
};
EDIT:
There are a couple of other ways to approach the problem. This is your best friend while starting in the WF4 world.
Check WF\Basic\CustomActivities\Code-Bodied for a little push with this particular case.