Support for interruptions in bot - azure

Do you know if there is way to implement global message handlers that can support commands like stop, bye, cancel, exit Virtual assistant bot ? I am trying to implement something like this.
I have a virtual assistant built already and it has couple of Skills or Skill Bots.
When user is in the multi turn conversation with a Skill, user should be able to exit out of skill by commands like stop, bye, cancel, exit.
I found old v3 doc but nothing for v4.

Check the documentations provided here Handling User Interruption They explain how to handle user interruption for SDK v4
Find below an example of how you can configure this in the Virtual Assistant.
In your MainDialog.cs
Add the following for your OnContinueDialogAsync: Keeping in mind that you can change and edit this as you see fit just be sure to check the OnInterruptDialogAsync result (status in this example) before you continue
protected override async Task<DialogTurnResult> OnContinueDialogAsync(DialogContext innerDc, CancellationToken cancellationToken = default(CancellationToken))
{
var status = await OnInterruptDialogAsync(innerDc, cancellationToken).ConfigureAwait(false);
if (status == InterruptionAction.Resume)
{
// Resume the waiting dialog after interruption
await innerDc.RepromptDialogAsync().ConfigureAwait(false);
return EndOfTurn;
}
else if (status == InterruptionAction.Waiting)
{
// Stack is already waiting for a response, shelve inner stack
return EndOfTurn;
}
else
{
var activity = innerDc.Context.Activity;
if (activity.IsStartActivity())
{
await OnStartAsync(innerDc).ConfigureAwait(false);
}
switch (activity.Type)
{
case ActivityTypes.Message:
{
// Note: This check is a workaround for adaptive card buttons that should map to an event (i.e. startOnboarding button in intro card)
if (activity.Value != null)
{
await OnEventAsync(innerDc).ConfigureAwait(false);
}
else
{
var result = await innerDc.ContinueDialogAsync().ConfigureAwait(false);
switch (result.Status)
{
case DialogTurnStatus.Empty:
{
await RouteAsync(innerDc).ConfigureAwait(false);
break;
}
case DialogTurnStatus.Complete:
{
// End active dialog
await innerDc.EndDialogAsync().ConfigureAwait(false);
break;
}
default:
{
break;
}
}
}
// If the active dialog was ended on this turn (either on single-turn dialog, or on continueDialogAsync) run CompleteAsync method.
if (innerDc.ActiveDialog == null)
{
await CompleteAsync(innerDc).ConfigureAwait(false);
}
break;
}
case ActivityTypes.Event:
{
//do something for event activity
break;
}
case ActivityTypes.Invoke:
{
// Used by Teams for Authentication scenarios.
break;
}
default:
{
await OnSystemMessageAsync(innerDc).ConfigureAwait(false);
break;
}
}
return EndOfTurn;
}
}
And override OnInterruptDialogAsync like below example:
This example includes LUIS but you can do whatever you want in OnInterruptDialogAsync
protected override async Task<InterruptionAction> OnInterruptDialogAsync(DialogContext dc, CancellationToken cancellationToken = default(CancellationToken))
{
var result = InterruptionAction.NoAction;
if (dc.Context.Activity.Type == ActivityTypes.Message && !string.IsNullOrEmpty(dc.Context.Activity.Text))
{
// get current activity locale
var localeConfig = _services.GetCognitiveModels();
// check general luis intent
localeConfig.LuisServices.TryGetValue("General", out var luisService);
if (luisService == null)
{
throw new Exception("The specified LUIS Model could not be found in your Skill configuration.");
}
else
{
var luisResult = await luisService.RecognizeAsync<GeneralLuis>(dc.Context, cancellationToken);
var topIntent = luisResult.TopIntent();
if (topIntent.score > 0.5)
{
switch (topIntent.intent)
{
case GeneralLuis.Intent.Cancel:
{
result = await OnCancel(dc);
break;
}
case GeneralLuis.Intent.Help:
{
result = await OnHelp(dc);
break;
}
case GeneralLuis.Intent.Logout:
{
result = await OnLogout(dc);
break;
}
}
}
}
}
return result;
}

Related

Azure cognitive Service - Speech sample code failed with authentication error

The program returns: CANCELED: Reason=Error ErrorDetails=WebSocket Upgrade failed with an authentication error (401). Please check for correct subscription key (or authorization token) and region name. SessionId: cbfcdf7f26304343a08de6c398652053
I'm using my free trial subscription key and westus region. This is the sample code found here: https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/quickstarts/speech-to-text-from-microphone?tabs=unity%2Cx-android%2Clinux%2Cjava-runtime&pivots=programming-language-csharp
using UnityEngine;
using UnityEngine.UI;
using Microsoft.CognitiveServices.Speech;
#if PLATFORM_ANDROID
using UnityEngine.Android;
#endif
#if PLATFORM_IOS
using UnityEngine.iOS;
using System.Collections;
#endif
public class Helloworld : MonoBehaviour
{
// Hook up the two properties below with a Text and Button object in your UI.
public Text outputText;
public Button startRecoButton;
private object threadLocker = new object();
private bool waitingForReco;
private string message;
private bool micPermissionGranted = false;
#if PLATFORM_ANDROID || PLATFORM_IOS
// Required to manifest microphone permission, cf.
// https://docs.unity3d.com/Manual/android-manifest.html
private Microphone mic;
#endif
public async void ButtonClick()
{
// Creates an instance of a speech config with specified subscription key and service region.
// Replace with your own subscription key and service region (e.g., "westus").
var config = SpeechConfig.FromSubscription("yourSubscriptionKey", "yourRegion");
// Make sure to dispose the recognizer after use!
using (var recognizer = new SpeechRecognizer(config))
{
lock (threadLocker)
{
waitingForReco = true;
}
// Starts speech recognition, and returns after a single utterance is recognized. The end of a
// single utterance is determined by listening for silence at the end or until a maximum of 15
// seconds of audio is processed. The task returns the recognition text as result.
// Note: Since RecognizeOnceAsync() returns only a single utterance, it is suitable only for single
// shot recognition like command or query.
// For long-running multi-utterance recognition, use StartContinuousRecognitionAsync() instead.
var result = await recognizer.RecognizeOnceAsync().ConfigureAwait(false);
// Checks result.
string newMessage = string.Empty;
if (result.Reason == ResultReason.RecognizedSpeech)
{
newMessage = result.Text;
}
else if (result.Reason == ResultReason.NoMatch)
{
newMessage = "NOMATCH: Speech could not be recognized.";
}
else if (result.Reason == ResultReason.Canceled)
{
var cancellation = CancellationDetails.FromResult(result);
newMessage = $"CANCELED: Reason={cancellation.Reason} ErrorDetails={cancellation.ErrorDetails}";
}
lock (threadLocker)
{
message = newMessage;
waitingForReco = false;
}
}
}
void Start()
{
if (outputText == null)
{
UnityEngine.Debug.LogError("outputText property is null! Assign a UI Text element to it.");
}
else if (startRecoButton == null)
{
message = "startRecoButton property is null! Assign a UI Button to it.";
UnityEngine.Debug.LogError(message);
}
else
{
// Continue with normal initialization, Text and Button objects are present.
#if PLATFORM_ANDROID
// Request to use the microphone, cf.
// https://docs.unity3d.com/Manual/android-RequestingPermissions.html
message = "Waiting for mic permission";
if (!Permission.HasUserAuthorizedPermission(Permission.Microphone))
{
Permission.RequestUserPermission(Permission.Microphone);
}
#elif PLATFORM_IOS
if (!Application.HasUserAuthorization(UserAuthorization.Microphone))
{
Application.RequestUserAuthorization(UserAuthorization.Microphone);
}
#else
micPermissionGranted = true;
message = "Click button to recognize speech";
#endif
startRecoButton.onClick.AddListener(ButtonClick);
}
}
void Update()
{
#if PLATFORM_ANDROID
if (!micPermissionGranted && Permission.HasUserAuthorizedPermission(Permission.Microphone))
{
micPermissionGranted = true;
message = "Click button to recognize speech";
}
#elif PLATFORM_IOS
if (!micPermissionGranted && Application.HasUserAuthorization(UserAuthorization.Microphone))
{
micPermissionGranted = true;
message = "Click button to recognize speech";
}
#endif
lock (threadLocker)
{
if (startRecoButton != null)
{
startRecoButton.interactable = !waitingForReco && micPermissionGranted;
}
if (outputText != null)
{
outputText.text = message;
}
}
}
}
The sample code you pasted above still has the placeholder values for region and subscription key. Just double checking that you did in fact replace those strings with your own subscription key and region? If that's true, can you please turn on logging, run the code again, and then provide the log? We can help diagnose from there...
To turn on logging, see https://aka.ms/speech/logging.

Speech is not being recognized with default dictation grammar in my UWP application.

Speech is not being recognized with default dictation grammar in my UWP application. However, it is perfectly recognized when I use programmatic list constraint. Below is the speech recognition part of my code for reference. If I do not comment the 5th line, this works fine. Am I doing something wrong below:
speechRecognizer = new SpeechRecognizer();
bool PermissionGained = await CheckMicrophonePermission();
if (PermissionGained)
{
//speechRecognizer.Constraints.Add(new SpeechRecognitionListConstraint(Grammar.GrammarCommands.GrammarConstraintList));
await speechRecognizer.CompileConstraintsAsync();
//recognize speech input at any point of time
speechRecognizer.ContinuousRecognitionSession.ResultGenerated +=
async (s, e1) =>
{
if ((e1.Result != null))
{
await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
async () =>
{
await ParsespeechCommand(e1.Result);
});
speechRecognizer.ContinuousRecognitionSession.Resume();
}
};
await speechRecognizer.ContinuousRecognitionSession.StartAsync(SpeechContinuousRecognitionMode.PauseOnRecognition);
}
As discussed I made a demo and made some changes from the codes.
Here are my codes:
private async void myBtn_Click(object sender, RoutedEventArgs e)
{
//here I remove the checkPermission process
var speechRecognizer = new SpeechRecognizer();
if (true)
{
//speechRecognizer.Constraints.Add(new SpeechRecognitionListConstraint(new List<string> { "winffee", "Elvis", "noob" }));
await speechRecognizer.CompileConstraintsAsync();
//recognize speech input at any point of time
speechRecognizer.ContinuousRecognitionSession.ResultGenerated +=
async (s, e1) =>
{
if ((e1.Result != null))
{
await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
() =>//here I remove the async
{
var result = e1.Result;//here I remove the method to focus on the e1.Result.
});
speechRecognizer.ContinuousRecognitionSession.Resume();
}
};
await speechRecognizer.ContinuousRecognitionSession.StartAsync(SpeechContinuousRecognitionMode.PauseOnRecognition);
}
}
The whole function is triggered by a button click event. And I made comment on every change position(totally 3 places).

RFCommConnectionTrigger in Windows Universal Apps To detect Incoming Bluetooth Connection

I am working on a Windows Universal App. I Want to get the Data from a Bluetooth Device to the Windows Phone. I am Using the Concept of RFCommCommunicationTrigger for this Purpose.
Here's the code Snippet I am Using
var rfTrigger = new RfcommConnectionTrigger();
// Specify what the service ID is
rfTrigger.InboundConnection.LocalServiceId = RfcommServiceId.FromUuid(new Guid("<some_base_guid>"));
//Register RFComm trigger
var rfReg = RegisterTaskOnce(
"HWRFCommTrigger",
"BackgroundLibrary.RFBackgroundTask",
rfTrigger, null
);
SetCompletedOnce(rfReg, OnTaskCompleted);
Here the Function of RegisterTaskOnce
static private IBackgroundTaskRegistration RegisterTaskOnce(string taskName, string entryPoint, IBackgroundTrigger trigger, params IBackgroundCondition[] conditions)
{
// Validate
if (string.IsNullOrEmpty(taskName)) throw new ArgumentException("taskName");
if (string.IsNullOrEmpty(entryPoint)) throw new ArgumentException("entryPoint");
if (trigger == null) throw new ArgumentNullException("trigger");
// Look to see if the name is already registered
var existingReg = (from reg in BackgroundTaskRegistration.AllTasks
where reg.Value.Name == taskName
select reg.Value).FirstOrDefault();
Debug.WriteLine("Background task "+ taskName+" is already running in the Background");
// If already registered, just return the existing registration
if (existingReg != null)
{
return existingReg;
}
// Create the builder
var builder = new BackgroundTaskBuilder();
builder.TaskEntryPoint = entryPoint;
builder.Name = taskName;
builder.SetTrigger(trigger);
// Conditions?
if (conditions != null)
{
foreach (var condition in conditions)
{
builder.AddCondition(condition);
}
}
// Register
return builder.Register();
}
Here's the code for SetCompletedOnce this will add a Handler only once
static private void SetCompletedOnce(IBackgroundTaskRegistration reg, BackgroundTaskCompletedEventHandler handler)
{
// Validate
if (reg == null) throw new ArgumentNullException("reg");
if (handler == null) throw new ArgumentNullException("handler");
// Unsubscribe in case already subscribed
reg.Completed -= handler;
// Subscribe
reg.Completed += handler;
}
I have also Written the BackgroundLibrary.RFBackgroundTask.cs
public sealed class RFBackgroundTask : IBackgroundTask
{
public async void Run(IBackgroundTaskInstance taskInstance)
{
BackgroundTaskDeferral deferral = taskInstance.GetDeferral();
try
{
Debug.WriteLine(taskInstance.TriggerDetails.GetType());
taskInstance.Canceled += new BackgroundTaskCanceledEventHandler(OnCanceled);
Debug.WriteLine("RFComm Task Running");
Debug.WriteLine(taskInstance.TriggerDetails.GetType().ToString());
}
catch (System.Exception e)
{
Debug.WriteLine("RFComm Task Error: {0}", e.Message);
}
deferral.Complete();
}
}
The Run Method is Invoked Every Time The Device tries to Open the Connection.
The type of the Trigger that is obtained (the type I am debugging in the run method of the RFBackgroundTask.cs) is printed as
Windows.Devices.Bluetooth.Background.RfcommConnectionTriggerDetails
But I am Unable use that because I dont have this Class in the BackgroundLibrary project.
The Documentation says that this Provides information about the Bluetooth device that caused this trigger to fire.
It has Variables like Socket,RemoteDevice etc.
I think I am Missing something very simple
Can you please help me out .
Once your background task is launched, simply cast the TriggerDetails object to an RfcommConnectionTriggerDetails object:
public sealed class RFBackgroundTask : IBackgroundTask
{
public async void Run(IBackgroundTaskInstance taskInstance)
{
BackgroundTaskDeferral deferral = taskInstance.GetDeferral();
try
{
taskInstance.Canceled += new BackgroundTaskCanceledEventHandler(OnCanceled);
RfcommConnectionTriggerDetails details = (RfcommConnectionTriggerDetails)taskInstance.TriggerDetails;
StreamSocket = details.Socket; // Rfcomm Socket
// Access other properties...
}
catch (System.Exception e)
{
Debug.WriteLine("RFComm Task Error: {0}", e.Message);
}
deferral.Complete();
}
}

E_SCN_READINCOMPATIBLE Notification error thrown while scanning bar code on MC9090G

I'm using EMDK 2.5 (VS2008 and VC# and .NetCF3.5) Barcode2 class from the library to write a sample application to scan bar codes. I followed the samples available in EMDK namely CS_Barcode2Sample1 project.Every time I hardware trigger the scan the notification "E_SCN_READINCOMPATIBLE" is thrown and not able to retrieve the scanned data. The documentation doesn't say much about the cause of E_SCN_READINCOMPATIBLE notification and no luck from Google search. I tried several options including making use of Symbol.Barcode and the outcome is same.
I also tried EMDK 2.3 but the result is same.
I've pasted the whole code here....
public partial class Form1 : Form
{
private Barcode2 myBarcode2 = null;
public Form1()
{
InitializeComponent();
InitBarcode();
}
public bool InitBarcode()
{
// If the Barcode2 object is already initialized then fail the initialization.
if (myBarcode2 != null)
{
return false;
}
else // Else initialize the reader.
{
try
{
Symbol.Barcode2.Device[] AvailableDevices = Symbol.Barcode2.Devices.SupportedDevices;
if (AvailableDevices.Length == 0)
{
return false;
}
if (AvailableDevices.Length == 1)
{
//get the first available scanner in the list
Symbol.Barcode2.Device MyDevice = AvailableDevices[0];
// Create the reader, based on selected device.
myBarcode2 = new Barcode2(MyDevice);
// Attach a scan notification handler.
//this.myScanNotifyHandler = new Barcode2.OnScanHandler(myBarcode2_ScanNotify);
myBarcode2.OnScan += myBarcode2_ScanNotify;
// Attach a status notification handler.
//this.myStatusNotifyHandler = new Barcode2.OnStatusHandler(myBarcode2_StatusNotify);
myBarcode2.OnStatus += myBarcode2_StatusNotify;
myBarcode2.Config.TriggerMode = TRIGGERMODES.HARD;
// Submit a scan.
myBarcode2.Scan(5000);
}
}
catch (OperationFailureException ex)
{
MessageBox.Show("Exception Raised 1");
return false;
}
catch (InvalidRequestException ex)
{
MessageBox.Show("Exception Raised 2");
return false;
}
catch (InvalidIndexerException ex)
{
MessageBox.Show("Exception Raised 3");
return false;
}
}
return false;
}
private void myBarcode2_ScanNotify(ScanDataCollection scanDataCollection)
{
// Checks if the BeginInvoke method is required because the OnScan delegate is called by a different thread
if (this.InvokeRequired)
{
// Executes the OnScan delegate asynchronously on the main thread
this.BeginInvoke(new Barcode2.OnScanHandler(myBarcode2_ScanNotify), new object[] { scanDataCollection });
}
else
{
// Get ScanData
ScanData scanData = scanDataCollection.GetFirst;
int i;
switch (scanData.Result)
{
case Symbol.Barcode2.Results.SUCCESS:
String str = scanData.Text;
myBarcode2.Config.TriggerMode = TRIGGERMODES.HARD;
myBarcode2.Scan(5000);
break;
case Symbol.Barcode2.Results.E_SCN_READTIMEOUT:
break;
case Symbol.Barcode2.Results.CANCELED:
break;
case Symbol.Barcode2.Results.E_SCN_DEVICEFAILURE:
i = 93;
break;
default:
if (scanData.Result == Symbol.Barcode2.Results.E_SCN_READINCOMPATIBLE)
{
// If the failure is E_SCN_READINCOMPATIBLE, exit the application.
MessageBox.Show("Fatal Error");
this.Close();
return;
}
break;
}
}
}
private void myBarcode2_StatusNotify(StatusData statusData)
{
// Checks if the Invoke method is required because the OnStatus delegate is called by a different thread
if (this.InvokeRequired)
{
// Executes the OnStatus delegate on the main thread
this.Invoke(new Barcode2.OnStatusHandler(myBarcode2_StatusNotify), new object[] { statusData });
}
else
{
int i;
switch (statusData.State)
{
case States.IDLE:
break;
case States.READY:
break;
default:
break;
}
}
}
}
}
I've went thru this recently also, as I observed, it probably due to the scanner device is occupied by other application, where the scan request has been queued already, you can go to memory management, and kill the suspect app, and try your app again.
Refer to the Symbol FAQ

How to create and call the method(Using Thread C#) in for each loop?

I wanted to call my custom method in the Thread.
public void LoopOverAllLists(String _webAndSiteXml)
{
try
{
XmlNode _nodelist = SharePoint.ListsGetListCollection();
foreach (System.Xml.XmlNode _item in _nodelist.ChildNodes)
{
string title = _item.Attributes["Title"].Value;
//check for hidden list
if (_item.Attributes["Hidden"].Value.ToLower() == "false")
{
switch (_item.Attributes["ServerTemplate"].Value)
{
//Check whether list is document library
case SharePoint.LIST_ID_DOCUMENT_LIBRARY:
case SharePoint.LIST_ID_XML_FORMS:
case SharePoint.Publishing_ID_Pages:
{
//Get all documents info
try
{
GetAllDocumentsInfo(_item, _webAndSiteXml);
}
catch
{
}
break;
}
//Check whether list is having attachment
case SharePoint.LIST_ID_GENERIC:
case SharePoint.LIST_ID_ANNOUNCEMENTS:
case SharePoint.LIST_ID_CONTACTS:
case SharePoint.LIST_ID_TASKS:
case SharePoint.LIST_ID_EVENTS:
case SharePoint.LIST_ID_CUSTOM_GRID:
case SharePoint.LIST_ID_MEETING_SERIES:
case SharePoint.LIST_ID_MEETING_AGENDA:
case SharePoint.LIST_ID_MEETING_ATTENDEES:
case SharePoint.LIST_ID_MEETING_DECISIONS:
case SharePoint.LIST_ID_MEETING_OBJECTIVES:
case SharePoint.LIST_ID_MEETING_TTB:
case SharePoint.LIST_ID_MEETING_WS_PAGES:
case SharePoint.LIST_ID_PORTAL_SITE_LIST:
{
//Get all list items info having attachment
try
{
GetAllListItemsInfoOnlyAttachments(_item, _webAndSiteXml);
}
catch
{
}
break;
}
default:
GetAllListItemsInfoOnlyAttachments(_item, _webAndSiteXml);
break;
}
// Get All the List Forms
try
{
GetAllListForms(title, _webAndSiteXml);
}
catch
{
}
}
}
}
catch (Exception ex)
{
throw ex;
}
}
in above method three methods which is " GetAllDocumentsInfo , GetAllListItemsInfoOnlyAttachments and GetAllListForms " I wanted to call these function using thread in C#.
Thanks
Here is how I would approach the problem. Notice that I encapsulated the contents of the foreach loop into a separate method and then queued the execution of that method into the ThreadPool so that each iteration of the loop occurs in parallel. I also use a well established pattern to wait for all pending work items to complete. This code is compatible with .NET 3.5.
public void LoopOverAllLists(String _webAndSiteXml)
{
int pending = 1; // Used to track the number of pending work items.
var finished = new ManualResetEvent(false); // Used to wait for all work items to complete.
XmlNode nodes = SharePoint.ListsGetListCollection();
foreach (XmlNode item in nodes)
{
XmlNode capture = item; // This is required to capture the loop variable correctly.
Interlocked.Increment(ref pending); // There is another work item in progress.
ThreadPool.QueueUserWorkItem(
(state) =>
{
try
{
ProcessNode(capture);
}
finally
{
// Signal the event if this is the last work item to complete.
if (Interlocked.Decrement(ref pending) == 0) finished.Set();
}
}, null);
}
// Signal the event if the for loop was last work item to complete.
if (Interlocked.Decrement(ref pending) == 0) finished.Set();
// Wait for all work items to complete.
finished.WaitOne();
}
private void ProcessNode(XmlNode item)
{
// Put the contents of your loop here.
}
instead of calling
GetAllDocumentsInfo(_item, _webAndSiteXml);
use
Task.Factory.StartNew(() => GetAllDocumentsInfo(_item, _webAndSiteXml));
repeat this pattern for the other method calls as well

Resources