I try to test my action in the Google Actions Simulator. Unfortunately, the simulator does not seems to recognize the difference between the phone surface and the smart speaker surface within the simulator.
I tried to console log the screentest variable. In the logs both the phone & speaker surface show 'true', which clearly is not correct. I also checked to 'conversation' data log. Both phone & speaker output contain SCREEN_OUTPUT.
app.intent('Default Welcome Intent', (conv) => {
let screentest = conv.available.surfaces.capabilities.has('actions.capability.SCREEN_OUTPUT')
console.log(screentest)
if (screentest === true) {
conv.add('Text with screen')
} else if (screentest === false) {
conv.add('Text without screen')
} else {
conv.add('impossible')
}
})
Expected results: when using the speaker surface inside the simulator, the output of the assistant should be 'Text without Screen'.
Actual results: Both phone & speaker surface inside the simulator generate the answer: 'Text with screen'.
The issue is that you're not quite checking for surfaces correctly.
There are two sets of capabilities reported:
The capabilities available on the surface the user is currently using. If you're using the actions-on-google library, these are available using conv.surface.capabilties.has()
The capabilities available on any surface that the user has connected to their account. These are available using conv.available.surfaces.capabilities.has() if you're using the actions-on-google library.
You're currently using the second one when you should be checking the first one to see what the user is currently using.
You'll want to use the second in case there is something that you want to display to make sure they can handle it before suggesting you switch to it.
Related
I'm trying to support web-bluetooth to connect to my devices and perform a simple task (such as playing LED).
However, the device information showed on the scanning dialog when calling navigator.bluetooth.requestDevice is not so clear. It only shows the device name and a random (?) hex string.
The problem here is all my devices have the same name (AWESOME_LED), thus it's not easy for the user to select the correct LED if all scanning items show the same device name info. As far as I know, we can not custom to add more info showing on the scanning dialog.
I come up with a new solution that is changing the device name to unique for each LED with the format AWESOME_LED + [uniqueid] e.g AWESOME_LED1, AWESOME_LED2, AWESOME_LED3 so that the user can distinguish one from the others.
My question are:
Is there any alternative solution without making the device name unique?
If not, is there any problem / rejection / limitation from Apple or Google for my current app on App Store / Google Play by not using the same device name for all devices? I have been investigating it at Apple forums / Accessory Design Guidelines and looks like there are no problems, just to make sure if anyone has faced trouble from Apple / Google.
Thanks for your help.
Scanning dialog
My question are:
Is there any alternative solution without making the device name unique?
The browser prompt is not customisable yet. One solution you highlighted already is to make your LED device name unique. If you're able to control the device, why not having one AWESOME_DEVICE name and a GATT characteristic you can write to that controls individual LED colors. Maybe something like:
const device = await navigator.bluetooth.requestDevice({
filters: [{ name: "AWESOME_DEVICE" }],
});
const server = await device.gatt.connect();
const service = await server.getPrimaryService(0x1234); // Your service UUID
const characteristic = await service.getCharacteristic(0x5678); // Your characteristic UUID
// Set LED #1 to red color.
await characteristic.writeValue(
new Uint8Array(/*ledIndex=*/ 1, /*r=*/ 255, /*g=*/ 0, /*b=*/ 0)
);
If not, is there any problem / rejection / limitation from Apple or Google for my current app on App Store / Google Play by not using the
same device name for all devices? I have been investigating it at
[Apple forums][1] / [Accessory Design Guidelines][2] and looks like
there are no problems, just to make sure if anyone has faced trouble
from Apple / Google.
None that I'm aware of.
I have a hololens app I am creating that requires the best accuracy possible for hologram placement. This application will be used by numerous individuals. Whenever I try to show the application progress, I have to have the user go through the calibration process, otherwise the holograms appear to have way too much drift.
I would like to be able to call the hololens calibration process automatically when the application opens. Later, after I set up user authentication and id management, I will call the calibration process when a new user is found.
https://learn.microsoft.com/en-us/windows/mixed-reality/calibration
I have looked into the calibration (via the above documentation and elsewhere) and it seems that all it is setting is IPD. However the alternative solutions I have found that allow for dynamic ipd adjustment appear to be invalid for UWP Store apps. This makes them unusable for me.
I am looking for any help or direction, or if this is even possible. Thank you.
Yes, it is possible to to this, you need to use the LaunchUriAsync protocol to launch the following URI: ms-hololenssetup://EyeTracking
Here is an example implementation, obtained from the LaunchUri example in MRTK
public void LaunchEyeTracking()
{
#if WINDOWS_UWP
UnityEngine.WSA.Application.InvokeOnUIThread(async () =>
{
bool result = await global::Windows.System.Launcher.LaunchUriAsync(new System.Uri("ms-hololenssetup://EyeTracking"));
if (!result)
{
Debug.LogError("Launching URI failed to launch.");
}
}, false);
#else
Debug.LogError("Launching eye tracking not supported Windows UWP");
#endif
}
I am building an action using DialogFlow and Firebase cloud functions. I have a simple check to either ask a question or close the conversation depending on user's device type.
if (conv.hasScreen) {
response += `Do you want to see a picture?`;
conv.ask(response);
return;
}
else{
conv.close(response);
return;
}
I tested using Google Home mini, as expected, the conversation gracefully closed. But when I tested on a phone, the if check failed and the conversation was closed again. I was expecting the contestation to continue and assistant would ask me to show a picture but it did not happen. What am I doing wrong?
It looks like the syntax is simply conv.screen. As the property hasScreen does not exist, the conditional always returns undefined, which is a falsey value.
Take a look at the following to understand Surface Capabilities.
Are you using the following statement or not?
const hasScreen =
conv.surface.capabilities.has('actions.capability.SCREEN_OUTPUT');
I implemented chrome extension which using chrome.desktopCapture.chooseDesktopMedia to retrieve screen id.
This is my background script:
chrome.runtime.onConnect.addListener(function (port) {
port.onMessage.addListener(messageHandler);
// listen to "content-script.js"
function messageHandler(message) {
if(message == 'get-screen-id') {
chrome.desktopCapture.chooseDesktopMedia(['screen', 'window'], port.sender.tab, onUserAction);
}
}
function onUserAction(sourceId) {
//Access denied
if(!sourceId || !sourceId.length) {
return port.postMessage('permission-denie');
}
port.postMessage({
sourceId: sourceId
});
}
});
I need to get shared monitor info(resolution, landscape or portrait).
My question is: If customer using more than one monitor, how can i determine which monitor he picked?
Can i add for example "system.display" permissions to my extension and get picked monitor info from "chrome.system.display.getInfo"?
You are right. You could add system.display permission and call chrome.system.display.getDisplayLayout(callbackFuncion(DisplayLayout)) and handle the DisplayLayout.position in the callback to get the layout and the chrome.system.display.getInfo to handle the array of displayInfo in the callback. You should look for 'isPrimary' value
This is a year old question, but I came across it since I was after the same information and I finally managed to figure how you can identify which monitor the user selected for screen-sharing in Chrome.
First of all: this information will not come from the extension that you probably built for screen-sharing in Chrome, because:
The chrome.desktopCapture.chooseDesktopMedia API callback only returns a sourceId, which is a string that represents a stream id, that you can then use to call the getMediaSource API to build the media stream.
The chrome.system.display.getInfo will give you a list of the displays, yes, but from that info you can't tell which one is being shared, and there is no way to match the sourceId with any of the fields returned for each display.
So... the solution I've found comes from the MediaStream object itself. Once you have the stream, after calling getMediaSource, you need to get the video track, and in there you will find a property called "label". This label gives you an idea of which screen the user picked.
You can get the video track with something like:
const videoTrack = mediaStream.getVideoTracks()[0];
(Check the getVideoTracks API here: https://developer.mozilla.org/en-US/docs/Web/API/MediaStream/getVideoTracks).
If you print that object, you will see the "label" field. In Chrome screen 1 shows as "0:0", whereas screen 2 shows as "1:0", and I assume screen i would be "i-1:0" (I've only tested with 2 screens).
Here is a capture of that object printed in the console:
And not only works for Chrome, but for other browsers that implement it! In Firefox they show up as "Screen i":
Also, if you check Chrome chrome://webrtc-internals you'll see this is what they show in the addStream event:
And that's it! It's not ideal, since this is a label, more than a real screen identifier, but well, it's something to work with. Once you have the screen identified, in Chrome you can work with the chrome.system.display.getInfo to get information for that display.
I am running Win10 IoT on a pi 2. I need to be able to take pictures that are focused but cannot get the focus working. The application is a background app so I don't have a way of previewing the camera on a display. Is there any way of doing this? Currently I have
await _mediaCapture.StartPreviewAsync();
_mediaCapture.VideoDeviceController.FocusControl.Configure(new FocusSettings
{
Mode = FocusMode.Continuous,
WaitForFocus = true
});
await _mediaCapture.VideoDeviceController.FocusControl.FocusAsync();
await _mediaCapture.CapturePhotoToStreamAsync(ImageEncodingProperties.CreateJpeg(), stream);
await _mediaCapture.StopPreviewAsync();
but I am getting the error
WinRT information: Preview sink not set
when I try to focus. All of the examples I've seen online show that the preview is output to a control and I assume it wires a sink up automagically. Is there a way to do this manually through code? Possibly without the preview?
I wonder if the code may work even without FocusControl.
I propose you follow Customer Media Sink implementation example and use of StartPreviewToCustomSinkIdAsync method described at http://www.codeproject.com/Tips/772038/Custom-Media-Sink-for-Use-with-Media-Foundation-To
I didn't find a way to do this. I ended up converting the background app to a UI app with a Page containing a CaptureElement control in order to preview and focus.
Instead of adding a UI, just create a CaptureElement and set the source to the _mediaCapture before calling await _mediaCapture.StartPreviewAsync();
Something like:
_captureElement = new CaptureElement { Stretch = Stretch.Uniform };
_mediaCapture = new MediaCapture();
await _mediaCapture.InitializeAsync(...);
_captureElement.Source = _mediaCapture;
await _mediaCapture.StartPreviewAsync();