This is my first post on StackOverflow (long-time lurker, first-time poster), so go easy on me. ^__^;
For those having trouble in implementing play/pause/resume functionality with a STATIC mp3 I’m assuming the process is the same, so hopefully, this post will help you guys as well.
I’m working on building a live mp3 streaming Google Action, and I seem to be having issues with implementing it in the new Actions Console https://console.actions.google.com/
According to the Google Actions documentation found here:
https://developers.google.com/assistant/conversational/prompts-media - Last updated 2021-03-10 UTC.
I should be able to invoke a Media Response to play an mp3 back to the user using the YAML / JSON example provided in the above link, however, it seems that playing, pausing, and resuming doesn’t work correctly with a streaming mp3 URL.
TLDR; Here's a shorter version of the write up:
https://imgur.com/a/FIgOsl8
For a more detailed analysis see below:
STEPS TO REPRODUCE
Starting with the example provided in the documentation and popping the JSON version sample code (posted here for convenience) in the On Enter section of the scene; I was able to play the media fine.
{
"candidates": [
{
"first_simple": {
"variants": [
{
"speech": "This is a media response."
}
]
},
"content": {
"media": {
"optional_media_controls": [
"PAUSED",
"STOPPED"
],
"media_objects": [
{
"name": "Media name",
"description": "Media description",
"url": "https://storage.googleapis.com/automotive-media/Jazz_In_Paris.mp3",
"image": {
"large": {
"url": "https://storage.googleapis.com/automotive-media/album_art.jpg",
"alt": "Jazz in Paris album art"
}
}
}
],
"media_type": "AUDIO"
}
}
}
]
}
Note: In the above JSON I removed the start_offset node because it’s currently not supported by iOS and is probably put in there as an example for testing purposes.
Here’s an example of the static mp3 media response playing for reference:
https://downloaddave.com/reviews/clients/momentum-br/ga-sr/Screenshot_streaming_playing_no_error_with_test_mp3.png
I noticed that pausing and resuming the static mp3 does not work unless you enabled the following System Intents:
MEDIA_STATUS_PAUSED
MEDIA_STATUS_STOPPED
MEDIA_STATUS_FAILED
MEDIA_STATUS_FINISHED
Otherwise, if you click on the “pause” icon on the Media Response Player or invoke the pause earcon (earcon = ear + icon) you will encounter the following errors:
Sorry, [Your Action’s Display Name] isn't responding right now. Please try again soon.
Did not find any handling for intent event 'actions.intent.MEDIA_STATUS_PAUSED' on scene 'playStreamingAudio'
{
"endConversation": {}
}
Under the Error and status handling section of the scene I added the system intents as seen in the following screenshot.
https://downloaddave.com/reviews/clients/momentum-br/ga-sr/playStreamingAudio_Scene_Configuration_000.png
Note that if I just transition the MEDIA_STATUS_PAUSED to “No Transition” it gives me an error message, Event handler for ‘playStreamingAudio’ has an empty function call and/or empty transition.
If it goes to “End Conversation” it ends the test and exits out of the Media Response Card rather than giving me the option to resume (which seems like a bad user/conversational flow and probably won't pass review).
Tapping the “pause” icon, typing, or saying “pause” doesn’t work unless the MEDIA_STATUS_PAUSED transitions to another Scene which I’ve called pauseStreamingAudio.
In the pauseStreamingAudio scene, I added a prompt letting the user know they can say “play” or “cancel” along with suggestions indicating the same.
{
"candidates": [
{
"first_simple": {
"variants": [
{
"speech": "You can say play to resume audio or cancel to quit."
}
]
},
"suggestions": [{
"title": "Play"
}, {
"title": "Cancel"
}]
}
]
}
From the pauseStreamingAudio Scene, I added a custom intent “play” to go back to the previous Scene I’ve called playSreamingAudio.
I’m not sure if I’m doing this right BUT IT WORKS!
Streaming mp3
Now that I got the foundation working I swapped out the static mp3 to the streaming audio. Here is the Sample Code JSON Builder with streaming mp3 link & “start_offset” removed and the streaming mp3 link.
{
"candidates": [
{
"first_simple": {
"variants": [
{
"speech": "This is a media response."
}
]
},
"content": {
"media": {
"optional_media_controls": [
"PAUSED",
"STOPPED"
],
"media_objects": [
{
"name": "Media name",
"description": "Media description",
"url": "https://prod-35-230-37-193.wostreaming.net/momentum-kvmifmaac-ibc2",
"image": {
"large": {
"url": "https://storage.googleapis.com/automotive-media/album_art.jpg",
"alt": "Jazz in Paris album art"
}
}
}
],
"media_type": "AUDIO"
}
}
}
]
}
The Content-Type of the streaming file that I’m testing with doesn't specifically end in a *.mp3 and when I check the content type is reads as audio/aacp.
Codec: ADTS
Type: Audio
Channels: Stereo
Sample Rate: 44100 Hz
Bits per Sample: 32
AAC Extension: SBR+PS
This works as and I'm able to stream audio form the source file. See screenshot below.
https://downloaddave.com/reviews/clients/momentum-br/ga-sr/Smart_Display_Time_Error_Infinit_NaN_Nan_redlined.png
However, there is a display error on the Media Response Player at the time index by the bottom right Infinity:NaN:NaN (highlighted in the red box).
Likely related, I can no longer trigger the Pause System Intent anymore. Instead, I get the following error:
https://downloaddave.com/reviews/clients/momentum-br/ga-sr/Screenshot_streaming_pause_error.png
Notice that the drop-down is open and there is no response for me to use and troubleshoot.
I also tried looking through the Actions on Google documentation to see if there could be something wrong with the audio stream I was providing, the best thing I could find was,
“Audio for playback must be in a correctly formatted MP3 file. MP3 files must be hosted on a web server and be publicly available through an HTTPS URL. Live streaming is only supported for the MP3 format.”
I found some info on mp3 specs on the SSML page here, but I’m not sure if this applies to the Media Response https://developers.google.com/assistant/conversational/ssml#audio - Last updated 2021-05-25 UTC.
Does anyone have any ideas on how I can get this working or even troubleshoot this?
Could some of these circumstances be an issue with the Media Player itself? How would one go about fixing this?
Anyway, I hope this helps somebody out there & thanks very much in advance. Any help is most appreciated.
Related
I have setup my Dialogflow CX and Messenger on my web site and I want to execute commands with Google Tag manager.
Basically what I want to do is that if a user scrolls more than 75% of a page, vertically, GTM should trigger this example ( taken from https://cloud.google.com/dialogflow/cx/docs/concept/integration/dialogflow-messenger#rendercustomcard )
const dfMessenger = document.querySelector('df-messenger');
const payload = [
{
"type": "info",
"title": "Info item title",
"subtitle": "Info item subtitle",
"image": {
"src": {
"rawUrl": "https://example.com/images/logo.png"
}
},
"actionLink": "https://example.com"
}];
dfMessenger.renderCustomCard(payload);
This code snippet works fine if I embed it in my web page, and also in when GTM triggers and embeds the tag after a scroll the snippet. But when I try the other types of cards, List type is what I would like to use in my case, I the following in my browsers console "DfMessenger: Could not render undefined".
Any clue if this is due to me triggering things from GTM or any ideas what I could test?
Posting this answer from #Per Olsson as a wikianswer:
I figured out what was wrong with const dfMessenger = document.querySelector('df-messenger') dfMessenger.addEventListener('df-request-sent', function (event) { console.log(event) and compared with the objects and found a misspelled wording. Everything works but you have to be really careful with spelling. I still think the documentation is a bit poor though, but that is not for this forum.
I am hoping to get some advice with regards to calculating core web vitals without interacting with PerformanceObserver api but instead to use Chrome trace events.
Since the puppeteer operation is done at scale I prefer to not to interact with the page using page.evaluate but instead calculate the metrics if possible from the data I get using:
await page.tracing.start();
await page.goto(url, { waitUntil: "networkidle0" });
writeFile("dump.json", (await page.tracing.stop()).toString()); -> Can I get web vitals/perf metrics here?
I can see in the output of the trace, events like this:
{
"args": {
"data": {
"candidateIndex": 1,
"isMainFrame": true,
"navigationId": "927C29B0D3BD3783D85D54D161903DEF",
"nodeId": 92,
"size": 145574,
"type": "image"
},
"frame": "8DCDB2C2311B3C524C094C8C4555E0FB"
},
"cat": "loading,rail,devtools.timeline",
"name": "largestContentfulPaint::Candidate",
"ph": "I",
"pid": 94113,
"s": "t",
"tid": 775,
"ts": 437767356341
}
Would be good to know if I am barking up the wrong tree here or I can extract/calculate perf metrics using only the traceEvents data. I am assuming this should be possible since the dump has all the events that took place during page load and the devtools seems to place them metrics on the timeline too.
Many Thanks.
The PerformanceTimeline domain used by the Chrome DevTools protocol may contain the kind of information you're looking for, similar to your screenshot.
The FCP, LCP, and CLS vitals are also recorded in the trace data and accessible via Puppeteer, but there are some caveats to consider:
The correct trace categories should be recorded. Refer to the categories used by DevTools.
The render and frame IDs should be used to disambiguate records between the top-level frame and any iframes. You can get these IDs from the TracingStartedInBrowser event.
I have developed interactive canvas application and it was working fine with devices having display.
Suddenly when I checked it today it says "Application is not responding, try again later.". When I checked in test simulator and gone through debug I have received following error printed in debug.
"sharedDebugInfoList": [
{
"name": "ResponseValidation",
"debugInfo": "",
"subDebugEntryList": [
{
"name": "MalformedResponse",
"debugInfo": "expected_inputs[0].input_prompt.rich_initial_prompt.items[1].html_response: HtmlResponse is not supported on this device..",
"subDebugEntryList": []
}
]
}
],
It was working and users were using it in their mobile device, but this sudden error made me blind to understand it. I have not made even 1 line of change in my code. I have even checked cloud logs and there is nothing. Here is what I am doing when user enters in my action.
app.intent('welcome', (conv) => {
console.log('capabilities = ',conv.surface.capabilities)
if (!conv.surface.capabilities.has('actions.capability.SCREEN_OUTPUT')) {
conv.close('Sorry, this device does not support Interactive Canvas!');
return;
}
conv.ask(`Welcome! Let's start the game`);
conv.ask(new HtmlResponse({
url: '<url of my website having interactive canvas support>',
}));
});
Here is the action that I am facing error.
The action seems to be working ok for me.
The most common reason I've seen for it not working is that the category wasn't set to "Games & Fun" (or was cleared for some reason) or the Interactive Canvas checkbox wasn't set.
To make sure you are still in the "Games & Fun" category, go to the Actions Console, the "Deploy" tab, and the "Directory" information.
Then, towards the bottom of that same page, make sure you have the Interactive Canvas checkbox set.
I'm stuck in trying to figure this out and I hope someone out there can help me out. I am using the Dialogflow console to create a bot that requests a user to report "something" by providing his/her location and describing the incident. The bot is integrated with Facebook Messenger. One of my intents has a follow up intent which also has a follow up intent like:
intent 1
|
intent 2
| intent 3
Intent 1 requests for the user's location, intent 2 retrieves the user's location and asks the user to describe the location. Intent 3 SHOULD have all the data in context as it's fulfilled by the a webhook. All the data SHOULD be posted to my server. The problem is that I have failed to get the location data (maybe lat and long) I notice that the data comes back in the following format after the fired event FACEBOOK_LOCATION:
{
"originalDetectIntentRequest": {
"source": "facebook",
"payload": {
"postback": {
"data": {
"lat": 14.556761479425,
"long": 121.05444780425
},
"payload": "FACEBOOK_LOCATION"
},
"sender": {
"id": "1588949991188331"
}
}
}
My question is how to I carry that payload data into my Dialogflow Intent Parameters so that they are carried in context until my webhook is fired? I hope i've explained it well. Thanks for the help guys.
You can use the output contexts to save the parameters.
{
"fulfillmentText":"This is a text response",
"fulfillmentMessages":[ ],
"source":"example.com",
"payload":{
"google":{ },
"facebook":{ },
"slack":{ }
},
"outputContexts":[
{
"name":"context name",
"lifespanCount":5,
"parameters":{
"param":"param value"
}
}
],
"followupEventInput":{ }
}
Once you save the parameters, in the subsequent requests, you can access the parameters by accessing saved context. The lifespanCount will decide how many subsequent calls this context is valid. So in the above, eg. parameters saved in intent 1 will be available till intent 5 (if you have 2 more follow up intents)
You can follow more details here.
I personally like to use the client library to develop webhooks as they are easy to use, featureful and reduces JSON manipulation errors. If you like to use NodeJs based client, you can follow this link.
To expand on Abhinav's answer (and point out what caught me up on this issue). You need to make sure that the entities you extracted have the lifespan to make it to your webhook fulfillment call.
You can adjust the count by editing the number and saving.
The lifespanCount will decide how many subsequent calls this context is valid. - Abhinav
If your parameters are not showing up in your output context they probably don't have the appropriate lifespan.
I'm writing a Chrome Extension in Javascript and I want to get the current time for the playing video on youtube.com. I tried using the answer from question Getting Current YouTube Video Time , e.g.:
ytplayer = document.getElementById("movie_player");
ytplayer.getCurrentTime();
However I do get following error: "Uncaught TypeError: Cannot read property 'getCurrentTime' of null";
What I am doing wrong? I tried different values for the ElementId - movie_player, player....
Any help is appreciated. Thanks in advance.
Cheers.
edit:
Here is my manifest:
{
"manifest_version": 2,
"name": "",
"description": "",
"version": "1.0",
"browser_action": {
"default_icon": "icon.png",
"default_popup": "basic.html"
},
"permissions": [
"tabs",
"activeTab", "http://*/*", "https://*/*"
]
}
Another thing is:
If I execute this code:
ytplayer = document.getElementById("movie_player");
ytplayer.getCurrentTime();
In the Javascript console on a Youtube Page it works fine and returns the current time.
If, however I execute this value from the extension or the console of the extension, the first line return value null.
So, as Ben assumed below, the issue seems to be that my extension doesn't even access the Youtube page.
Any help is appreciated, so thanks in advance.
Cheers
Use the following code works for chrome extension:
video = document.getElementsByClassName('video-stream')[0];
console.log(video);
console.log(video.currentTime);
In 2020, it seems we should use: player.playerInfo.currentTime.
full code on codepen
As Ben correctly assumed, you're executing the code in the context of your background page, which is wrong.
To interact with other pages, you need to inject a content script in them. See overview of that here. You'll probably need to learn how Messaging works so you can gather data in a content script and communicate it to the background page.
To make a minimal example, you can have a file inject.js
// WARNING: This example is out of date
// For the current HTML5 player, see other answers' code snippets
ytplayer = document.getElementById("movie_player");
ytplayer.getCurrentTime();
And in your background page script, you can inject that into the current page as follows (when the button is clicked, for instance)
chrome.browserAction.onClicked.addListener(function(tab) {
chrome.tabs.executeScript({
file: 'inject.js'
});
});
Then, you'll see the result of the execution in the console of the currently open page. To report the result back, you'll need to use chrome.runtime.sendMessage
you can only use getElementById when you´r referencing to the correct page. You´r using the right id. if you´r trying to access the play form another page you can use the jquery .load() function
---------EDIT----------
in the sample they do it like so:
function getCurrentTime() {
var currentTime = player.getCurrentTime();
return roundNumber(currentTime, 3);
}