One of my goals with learning dialogFlow and programmatic webhooks to handle intens and responses is to see if it's possible to create an audio rich quiz.
In this quiz there's a section where I would like to have multiple questions/responses with a continuous audio background playing at the same time. Is this possible or must the Google Assistant or Google Home Speaker be silent when waiting for user input?
Also, if it is possible with the above, is it also possible to make crossfades between audio backgrounds triggered by events of some kind? That is like a change of scenery without interruption/silence.
There are a number of approaches to this, depending on your exact needs and limitations you have. Not everything is completely possible, but you may be able to get close.
Dialog with Background Audio
The easiest way to do this would be to use Google's version of SSML with parallel elements using the <par> and <media> tags. (But be aware that these are non-standard tags, if you want to be able to use them elsewhere.) With this, you would have one "track" for the dialog and one for the audio. It might look something like this:
<speak><par>
<media xml:id="track-0" begin="0s" soundLevel="+0dB">
<audio src="https://actions.google.com/sounds/v1/crowds/crowd_talking.ogg" >crowd talking</audio>
</media>
<media xml:id="track-1" begin="0.75s" soundLevel="+0dB">
<seq>
<media>
<speak><p>Well, hello there</p></speak>
</media>
<media begin="2.0s">
<speak><p>How are you?</p></speak>
</media>
</seq>
</media>
</par></speak>
Is there an easy way to design this?
You might want to check out the Nightingale Visual SSML Editor which has also been released as an open source project. It can help get you started, but tweak the SSML yourself.
What about cross-fades?
Sure! Just indicate that you're fading one track out, and fading another in, starting at some offset to the end of the other track.
<speak><par>
<media xml:id="track-0" begin="0s" fadeOutDur="6s">
<audio src="https://actions.google.com/sounds/v1/alarms/digital_watch_alarm_long.ogg" >digital watch alarm long</audio>
</media>
<media xml:id="track-1" begin="track-0.end-6s" fadeInDur="6s">
<audio src="https://actions.google.com/sounds/v1/human_sounds/baby_cry_long.ogg" >baby cry long</audio>
</media>
</par></speak>
And I can have this playing while the microphone is open and listening to a user?
Well... no. Not with SSML.
If you think about this as a conversation with someone, we typically require audio cues to know when to speak. Asking a question and then giving quiet space to reply is an excellent way to do this. In person - we have other cues (we can see the other person pause, for example), but if we just have audio, we only have silence.
The Assistant works based on this sort of conversational model, so wants to make it clear to the user when it is their turn to speak.
You can argue that for some kinds of "conversations", it is normal to have a different audio cue that prompts the user. And you'd be correct. But the Assistant needs to be a general-purpose Assistant.
Then how can I do this?
Since you're making a quiz, you can use the Interactive Canvas to create a page with HTML and JavaScript. With this, you can use an <audio> tag or the MediaStream JavaScript API to create the background audio. With this, you can have the microphone open while this media still plays, but there isn't any event that lets you know when the mic is open so you can duck the audio, which is a bit of a downside.
This gets around the audio-cue issues because there are visual cues when the microphone is open.
You can also use the SSML <mark> tag to trigger events in your JavaScript so you know when the SSML from the server begins, ends, or hits other points in the audio stream.
If there are visual cues, can I use the Interactive Canvas on a smart speaker?
Well... no.
But what you can do is use runtime surface capabilities to determine if the Interactive Canvas is supported.
If it is - use it (possibly along with SSML).
If not, use the SSML method.
Related
I am using dialogflow fulfillment, In my welcome intent first the audio file plays and then bot speaks.
is there a way to make both the things work simultaneously, means audio file should play in background in low volume and bot speak the welcome text over audio.
app.intent('welcome', (conv) => {
conv.ask(`<speak><audio src="https://actions.google.com/sounds/v1/cartoon/cartoon_cowbell.ogg"></audio><speak>Welcome! What would you like to check?</speak></speak>`);
});
Yes, you can use the <par> and <media> tags in Google's SSML to create parallel "tracks" of audio that are played at the same time. Note that these aren't standard SSML, but are Google extensions.
The <par> tag indicates that the contents are all to be played in parallel. Inside it, you typically have multiple <media> blocks (although you can also have other <par> blocks, as well as <seq> blocks for sequential portions). Each <media> block can have attributes indicating some adjustments for that block (such as volume or start offsets) and contains either <speak> or <audio> blocks.
So to do what you describe, you could have something like this:
<speak>
<par>
<media>
<audio src="https://actions.google.com/sounds/v1/cartoon/cartoon_cowbell.ogg"></audio>
</media>
<media>
<speak>Welcome! What would you like to check?</speak>
</media>
</par>
</speak>
Google also provides a tool to help you design this: https://actions-on-google-labs.github.io/nightingale-ssml-editor/
I'm building a feature in my app that makes use of input type file like this...
<input type="file" onChange={this.loadFile} accept="image/*" capture="camera" />
So when this input is activated on a phone it launches the phone camera. The resulting image is then saved in the file input. I then use HTML5 to base64 encode that image to a string and send it up to the server in an async post.
Here is the problem...
I then display these images taken by the camera in a list in a different part of the app. When I view these photos on the same device it looks fine. However, if I load the application into a desktop browser like chrome, the images appear sideways.
All the research I've done points to the fact that the phone is assuming I am taking a landscape picture when I'm really trying to take a portrait. It makes sense then that it would be displayed sideway, but it doesn't explain to me when it appears differently in different browsers. Also, the solution that I keep on hearing is that if I want to fix this, I need to open the image in a third party desktop app like photoshop, rotate it and save it. But that in the context I mentioned above is obviously not possible?
Is there a way to fix the orientation of the camera in the app? Or any other way to avoid this silly problem?
Thanks
I want to play a sound from a Chrome Extension, but I read that Content Scripts cannot play sounds. So I put this code in my background script:
var audio = new Audio("alert.ogg");
audio.play();
but still no sound. From the Console of the _generated_backgound_page.html I was able to inspect the Network tab and see that the sound file is loaded successfully, but I hear nothing.
Also, in another HTML page, I could load another Javascript, which has the above code, and it works (I can hear the sound.)
The fact that it is in the background script seems to have made it fail. Also, note that I do not have a background PAGE associated with the background SCRIPT. Does that matter？
edit: problem solved. The above code works but I forgot to listen to the right sound output channel.
You can generate the background page with javascript. I think the sound must be in a html page to be loaded and played.
This should work:
document.write('<audio id="player" src="data:audio/wav;base64,UklGRhwMAABXQVZFZm10IBAAAAABAAEAgD4AAIA+AAABAAgAZGF0Ya4LAACAgICAgICAgICAgICAgICAgICAgICAgICAf3hxeH+AfXZ1eHx6dnR5fYGFgoOKi42aloubq6GOjI2Op7ythXJ0eYF5aV1AOFFib32HmZSHhpCalIiYi4SRkZaLfnhxaWptb21qaWBea2BRYmZTVmFgWFNXVVVhaGdbYGhZbXh1gXZ1goeIlot1k6yxtKaOkaWhq7KonKCZoaCjoKWuqqmurK6ztrO7tbTAvru/vb68vbW6vLGqsLOfm5yal5KKhoyBeHt2dXBnbmljVlJWUEBBPDw9Mi4zKRwhIBYaGRQcHBURGB0XFxwhGxocJSstMjg6PTc6PUxVV1lWV2JqaXN0coCHhIyPjpOenqWppK6xu72yxMu9us7Pw83Wy9nY29ve6OPr6uvs6ezu6ejk6erm3uPj3dbT1sjBzdDFuMHAt7m1r7W6qaCupJOTkpWPgHqAd3JrbGlnY1peX1hTUk9PTFRKR0RFQkRBRUVEQkdBPjs9Pzo6NT04Njs+PTxAPzo/Ojk6PEA5PUJAQD04PkRCREZLUk1KT1BRUVdXU1VRV1tZV1xgXltcXF9hXl9eY2VmZmlna3J0b3F3eHyBfX+JgIWJiouTlZCTmpybnqSgnqyrqrO3srK2uL2/u7jAwMLFxsfEv8XLzcrIy83JzcrP0s3M0dTP0drY1dPR1dzc19za19XX2dnU1NjU0dXPzdHQy8rMysfGxMLBvLu3ta+sraeioJ2YlI+MioeFfX55cnJsaWVjXVlbVE5RTktHRUVAPDw3NC8uLyknKSIiJiUdHiEeGx4eHRwZHB8cHiAfHh8eHSEhISMoJyMnKisrLCszNy8yOTg9QEJFRUVITVFOTlJVWltaXmNfX2ZqZ21xb3R3eHqAhoeJkZKTlZmhpJ6kqKeur6yxtLW1trW4t6+us7axrbK2tLa6ury7u7u9u7vCwb+/vr7Ev7y9v8G8vby6vru4uLq+tri8ubi5t7W4uLW5uLKxs7G0tLGwt7Wvs7avr7O0tLW4trS4uLO1trW1trm1tLm0r7Kyr66wramsqaKlp52bmpeWl5KQkImEhIB8fXh3eHJrbW5mYGNcWFhUUE1LRENDQUI9ODcxLy8vMCsqLCgoKCgpKScoKCYoKygpKyssLi0sLi0uMDIwMTIuLzQ0Njg4Njc8ODlBQ0A/RUdGSU5RUVFUV1pdXWFjZGdpbG1vcXJ2eXh6fICAgIWIio2OkJGSlJWanJqbnZ2cn6Kkp6enq62srbCysrO1uLy4uL+/vL7CwMHAvb/Cvbq9vLm5uba2t7Sysq+urqyqqaalpqShoJ+enZuamZqXlZWTkpGSkpCNjpCMioqLioiHhoeGhYSGg4GDhoKDg4GBg4GBgoGBgoOChISChISChIWDg4WEgoSEgYODgYGCgYGAgICAgX99f398fX18e3p6e3t7enp7fHx4e3x6e3x7fHx9fX59fn1+fX19fH19fnx9fn19fX18fHx7fHx6fH18fXx8fHx7fH1+fXx+f319fn19fn1+gH9+f4B/fn+AgICAgH+AgICAgIGAgICAgH9+f4B+f35+fn58e3t8e3p5eXh4d3Z1dHRzcXBvb21sbmxqaWhlZmVjYmFfX2BfXV1cXFxaWVlaWVlYV1hYV1hYWVhZWFlaWllbXFpbXV5fX15fYWJhYmNiYWJhYWJjZGVmZ2hqbG1ub3Fxc3V3dnd6e3t8e3x+f3+AgICAgoGBgoKDhISFh4aHiYqKi4uMjYyOj4+QkZKUlZWXmJmbm52enqCioqSlpqeoqaqrrK2ur7CxsrGys7O0tbW2tba3t7i3uLe4t7a3t7i3tre2tba1tLSzsrKysbCvrq2sq6qop6alo6OioJ+dnJqZmJeWlJKSkI+OjoyLioiIh4WEg4GBgH9+fXt6eXh3d3V0c3JxcG9ubWxsamppaWhnZmVlZGRjYmNiYWBhYGBfYF9fXl5fXl1dXVxdXF1dXF1cXF1cXF1dXV5dXV5fXl9eX19gYGFgYWJhYmFiY2NiY2RjZGNkZWRlZGVmZmVmZmVmZ2dmZ2hnaGhnaGloZ2hpaWhpamlqaWpqa2pra2xtbGxtbm1ubm5vcG9wcXBxcnFycnN0c3N0dXV2d3d4eHh5ent6e3x9fn5/f4CAgIGCg4SEhYaGh4iIiYqLi4uMjY2Oj5CQkZGSk5OUlJWWlpeYl5iZmZqbm5ybnJ2cnZ6en56fn6ChoKChoqGio6KjpKOko6SjpKWkpaSkpKSlpKWkpaSlpKSlpKOkpKOko6KioaKhoaCfoJ+enp2dnJybmpmZmJeXlpWUk5STkZGQj4+OjYyLioqJh4eGhYSEgoKBgIB/fn59fHt7enl5eHd3dnZ1dHRzc3JycXBxcG9vbm5tbWxrbGxraWppaWhpaGdnZ2dmZ2ZlZmVmZWRlZGVkY2RjZGNkZGRkZGRkZGRkZGRjZGRkY2RjZGNkZWRlZGVmZWZmZ2ZnZ2doaWhpaWpra2xsbW5tbm9ub29wcXFycnNzdHV1dXZ2d3d4eXl6enp7fHx9fX5+f4CAgIGAgYGCgoOEhISFhoWGhoeIh4iJiImKiYqLiouLjI2MjI2OjY6Pj46PkI+QkZCRkJGQkZGSkZKRkpGSkZGRkZKRkpKRkpGSkZKRkpGSkZKRkpGSkZCRkZCRkI+Qj5CPkI+Pjo+OjY6Njo2MjYyLjIuMi4qLioqJiomJiImIh4iHh4aHhoaFhoWFhIWEg4SDg4KDgoKBgoGAgYCBgICAgICAf4CAf39+f35/fn1+fX59fHx9fH18e3x7fHt6e3p7ent6e3p5enl6enl6eXp5eXl4eXh5eHl4eXh5eHl4eXh5eHh3eHh4d3h4d3h3d3h4d3l4eHd4d3h3eHd4d3h3eHh4eXh5eHl4eHl4eXh5enl6eXp5enl6eXp5ent6ent6e3x7fHx9fH18fX19fn1+fX5/fn9+f4B/gH+Af4CAgICAgIGAgYCBgoGCgYKCgoKDgoOEg4OEg4SFhIWEhYSFhoWGhYaHhoeHhoeGh4iHiIiHiImIiImKiYqJiYqJiouKi4qLiouKi4qLiouKi4qLiouKi4qLi4qLiouKi4qLiomJiomIiYiJiImIh4iIh4iHhoeGhYWGhYaFhIWEg4OEg4KDgoOCgYKBgIGAgICAgH+Af39+f359fn18fX19fHx8e3t6e3p7enl6eXp5enl6enl5eXh5eHh5eHl4eXh5eHl4eHd5eHd3eHl4d3h3eHd4d3h3eHh4d3h4d3h3d3h5eHl4eXh5eHl5eXp5enl6eXp7ent6e3p7e3t7fHt8e3x8fHx9fH1+fX59fn9+f35/gH+AgICAgICAgYGAgYKBgoGCgoKDgoOEg4SEhIWFhIWFhoWGhYaGhoaHhoeGh4aHhoeIh4iHiIeHiIeIh4iHiIeIiIiHiIeIh4iHiIiHiIeIh4iHiIeIh4eIh4eIh4aHh4aHhoeGh4aHhoWGhYaFhoWFhIWEhYSFhIWEhISDhIOEg4OCg4OCg4KDgYKCgYKCgYCBgIGAgYCBgICAgICAgICAf4B/f4B/gH+Af35/fn9+f35/fn1+fn19fn1+fX59fn19fX19fH18fXx9fH18fXx9fH18fXx8fHt8e3x7fHt8e3x7fHt8e3x7fHt8e3x7fHt8e3x7fHt8e3x8e3x7fHt8e3x7fHx8fXx9fH18fX5+fX59fn9+f35+f35/gH+Af4B/gICAgICAgICAgICAgYCBgIGAgIGAgYGBgoGCgYKBgoGCgYKBgoGCgoKDgoOCg4KDgoOCg4KDgoOCg4KDgoOCg4KDgoOCg4KDgoOCg4KDgoOCg4KDgoOCg4KDgoOCg4KDgoOCg4KCgoGCgYKBgoGCgYKBgoGCgYKBgoGCgYKBgoGCgYKBgoGCgYKBgoGCgYKBgoGBgYCBgIGAgYCBgIGAgYCBgIGAgYCBgIGAgYCBgIGAgYCAgICBgIGAgYCBgIGAgYCBgIGAgYCBgExJU1RCAAAASU5GT0lDUkQMAAAAMjAwOC0wOS0yMQAASUVORwMAAAAgAAABSVNGVBYAAABTb255IFNvdW5kIEZvcmdlIDguMAAA" >');
document.getElementById('player').play();
I used a base64 audio file but you can use a normal file too.
I am a designer whose main marketing strategy is multi browser compatibility. I assure my clients that the site will work even in IE6 (!).
Of late i have been pondering over the question of moving to HTML 5. The reason behind my apprehension is that IE6 is still a major player in terms of market share and i don't want to lose it.
Is there any way of moving to HTML 5 and still promise multi browser compatibility?
Thank you.
Yes, by taking baby steps.
To start with, you can switch to the HTML5 doctype: <!DOCTYPE html>. This switches just about every browser out there into "standards" mode, the same as an HTML 4 strict doctype.
Then there's the new elements. Internet Explorer can't natively style them, but a handy little bit of javascript fixes that up: http://code.google.com/p/html5shiv/
If you or your tools aren't ready for that (e.g. some CMSs strip out HTML tags they don't understand), then in the interim you could use classes, e.g. instead of <article>, use <div class="article">.
As for the new form controls, they're backwards compatible too. So <input type="email"> will work exactly the same way as <input type="text"> in browsers that don't support it. If necessary you can use javascript to fill in the gaps. See http://diveintohtml5.ep.io/forms.html for more on that.
As for <video> and <audio>, you can fall back to <object> for older browsers - e.g. http://camendesign.com/code/video_for_everybody. Meanwhile <canvas> can be emulated in javascript, e.g. http://code.google.com/p/explorercanvas/.
"To HTML5" is a fairly broad statement. Even if you have the new HTML5 doctype set (the simple <!DOCTYPE html>) you don't have to go all out and use every aspect of HTML5, only what is appropriate to your project.
If you are keen to get on board with HTML5, I recommend reading "How to use HTML5 in your client work right now" for examples of how you can use certain aspects of HTML5 with few (if any) drawbacks.
Disclaimer: I am one of the curators of HTML5 Doctor.
From there it boils down to whether or not your project will benefit from the features of HTML5 and if you can afford to implement these features. For example, if all of your IE users also have JavaScript enabled you can use html5shiv to get IE to recognise the new elements, enabling you to use them and style them.
As for the new JS APIs and CSS properties that people often group with the term "HTML5", unless your site absolutely requires that you use the technology (perhaps something like geolocation), then it could simply be a matter of progressive enhancement. If webkit/firefox users get rounded corners from CSS3 and IE users don't, is that really such a big deal?
As a rule of thumb I would not develop a site purely in HTML 5 but would consider using it for certain, richer, parts of the site. Remember that it is still not recommended by W3C and IE barely supports it at all.
This blog has a good discussion on it: http://blogs.forrester.com/ronald_rogowski/10-05-10-what_should_customer_experience_professionals_do_about_html5
yes there are several ways.
but if you DONT need html5 elements like video tags, or html5 api's like browser databases, stay at XHTML,because it is still not recommended by W3C. There you can use simple fallbacks for ie6.
do you need html5 elements and apis?
As a last resort you could use Chrome Frame: http://code.google.com/chrome/chromeframe/
As for "a major player in terms of market share", that really depends on your audience. Even Microsoft is marketing its new IE versions rather aggressively. And I don't think there's shame in charging your IE 6 users an extra plugin installation fee. After all, their browser is 10 years old, which is about 100 Internet years, isn't it?
I have no choice but to play wav files directly in the browser (serverside encoding to mp3 isn't an option, unfortunately.)
What's the best way to do this? I'd really like to take advantage of the HTML 5 audio tag but my target audience includes many, many teens using IE6. As far as I'm aware flash isn't an option, but speedy playback really is critical.
Thanks.
Nowadays, the best way is probably just to use the HTML5 <audio> tag. In the past, you might have done it like this:
Background:
<embed src="bgsound.wav" hidden="true" autostart="true" loop="1">
On Click:
Play Sound
It's been a few years since the last answer. Embed tag was good but I had an issue trying to trigger it to play in JavaScript. New audio tag works well is most browsers.
<audio src="cat9.wav" preload></audio>
You can trigger it manually with audioElement.play()
<audio controls="controls"><source src="http://blablabla.com/hghghgh/my%20file.wav" type="audio/x-wav" /></audio>
Due to unfixed bug (issue from 2012 year?) sometimes Chromium-like browsers cannot play .wav from redirected URLs because they "think" that file size is zero. But .mp3 is OK.
Opera 12.14 and Firefox play from < audio >
tag normally in the same case (redirected URL).
I tried JPlayer's 2.9.* lib in Drupal 7 with 3 players, but not successful (was black inactive rectangle 300*14 pixels for each player).