How can I use SSML in Dialogflow Fullfilment (Dutch language) - dialogflow-es

Question

is there a simple way to use SSML in the Fullfilment section using Actions-on-google functions. I tried all sorts of coding, but no good results. I'm using Dutch as default language.
In below example, Google Assistant is spelling each '<' etc.:
// Handle the Dialogflow intent named 'favorite color'.
// The intent collects a parameter named 'color'.
app.intent('favoriete kleur', (conv, {color}) => {
const luckyNumber = color.length;
const audioSound = 'https://www.example.com/MY_MP3_FILE.mp3'; // AoG currently only supports MP3!
if (conv.user.storage.userName) {
conv.ask(`<speak>${conv.user.storage.userName}, je geluksnummer is ${luckyNumber}<audio src="${audioSound}"><desc>Geluid wordt afgespeeld</desc></audio></speak>`); // Audio should have description
conv.ask(new Suggestions('Paars', 'Geel', 'Oranje'));
} else {
conv.ask(`<speak>Je geluksnummer is ${luckyNumber}<audio src="${audioSound}"><desc>Geluid wordt afgespeeld</desc></audio></speak>`);
conv.ask(new Suggestions('Paars', 'Geel', 'Oranje'));
}
});
Please find below the environment settings:
Index.js initiation settings:
'use strict';
// Import the Dialogflow module and response creation dependencies
// from the Actions on Google client library.
const {
dialogflow,
BasicCard,
Permission,
Suggestions,
Carousel,
MediaObject,
SimpleResponse,
Table,
Button
// Image,
} = require('actions-on-google');
// Import the firebase-functions package for deployment.
const functions = require('firebase-functions');
// Instantiate the Dialogflow client.
const app = dialogflow({debug: true});
package.json:
{
"name": "codelab-level-three",
"description": "Actions on Google Codelab Level 3",
"author": "Google Inc",
"private": true,
"scripts": {
"lint": "eslint .",
"serve": "firebase serve --only functions",
"shell": "firebase experimental:functions:shell",
"start": "npm run shell",
"deploy": "firebase deploy --only functions",
"logs": "firebase functions:log"
},
"dependencies": {
"actions-on-google": "^2.0.0",
"firebase-admin": "~5.8.1",
"firebase-functions": "^0.8.1",
"i18n": "^0.8.3"
},
"devDependencies": {
"eslint": "^4.19.0",
"eslint-config-google": "^0.9.1"
}
}
The produced payload looks like this:
"status": 200,
"headers": {
"content-type": "application/json;charset=utf-8"
},
"body": {
"payload": {
"google": {
"expectUserResponse": true,
"richResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "<speak>Paul, je geluksnummer is 5<audio src=\"https://www.example.com/MY_MP3_FILE.mp3\"><desc>Geluid wordt afgespeeld</desc></audio></speak>"
}
}
],
"suggestions": [
{
"title": "Paars"
},
{
"title": "Geel"
},
{
"title": "Oranje"
}
]
}
}
}
}
}

Answer 1

The most likely issue is that you're not including a closing </speak> tag. So you should probably write it as something like
conv.ask(`<speak>Je geluksnummer is ${luckyNumber}.` +
`<audio src="${audioSound}"></audio></speak>`);

Answer 2

The answer to my problem was easy: just read the f...ing manual. :-) Although both .OGG as also .MP3 are supported the website should provide HTTPS. Website which are not secured (like HTTP) are not supported. Below you can find the example function how you can test:
app.intent('favoriete muziek', conv => {
const Optie = conv.parameters.optie;
//Taalspecifieke meldingen
const SoundLib =
{
'1': {
description : 'Simple sound using .ogg',
audiosound : 'https://actions.google.com/sounds/v1/alarms/alarm_clock.ogg',
audiotext : 'You should hear an audio alarm signal',
},
'2': {
description : 'Music MP3 via HTTP',
audiosound : 'http://storage.googleapis.com/automotive-media/Jazz_In_Paris.mp3',
audiotext : 'You should hear a Jazz record called "Jazz in Paris" ',
},
'3': {
description : 'Longer MP3 file via HTTP',
audiosound : 'http://www.navyband.navy.mil/anthems/anthems/netherlands.mp3',
audiotext : 'You should hear now the Dutch National Anthem',
},
'4': {
description : 'short MP3 audio via HTTPS',
audiosound : 'https://ia802508.us.archive.org/5/items/testmp3testfile/mpthreetest.mp3',
audiotext : 'You should hear a short spoken intro text',
},
};
const Sound = SoundLib[Optie];
var spraakzin = "<speak>This text is using <say-as interpret-as='verbatim'>SSML</say-as> followed by an audio file in a SimpleResponse box: <audio src='" + Sound.audiosound + "'>The audio file could not be processed</audio></speak>";
if (!conv.surface.capabilities.has("actions.capability.MEDIA_RESPONSE_AUDIO")) {
conv.ask("Media response via audio is not supported on this device.");
return;
}
conv.ask(new SimpleResponse({
speech: spraakzin,
text: Sound.audiotext,
}));
});
More info can be found here: SSML examples - look at prerequisites of AUDIO

Answer 3

to add up to what Prisoner already said, there are some other problems.
app.intent('favoriete kleur', (conv, {color}) => {
const luckyNumber = color.length;
const audioSound = 'https://actions.google.com/sounds/v1/cartoon/clang_and_wobble.mp3'; // AoG currently only supports MP3!
if (conv.user.storage.userName) {
conv.ask(`<speak>${conv.user.storage.userName}, je geluksnummer is <audio src="${audioSound}"><desc>${luckyNumber}</desc></audio></speak>`); // Audio should have description
conv.ask(new Suggestions('Paars', 'Geel', 'Oranje'));
} else {
conv.ask(`<speak>Je geluksnummer is <audio src="${audioSound}"><desc>${luckyNumber}</desc></audio></speak>`);
conv.ask(new Suggestions('Paars', 'Geel', 'Oranje'));
}
});
AoG currently only support MP3 as audio format. See https://developers.google.com/actions/assistant/responses#media_responses Sorry, I was wrong. This only goes for Media Responses, NOT for embedded audio in SSML.
I removed concatenations in the code above. That's contraproductive and makes things more difficult to read than necessary. (Opinionated)
Audio output - when not only outputting a soundeffect, but text - should contain a description which will also be printed on the screen. The example in the code supplied should be okay.
But yes, what's the cause of your original problem is, is that you're not closing the audio tags. Actions on Google are pretty unforgiving concerning unclosed tags.
Hope that helped.