I'm trying to build an iOS Webapp that uses audio. While it has been a very fickle endeavor, I finally managed to get it to work in Safari Mobile (interestingly enough it worked in chrome mobile a long time before, I don't know why…). Yet when I save it as a webapp on the home screen, the audio stops working mysteriously…
Here is the audio code. window.helpers.gongis a base64 encoded mp3 file.
I checked the console output in the webapp via the desktop safari, yet there are no errors thrown.
Any ideas what might be going wrong?
window.helpers.audio = {
myAudioContext: null,
mySource: null,
myBuffer: null,
init: function() {
if ('AudioContext' in window) {
this.myAudioContext = new AudioContext();
} else if ('webkitAudioContext' in window) {
this.myAudioContext = new webkitAudioContext();
} else {
alert('Your browser does not support yet Web Audio API');
}
var self = this;
var load = (function (url) {
var arrayBuff = window.helpers.Base64Binary.decodeArrayBuffer(window.helpers.gong);
self.myAudioContext.decodeAudioData(arrayBuff, function(audioData) {
self.myBuffer = audioData;
});
}());
},
play: function() {
this.mySource = this.myAudioContext.createBufferSource();
this.mySource.buffer = this.myBuffer;
this.mySource.connect(this.myAudioContext.destination);
if ('AudioContext' in window) {
this.mySource.start(0);
} else if ('webkitAudioContext' in window) {
this.mySource.noteOn(0);
}
}
};
The code is called like this on load:
window.helpers.audio.init();
And later it is triggered through user action:
...
$('#canvas').click(function() {
if(this.playing == false) {
window.helpers.audio.play();
}
}.bind(this));
...
Ouch, the answer was blindingly simple:
I had the mute switch on the side of the iPhone set to mute the whole time.
So it turns out that safari plays audio even when the switch is on mute, yet when you save it as a web app, it doesn't work anymore.
If I understand correctly the audio works on desktop Safari, and not on mobile Safari?
This could be a result of a limitation placed on mobile Safari that requires any sound that is played to be a triggered in a user action (for example, a click).
Read more here:
http://buildingwebapps.blogspot.com/2012/04/state-of-html5-audio-in-mobile-safari.html
Related
I have a code that load a audio file with fetch and decode to an audiobuffer and then i create a bufersourcenode in web audio api to receive that audio buffer and play it when i press a button in my web page.
In chrome my code works fine. But in safari ... no sound.
Reading web audio api related questions with safari some people say that web audio api need to receive input from user in order to play sound.
In my case i have a button to be tapped in order to play the sound, a user input already. But it is not working.
I found an answer that tells the web audio api decodeAudiodata do not work with promises in safari and it must use an old syntax. I have tried the way the answer treat the decodeAudiodata but still no sound....
Please somebody can help me here? thanks for any help!
<button ontouchstart="bPlay1()">Button to play sound</button>
window.AudioContext = window.AudioContext || window.webkitAudioContext;
const ctx = new AudioContext();
let au1;
window.fetch("./sons/BumboSub.ogg")
.then(response => response.arrayBuffer())
.then(arrayBuffer => ctx.decodeAudioData(arrayBuffer,
audioBuffer => {
au1 = audioBuffer;
return au1;
},
error =>
console.error(error)
));
function bPlay1(){
ctx.resume();
bot = "Botão 1";
var playSound1b = ctx.createBufferSource();
var vb1 = document.getElementById('sld1').value;
playSound1b.buffer = au1;
var gain1b = ctx.createGain();
playSound1b.connect(gain1b);
gain1b.connect(ctx.destination);
gain1b.connect(dest);
gain1b.gain.value = vb1;
console.log(au1); ///shows in console!
console.log(playSound1b); ///shows in console!
playSound1b.start(ctx.currentTime);
}
I'm having an issue with my Google Assistant Action and using it in the Google Assistant Mobile app.
I am trying to play a tracklist of 1-3 minute mp3s using Media Responses and callbacks and it is working perfectly in the simulator and on my Google Home Mini, but not on the Google Assistant app on my phone.
What I've noticed happening is the MediaResponse callback isn't sent when I test on iPhone. The first MediaResponse will play but then the app is silent. It doesn't exit my action, though, it leaves the mic open and when I try to talk to it again whatever I say is sent to my action. This part is very similar to Starfish Mint's problem, though mine seems to work on my Google Home device. They said they fixed it by
"After waiting 6 months, We manage to solve it ourselves. On
MEDIA_FINISHED, we must return Audio text within your media response
to get subsequent MEDIA_FINISHED event. We tested this with playlist
of 100 media files and it plays like a charm."
though I'm not entirely sure what that means.
This might be an obvious answer to my question but where it says: Media responses are supported on Android phones and on Google Home , does this mean that they aren't supported on iPhone and that's the issue? Are there any workarounds for this, like using a Podcast action or something?
I have tried another audio playing app, the Music Player Sample app which is one of Google's sample Dialogflow apps and it also doesn't work on my phone though does in the other places. Maybe it is just an iPhone thing?
The thing that I find confusing, though, is when I look at the capabilities of the action on my phone: conv.surface.capabilities.has("actions.capability.MEDIA_RESPONSE_AUDIO")it includes actions.capability.MEDIA_RESPONSE_AUDIO in its capabilities. If it didn't have this I would be more inclined to believe it doesn't include iPhones but it seems weird that it would have it in the capabilities but then not work.
Here's the code where I am playing the first track:
app.intent('TreatmentResponse', (conv, context, params) => {
var treatmentTracks = [{url: 'url', name: 'name'},{url: 'url', name: 'name'}];
var result = playNext(treatmentTracks[0].url, treatmentTracks[0].name);
var response = result[0];
conv.data.currentTreatment = 'treatment';
conv.data.currentTreatmentName = 'treatmentName';
conv.data.treatmentPos = 1;
conv.data.treatmentTracks = treatmentTracks;
conv.ask("Excellent, I'll play some tracks in that category.");
conv.ask(response);
conv.ask(new Suggestions(['skip']));
});
and here is my callback function:
app.intent('Media Status', (conv) => {
const mediaStatus = conv.arguments.get('MEDIA_STATUS');
var { treatmentPos, treatmentTracks, currentTreatment, currentTreatmentName } = conv.data;
if (mediaStatus && mediaStatus.status === 'FINISHED' && treatmentPos < treatmentTracks.length) {
playNextTrack(conv, treatmentPos, treatmentTracks);
} else {
endConversation(conv, currentTreatment);
}
});
Here's playNextTrack()
function playNextTrack(conv, pos, medias) {
conv.data.treatmentPos = pos+1;
var result = playNext(medias[pos].url, medias[pos].name);
var response = result[0];
var ssml = result[1];
conv.ask(ssml);
conv.ask(response);
conv.ask(new Suggestions(['skip']));
}
and playNext()
function playNext(url, name) {
const response = new MediaObject({
name: name,
url: url,
});
var ssml = new SimpleResponse({
text: 'Up next:',
speech: '<speak><break time="1" /></speak>'
});
return [response, ssml];
}
The other issue is when the MediaResponse is playing on my iPhone if I interrupt it to say "Next" or "Skip", rather than using my "NextOrSkip" intent like it does in the simulator and on the Google Home Mini, it just says "sure" or "alright" [I don't have that in my code anywhere] and then is silent (and listening).
I've set up a wavesurfer audio model, which is working perfectly fine on chrome and firefox. It starts right away. When I want hit play on safari it waits for the whole file to be downloaded complete and only then it plays...I've experienced a similar problem on other pages that I open on safari as well....any ideas, why this could be the case and what to against it?
audioModel = WaveSurfer.create({
container: createdContainer,
waveColor: waveColor,
progressColor: waveColorProg,
height: height,
backend: 'MediaElement',
cursorWidth:0,
});
This might be the bug https://github.com/katspaugh/wavesurfer.js/issues/1215
Try the work around proposed by DrLongGhost which uses MediaElement in Safari (only). Other browsers work better with WebAudio backend. https://github.com/katspaugh/wavesurfer.js/issues/1215#issuecomment-415083308
// Only use MediaElement backend for Safari
const isSafari = /^((?!chrome|android).)*safari/i.test(navigator.userAgent || '') ||
/iPad|iPhone|iPod/i.test(navigator.userAgent || '');
const wavesurferArgs = {
container: document.getElementById('wavesurferContainerInternal'),
plugins
};
if (isSafari) {
wavesurferArgs.backend = 'MediaElement';
}
_wavesurfer = window.WaveSurfer.create(wavesurferArgs);
Update: ah you already are using MediaElement. We'll I'm not sure what the problem is.
In Chrome Extension Development we have Background Page Concepts. Is any thing similar available in Firefox Extension Development also. While Developing Chrome Extensions I have seen methods like
window.Bkg = chrome.extension.getBackgroundPage().Bkg;
$(function () {
var view = null;
if (Bkg.account.isLoggedIn()) {
view = new Views.Popup();
$("#content").append(view.render().el);
} else {
$("#content").append(Template('logged_out')());
Bkg.refresh();
}
}...........
Where the main logic are written in Background Page(like isLoggedIn etc) and from the Extension Popup page we are calling Background page. Here for instance the background page is always loaded which manages the session. How can we have similar functionality in Firefox Extension Development.
All communication between the background page (main.js) and content scripts (your popup script) occurs via events. You cannot call functions immediately, so you won't receive any return values, but you can send an event from a content script to the background script that sends an event back to the content script and calls a new function, like so:
main.js partial
// See docs below on how to create a panel/popup
panel.port.on('loggedIn', function(message) {
panel.port.emit('isLoggedIn', someBoolean);
});
panel.port.on('refresh', function() {
// do something to refresh the view
});
popup.js
var view = null;
addon.port.on('isLoggedIn', function(someBool) {
if (someBool) {
// Obviously not code that's going to work in FF, just want you to know how to structure
view = new Views.Popup();
$("#content").append(view.render().el);
} else {
$("#content").append(Template('logged_out')());
addon.port.emit('refresh');
}
});
addon.port.emit('loggedIn', 'This is a message. I can pass any var along with the event, but I don't have to');
You should read this stuff:
Panel
Communicating between scripts
Is it possible to build an 'incognito mode' for loading background web-pages in a browser extension?
I am writing a non-IE cross-browser extension that periodically checks web-pages on the user's behalf. There are two requirements:
Page checks are done in the background, to be as unobtrusive as possible. I believe this could be done by opening the page in a new unfocussed browser tab, or hidden in a sandboxed iframe in the extension's background page.
The page checks should operate in 'incognito mode', and not use/update the user's cookies, history, or local storage. This is to stop the checks polluting the user's actual browsing behavior as much as possible.
Any thoughts on how to implement this 'incognito mode'?
It would ideally work in as many browser types as possible (not IE).
My current ideas are:
Filter out cookie headers from incoming/outgoing http requests associated with the page checks (if I can identify all of these) (not possible in Safari?)
After each page check, filter out the page from the user's history.
Useful SO questions I've found:
Chrome extension: loading a hidden page (without iframe)
Firefox addon development, open a hidden web browser
Identify requests originating in the hiddenDOMWindow (or one of its iframes)
var Cu = Components.utils;
Cu.import('resource://gre/modules/Services.jsm');
Cu.import('resource://gre/modules/devtools/Console.jsm');
var win = Services.appShell.hiddenDOMWindow
var iframe = win.document.createElementNS('http://www.w3.org/1999/xhtml', 'iframe');
iframe.addEventListener('DOMContentLoaded', function(e) {
var win = e.originalTarget.defaultView;
console.log('done loaded', e.document.location);
if (win.frameElement && win.frameElement != iframe) {
//its a frame in the in iframe that load
}
}, false);
win.document.documentElement.appendChild(iframe);
must keep a global var reference to iframe we added.
then you can change the iframe location like this, and when its loaded it triggers the event listener above
iframe.contentWindow.location = 'http://www.bing.com/'
that DOMContentLoaded identifies all things loaded in that iframe. if the page has frames it detects that too.
to remove from history, into the DOMContentLoaded function use the history service to remove win.location from history:
https://developer.mozilla.org/en-US/docs/Using_the_Places_history_service
now to strip the cookies from requests in that page use this code:
const {classes: Cc, Constructor: CC, interfaces: Ci, utils: Cu, results: Cr, manager: Cm} = Components;
Cu.import('resource://gre/modules/Services.jsm');
var myTabToSpoofIn = Services.wm.getMostRecentBrowser('navigator:browser').gBrowser.tabContainer[0]; //will spoof in the first tab of your browser
var httpRequestObserver = {
observe: function (subject, topic, data) {
var httpChannel, requestURL;
if (topic == "http-on-modify-request") {
httpChannel = subject.QueryInterface(Ci.nsIHttpChannel);
var goodies = loadContextGoodies(httpChannel)
if (goodies) {
if (goodies.contentWindow.top == iframe.contentWindow.top) {
httpChannel.setRequestHeader('Cookie', '', false);
} else {
//this page load isnt in our iframe so ignore it
}
}
}
}
};
Services.obs.addObserver(httpRequestObserver, "http-on-modify-request", false);
//Services.obs.removeObserver(httpRequestObserver, "http-on-modify-request", false); //run this on shudown of your addon otherwise the observer stags registerd
//this function gets the contentWindow and other good stuff from loadContext of httpChannel
function loadContextGoodies(httpChannel) {
//httpChannel must be the subject of http-on-modify-request QI'ed to nsiHTTPChannel as is done on line 8 "httpChannel = subject.QueryInterface(Ci.nsIHttpChannel);"
//start loadContext stuff
var loadContext;
try {
var interfaceRequestor = httpChannel.notificationCallbacks.QueryInterface(Ci.nsIInterfaceRequestor);
//var DOMWindow = interfaceRequestor.getInterface(Components.interfaces.nsIDOMWindow); //not to be done anymore because: https://developer.mozilla.org/en-US/docs/Updating_extensions_for_Firefox_3.5#Getting_a_load_context_from_a_request //instead do the loadContext stuff below
try {
loadContext = interfaceRequestor.getInterface(Ci.nsILoadContext);
} catch (ex) {
try {
loadContext = subject.loadGroup.notificationCallbacks.getInterface(Ci.nsILoadContext);
} catch (ex2) {}
}
} catch (ex0) {}
if (!loadContext) {
//no load context so dont do anything although you can run this, which is your old code
//this probably means that its loading an ajax call or like a google ad thing
return null;
} else {
var contentWindow = loadContext.associatedWindow;
if (!contentWindow) {
//this channel does not have a window, its probably loading a resource
//this probably means that its loading an ajax call or like a google ad thing
return null;
} else {
var aDOMWindow = contentWindow.top.QueryInterface(Ci.nsIInterfaceRequestor)
.getInterface(Ci.nsIWebNavigation)
.QueryInterface(Ci.nsIDocShellTreeItem)
.rootTreeItem
.QueryInterface(Ci.nsIInterfaceRequestor)
.getInterface(Ci.nsIDOMWindow);
var gBrowser = aDOMWindow.gBrowser;
var aTab = gBrowser._getTabForContentWindow(contentWindow.top); //this is the clickable tab xul element, the one found in the tab strip of the firefox window, aTab.linkedBrowser is same as browser var above //can stylize tab like aTab.style.backgroundColor = 'blue'; //can stylize the tab like aTab.style.fontColor = 'red';
var browser = aTab.linkedBrowser; //this is the browser within the tab //this is where the example in the previous section ends
return {
aDOMWindow: aDOMWindow,
gBrowser: gBrowser,
aTab: aTab,
browser: browser,
contentWindow: contentWindow
};
}
}
//end loadContext stuff
}