Having implemented a couple years back a mechanism for signaling via a data channel message that a remote user muted their local video (e.g., set enable to false) and then taking the appropriate action on the remote side (e.g., showing the remote user avatar instead of the black video stream), i have been doing some testing on a non directly related function that got me looking at the states of the video tracks (i.e., the video tracks on the receive stream of the peer connection) and i notice that the muted state on the remote video fluctuates sometimes between true and false (though there is no actual change to the remote stream itself).
Hard to tell when this occurs exactly but seems that it MIGHT (no real idea whether this is actually the case or not) correlate to not attaching the media to an object (e.g., HTML video element for playback) for a long period of time (e.g., 10 seconds) and it seems that if it is attached in a short period the videotrack does not show a state of muted=true on the receiving side.
The W3 Media Capture and Streams Spec (see https://w3c.github.io/mediacapture-main/#track-muted) "A MediaStreamTrack is muted when the source is temporarily unable to provide the track with data. A track can be muted by a user. Often this action is outside the control of the application. This could be as a result of the user hitting a hardware switch or toggling a control in the operating system / browser chrome. A track can also be muted by the User Agent." The spec does not seem to address what the causes of this are or might be.
In the case of webRTC, can anyone provide some indication as to why the remote videostream as referenced from the webrtc peer connection might show a muted state of true when the media from the remote is actually flowing. Also, what might be the practical value or usage of the muted state on a remote video stream when it is not actually reflective of the remote state but of some local processing.
Thanks for any thoughts on this.
As the documentation says, the muted state varies from user action, network, or even the browser itself. In the case that it's muted while data is flowing, it could be because of your or the other users' browser (for example, could be many other reasons which don't really matter practically).
What is this for, you ask?
Many of these properties are used only for testing purposes (when delving deep into webrtc development), not suitable for production.
As it says in MDN:
When possible, avoid polling muted to monitor the track's muting
status. Instead, add event listeners for the mute and unmute events.
I just battled with this bug for the past 3 days and might have come to a conclusion now that it's solved.
Basically, I was writing a 2+ peer WebRTC app and needed to track different RTCPeerConnection objects with separate ids (like in a js object). When a third peer joined, my code had it asynchronously initialize multiple RTCPeerConnection objects and add local MediaStream tracks. This kept fluctuating the MediaStreamTrack for video on the revieving ends between muted and not muted.
Buggy code:
peerIDsArray.forEach(async(peerID) => {
// Initialize RTCPeerConnection object and configure it
// Runs in parellel
localMediaStream.getTracks().forEach(track => {
peerConnectionsObject[peerID].addTrack(track, localMediaStream)
})
})
This kept bugging out with mute/unmute MediaStreamTrack objects on receiving peers, and I think it's either the local MediaStream is not supposed to be tampered with asynchronously (in parallel) or you're not supposed to add tracks to multiple RTCPeerConnection objects at once. Running it in a simple loop solved the issue for me:
Working code:
for(let i = 0; i < peerIDsArray.length; i++) {
let peerID = peerIDsArray[i]
// Initialize RTCPeerConnection object and configure it
// Runs synchronously
localMediaStream.getTracks().forEach(track => {
peerConnectionsObject[peerID].addTrack(track, localMediaStream)
})
}
Related
I'm developing a system to control a range of IoT devices. Each set of devices is grouped into a "system" that monitors/controls a real-world process. For example system A may be managing process A and have:
3 cameras
1 accelerometer
1 magnetometer
5 thermocouples
The webserver maintains socket connections to each device. Users can connect (via a UI - again with WebSockets) to the webserver and receive updates about systems to which they are subscribed.
When a user wants to begin process A, they should press a 'start' button on the interface. This will start up the cameras, accelerometer, magnetometer, and thermocouples. These will begin sending data to the server. It also triggers the server to set the recording mode to true for each device, which means the server will write output to a database. My question:
Should I send a single 'start' request from javascript code in my UI to the server, and allow the server to start each device individually (how do I then handle an error, for example, if a single sensor isn't working - what about if two sensors don't work?). Or do I send individual requests from the UI to the server for each device, i.e. start camera 1, start camera 2, start accelerometer, start recording camera1, etc. and handle each success/error state individually?
My preference throughout the system so far has been the latter approach - one request, one response; with an HTTP error code. However, programming becomes more complex when there are many devices to control, for example - System B has 12 thermocouples.
Some components of the system are not vital - e.g. if 1 camera fails we can continue, however, if the accelerometer fails the whole system cannot run and so human monitoring is required. If the server started the devices individually from a single 'start' message, should I return an array of errors, or should the server know which components are vital and return a single error if a vital component fails? And in a failure state, should the server then handle stopping each sensor and returning to the original state - and what if that then fails? I foresee this code becoming quite complex with this approach.
I've been going back and forth over the best way to approach this for months, but I can't find much advice online around building complex, production-ready IoT systems for the real world. If anybody has any advice or could point me towards any papers/books/etc. I would really appreciate it.
Thanks in advance,
Tom
Suppose I have an IoT device which I'm about to control (lets say switch on/off) and monitor (e.g. collect temperature readings). It seems MQTT could be the right fit. I could publish messages to the device to control it and the device could publish messages to a broker to report temperature readings. So far so good.
The problems start to occur when I try to design the API to control the device.
Lets day the device subscribes to two topics:
/device-id/control/on
/device-id/control/off
Then I publish messages to these topics in some order. But given the fact that messaging is typically an asynchronous process there are no guarantees on the order of messages received by the device.
So in case two messages are published in the following order:
/device-id/control/on
/device-id/control/off
they could be received in the reversed order leaving the device turned on, which can have dramatic consequences, depending on the context.
Of course the API could be designed in some other way, for example there could be just one topic
/device-id/control
and the payload of individual messages would carry the meaning of an individual message (on/off). So in case messages are published to this topic in a given order they are expected to be received in the exact same order on the device.
But what if the order of publishes to individual topics cannot be guaranteed? Suppose the following architecture of a system for IoT devices:
/ control service \
application -> broker -> control service -> broker -> IoT device
\ control service /
The components of the system are:
an application which effectively controls the device by publishing messages to a broker
a typical message broker
a control service with some business logic
The important part is that as in most modern distributed systems the control service is a distributed, multi instance entity capable of processing multiple control messages from the application at a time. Therefore the order of messages published by the application can end up totally mixed when delivered to the IoT device.
Now given the fact that most MQTT brokers only implement QoS0 and QoS1 but no QoS2 it gets even more interesting as such control messages could potentially be delivered multiple times (assuming QoS1 - see https://stackoverflow.com/a/30959058/1776942).
My point is that separate topics for control messages is a bad idea. The same goes for a single topic. In both cases there are no message delivery order guarantees.
The only solution to this particular issue that comes to my mind is message versioning so that old (out-dated) messages could simply be skipped when delivered after another message with more recent version property.
Am I missing something?
Is message versioning the only solution to this problem?
Am I missing something?
Most definitely. The example you brought up is a generic control system, being attached to some message-oriented scheme. There are a number of patterns that can be used when referring to a message-based architecture. This article by Microsoft categorizes message patterns into two primary classes:
Commands and
Events
The most generic pattern of command behavior is to issue a command, then measure the state of the system to verify the command was carried out. If you forget to verify, your system has an open loop. Such open loops are (unfortunately) common in IT systems (because it's easy to forget), and often result in bugs and other bad behaviors such as the one described above. So, the proper way to handle a command is:
Issue the command
Inquire as to the state of the system
Evaluate next action
Events, on the other hand, are simply fired off. As the publisher of an event, it is not my business to worry about who receives the event, in what order, etc. Now, it should also be pointed out that the use of any decent message broker (e.g. RabbitMQ) generally carries strong guarantees that messages will be delivered in the order which they were originally published. Note that this does not mean they will be processed in order.
So, if you treat a command as an event, your system is guaranteed to act up sooner or later.
Is message versioning the only solution to this problem?
Message versioning typically refers to a property of the message class itself, rather than a particular instance of the class. It is often used when multiple versions of a message-based API exist and must be backwards-compatible with one another.
What you are instead referring to is unique message identifiers. Guids are particularly handy for making sure that each message gets its own unique id. However, I would argue that de-duplication in message-based architectures is an anti-pattern. One of the consequences of using messaging is that duplicates are possible, so you should try to design your system behaviors to be stateless and idempotent. If this is not possible, it should be considered that messaging may not be the correct communication solution for the need.
Using the command-event dichotomy as an example, you could perform the following transaction:
The controller issues the command, assigning a unique identifier to the command.
The control system receives the command and turns on.
The control system publishes the "light on" event notification, containing the unique id of the command that was used to turn on the light.
The controller receives the notification and correlates it to the original command.
In the event that the controller doesn't receive notification after some timeout, the controller can retry the command. Note that "light on" is an idempotent command, in that multiple calls to it will have the same effect.
When state changes, send the new state immediately and after that periodically every x seconds. With this solution your systems gets into desired state, after some time, even when it temporarily disconnects from the network (low battery).
BTW: You did not miss anything.
Apart from the comment that most brokers don't support QOS2 (I suspect you mean that a number of broker as a service offerings don't support QOS2, such as Amazon's AWS IoT service) you have covered most of the major points.
If message order really is that important then you will have to include some form of ordering marker in the message payload, be this a counter or timestamp.
I have some code in Lua that answers a call, and after performing a series of operations bridges the call to a new leg.
The operations take from a few seconds to several minutes.
To keep the client I need to play a sound the issue I have is that the playback is still going on after the call is bridged.
The specific question is, how to stop a sound called from a playback ?
My code looks like
session:answer()
session:execute("playback", '/some/file.wav')
.
.
.
local connectionString = '{bypass_media=true,origination_caller_id_number=555,destination_number=646}'
connectionString = connectionString .. 'sofia/external/192.168.0.1#1000'
session:execute('bridge', connectionString)
I had a similar task, and solved it by launching a new script for the outbound leg. When the outbound leg is answered, I send uuid_break to the inbound leg, and let the channels bridge together. It's done in Perl, but Lua should be quite similar: https://github.com/xlab1/freeswitch_secretary_bug (the scripts are in scripts directory).
From the mod_commands documentation:
uuid_break
Break out of media being sent to a channel. For example, if an audio
file is being played to a channel, issuing uuid_break will
discontinue the media and the call will move on in the dialplan,
script, or whatever is controlling the call.
Usage: uuid_break <uuid> [all]
If the all flag is used then all audio files/prompts/etc. that are
queued up to be played to the channel will be stopped and removed from
the queue, otherwise only the currently playing media will be stopped.
But in general, it's much easier to implement such scenarios via ESL: your program can handle multiple channels via ESL asynchronously, and perform all the needed playbacks and breaks easily. Here I made a simple prototype in Golang to implement a similar scenario via ESL: https://github.com/xlab1/go-fs-secretary-prototype (here I used the synchronous outbound ESL socket, but it shouldn't be too difficult to implement it also in asynchronous inbound mode).
I hope this helps :)
We are using socketIO on a large chat application.
At some points we want to dispatch "presence" (user availability) to all other users.
io.in('room1').emit('availability:update', {userid='xxx', isAvailable: false});
room1 may contains a lot of users (500 max). We observe a significant raise in our NodeJS load when many availability updates are triggered.
The idea was to use something similar to redis store with Socket IO. Have web browser clients to connect to different NodeJS servers.
When we want to emit to a room we dispatch the "emit to room1" payload to all other NodeJS processes using Redis PubSub ZeroMQ or even RabbitMQ for persistence. Each process will itself call his own io.in('room1').emit to target his subset of connected users.
One of the concern with this setup is that the inter-process communication may become quite busy and I was wondering if it may become a problem in the future.
Here is the architecture I have in mind.
Could you batch changes and only distribute them every 5 seconds or so? In other words, on each node server, simply take a 'snapshot' every X seconds of the current state of all users (e.g. 'connected', 'idle', etc.) and then send that to the other relevant servers in your cluster.
Each server then does the same, every 5 seconds or so it sends the same message - of only the changes in user state - as one batch object array to all connected clients.
Right now, I'm rather surprised you are attempting to send information about each user as a packet. Batching seems like it would solve your problem quite well, as it would also make better use of standard packet sizes that are normally transmitted via routers and switches.
You are looking for this library:
https://github.com/automattic/socket.io-redis
Which can be used with this emitter:
https://github.com/Automattic/socket.io-emitter
About available users function, I think there are two alternatives,you can create a "queue Users" where will contents "public data" from connected users or you can use exchanges binding information for show users connected. If you use an "user's queue", this will be the same for each "room" and you could update it when an user go out, "popping" its state message from queue (Although you will have to "reorganize" all queue message for it).
Nevertheless, I think that RabbitMQ is designed for asynchronous communication and it is not very useful approximation have a register for presence or not from users. I think it's better for applications where you don't know when the user will receive the message and its "real availability" ("fire and forget architectures"). ZeroMQ require more work from zero but you could implement something more specific for your situation with a better performance.
An publish/subscribe example from RabbitMQ site could be a good point to begin a new design like yours where a message it's sent to several users at same time. At summary, I will create two queues for user (receive and send queue messages) and I'll use specific exchanges for each "room chat" controlling that users are in each room using exchange binding's information. Always you have two queues for user and you create exchanges to binding it to one or more "chat rooms".
I hope this answer could be useful for you ,sorry for my bad English.
This is the common approach for sharing data across several Socket.io processes. You have done well, so far, with a single process and a single thread. I could lamely assume that you could pick any of the mentioned technologies for communicating shared data without hitting any performance issues.
If all you need is IPC, you could perhaps have a look at Faye. If, however, you need to have some data persisted, you could start a Redis cluster with as many Redis masters as you have CPUs, though this will add minor networking noise for Pub/Sub.
I have no clue if it's better to ask this here, or over on Programmers.SE, so if I have this wrong, please migrate.
First, a bit about what I'm trying to implement. I have a node.js application that takes messages from one source (a socket.io client), and then does processing on the message, which might result in zero or more messages back out, either to the sender, or other clients within that group.
For the processing, I would like to essentially just shove the message into a queue, then it works its way through various message processors that might kick off their own items, and eventually, the bit running socket.io is informed "Hey, send this message back"
As a concrete example, say a user signs into the service, that sign in message is then placed in the queue, where the authorization processor gets it, does it's thing, then places a message back in the queue saying the client's been authorized. This goes back to the socket.io socket that is connected to the client, along with other clients that might be interested. It can also go to other subsystems that might want to do more processing on authorization (looking up user info, sending more info to the client based on their data, etc).
If I wanted strong coupling, this would be easy, but I tried that before, and it just goes to a mess of spaghetti code that's very fragile, and I would like to avoid that. Another wrench in the setup is this should be cluster-able, which is where the real problem comes in. There might be more than one, say, authorization processor running. But the authorization message should be processed only once.
So, in short, I'm looking for a pattern/technique that will allow me to, essentially, have multiple "groups" of subscribers for a message, and the message will be processed only once per group.
I thought about maybe having each instance of a processor generate a unique name that would be used as a list in Reids. This name would then be registered with some sort of dispatch handler, and placed into a set for that group of subscribers. Then when a message arrives, the dispatch pulls a random member out of that set, and places it into that list. While it seems like this would work, it seems somewhat over-complicated and fragile.
The core problem is I've never designed a system like this, so I'm not even sure the proper terms to use or look up. If anyone can point me in the right direction for this, I would be most appreciative.
I think what your describing is similar to https://www.getbridge.com/ service. I it but ended up writing my own based on zeromq, it allows you to register services, req -> <- rec and channels which are pub / sub workers.
As for the design, I used a client -> broker -> services & channels which are all plug and play using auto discovery, you have the services register their schema with the brokers who open a tcp connection so that brokers on other servers can communicate with that broker groups services. Then internal services and clients connect via unix sockets or ipc channels which ever is preferred.
I ended up wrapping around the redis publish/subscribe functions a bit to do this. Each type of message processor gets a "group name", and there can be multiple instances of the processor within that group (so multiple instances of the program can run for clustering).
When publishing a message, I generate an incremental ID, then store the message in a string key with that ID, then publish the message ID.
On the receiving end, the first thing the subscriber does is attempt to add the message ID it just got from the publisher into a set of received messages for that group with sadd. If sadd returns 0, the message has already been grabbed by another instance, and it just returns. If it returns 1, the full message is pulled out of the string key and sent to the listener.
Of course, this relies on redis being single threaded, which I imagine will continue to be the case.
What you might be looking for is an AMQP protocol implementation,where you can have queue get custom exchanges,and implement a pub-sub model.
RabbitMQ - a popular amqp protocol implementation with lots of libraries
it also has node.js library