Elixir Interop: Instantiating supervised comms w/ long-running, data streaming python process - python-3.x

Question

Ubuntu 20.04.1 LTS
Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Elixir 1.10.3 (compiled with Erlang/OTP 22)
Our Elixir project requires instantiating supervised communications with a long-running, data streaming Python process. Data will be pushed to its Elixir counterpart once every second. Both processes are running on the same machine. (Exile doesn’t seem to be ready for production environments, Porcelain/Erlport appear to have been abandoned, Rambo is only suitable for transient jobs, apparently, while Ports suffers from this this fatal flaw.)
Any stable libraries ideally suited for this? If so, where can we find their recipes for this use case?

Answer 1

while Ports suffers from this this fatal flaw.
Is it really a problem for your use case?
I think you don't need any library. Ports give you everything you need and they are simple enough to be used without 3rd party library.
These are some key points if it's your first time using Ports:
they are great because the processes are run outside the Erlang VM. A crash in the python script doesn't effect your Elixir processes.
it's an easy solution to run long-running processes (in any language).
when a port is closed, it doesn't kill the python process. It just closes the in/out file descriptors. In python you need to detect when in/out fd are closed (you just need to check if you receive an EOF).
Don't use stdin/stdout for the port <-> python communication. Use file descriptors 3 and 4 (:nouse_stdio option when opening a port).
take a look at the {:packet, N} option, it will make your life much easier!
Only the process which opens the port can send and receive messages (to the python process via the port). It's usually better to open the port in a dedicated GenServer, which becomes the interface to your python program. In this way the genserver process takes care of the Port, it can be supervised, and many Elixir processes can use the port sending messages to the genserver.
I've used Ports + Python to run a realtime YOLO, to detect object in realtime. I wrote in detail an article on how to use Ports with a long-running Python program, which I think it could useful: Real-time Object Detection with Phoenix and Python. I describe how to start and manage ports with Python, how to define a binary protocol, detect crashes and wrap the port with a GenServer and to make it supervised.
And here a great article written by Saša Jurić: Outside Elixir