Does Apache Pulsar support message chunking in its Python client library?
https://pulsar.apache.org/api/python/2.10.1/pulsar.html#Client.create_producer <-- missing how to enable chunking
https://pulsar.apache.org/docs/next/client-libraries-python/ <--- no mention on how to enable chunking
https://pulsar.apache.org/docs/next/concepts-messaging/ <-- no mention if chunking if limited to a specific client library
Related
We have a Redis server that all clients attach to for a variety of data transfer and coordination tasks. We have a new requirement that we support video streaming. I would like to avoid running a dedicated service (with all the accompanying network and security requirements that entails) and just stream over Redis.
Redis seems like a good fit for real time streaming, in particular using Redis streams. I realize that "Redis streams" have no relation to "video streaming", however, our use case follows Redis stream structure well. We want to buffer X seconds of video continuously allowing clients to attach to that real-time stream at any time. We have no need to store history or serve static video content.
Redis seems like a good solution, my problem is I don't know how to
stream an appropriate video codec (Motion JPEG maybe?) over Redis.
I wouldn't know how to join a stream mid-broadcast (join at a keyframe
perhaps?).
I wouldn't know how to serialize the stream to bytes at
the server (Python based) and de-serialize the stream to a video codec and player on
the client (a browser). Perhaps it's as simple as seralization/deseralization in opencv or equivalent and I'm just over thinking it?
These are all features I would typically look to an API to perform, but is there an API is capable of this? I'm inexperienced in the field of video streaming.
At a high level, I prefer viewing streaming as a pub-sub problem. Where producers produce chunks of information and consumers read that information on need basis.
Some solution may not be readily available, we may need to perform the following steps:
Publish:
1. chunk-id : content
2. chunk-id-fwd : (nextChunkId)
3. videoId : latestChunkId (Assuming your realtime usecase is for live streams, this can help users access 'go-live' button)
Consume:
Start:
1. Get latest chunk
2. Get content from latest chunkId
3. Get nextChunkId from chunk-id-fwd
I have been trying to read this data (Topics) to my MSK Cluster from my React application, I have found very few tools and information that can help me, and I have not succeeded
How can I get this data from MSK cluster and display them in a React table?
Some of the items I found: https://www.npmjs.com/package/#aws-cdk/aws-msk, https://docs.aws.amazon.com/es_es/msk/latest/developerguide/client-access.html, https://akhq.io/, https://github.com/Microsoft/kafka-proxy-ws
First, #aws-cdk/aws-msk is a library to provision a cluster, not read from it.
React cannot consume from Kafka since it is client-side.
The AKHQ project (and similar Kafka GUI projects, like Confluent Control Center) generally use Java to render frontend HTML (AKHQ uses SSE with JSX, if you look at TailController.java) after using the native Java KafkaConsumer to poll a batch of records.
Basically, if your server and frontend are separate components, you need some type of proxy, such as the websocket from Microsoft (which has been archived so may no longer work with newer Kafka protocol versions), the Confluent REST Proxy, and/or your own backend server that is acting as a Kafka consumer that can expose messages to the frontend via some interface such as REST, GraphQL, WebSockets, or SSE.
The simplest solution, in my opinion, would be to use the REST Proxy
As we know MQTT have using subscribe/publish method. May i know what platform user can save the database using MQTT protocol. Its hivemq or mosquito support database so i can see previous data recorded from the sensor?
If MQTT can support database. What other method beside using apache webserver.
MQTT is a Pub/Sub protocol, it is purely for delivering messages. What happens to those messages once delivered is not the concern of the protocol.
If you want to store all messages then you are going to need to implement that yourself.
This is either as:
A dedicated client that will subscribe to # and then store the messages to a database.
Some brokers have a plugin API that will allow you to register hooks that can intercept every message and store that to a database.
You will have to research if any broker you want to use supports plugins of this nature.
I would like to generate ClientHello and ServerHello messages using NodeJs for testing purposes.
Is there an easy way to generate these messages using NodeJS API (the HTTPS all ready doing it but I can't find where).
If I can't use an existing API then how do I create the simplest messages according to the RFC using NodeJS (How to build the messages in a Buffer correctly)
I'm looking for a stable AMQP 1.0 client for NodeJS so that I can consume events from Azure Event Hubs.
I have tried using https://github.com/noodlefrenzy/node-amqp10, but it chokes sporadically on certain payloads with messages like malformed payload: Unknown code prefix: 0x30 and malformed payload: Unable to decode bytes from message body: 005375b00002810c5b7b226576656e7474797065223 (...).
I would appreciate any direction people can offer for a stable AMQP 1.0 client in NodeJS. I am surprised that I have not been able to find one so far, since otherwise it seems Azure is failing to support an entire developer base.
I'm the (co-)author of https://github.com/noodlefrenzy/node-amqp10. The error you were seeing was due to large messages being split across multiple transfer frames - something we hadn't encountered before, so hadn't implemented. We just added support for this in v2.1.0, so please check out our library again.
My co-author uses this in production against QPID and has been for months, and we both actively support it as we have time, so if there are any instabilities or missing features, we will try our best to address them. Often they are just issues where we haven't needed a feature of the (incredibly large) spec or have misinterpreted the (incredibly scattered) spec details.
You can also use the official JavaScript SDK for Azure Event Hubs, #azure/event-hubs which internally uses rhea