I'm new to RabbitMQ and MQ's in general. I'm using the rabbit.js Node.js module to interface with RabbitMQ, so all my application layer is going to mainly be in Node.js. What I'm wondering is, how do I manage RabbitMQ? How can I see everything that's going on with RabbitMQ, from what's messages are left in the queue to general configuration and administration?
I'm looking for something visual, but more importantly, easy to use and simple.
RabbitMQ has a web interface (part of the rabbitmq_management plugin which ships with RabbitMQ, but needs to be enabled) that allows you to see the servers, exchanges, queues, etc.
It's pretty easy to use. One thing I would recommend is to set the time-interval on the graphs to 10 minutes. I find if you set them to longer, say, an hour plus, the information (due to the way it's bucketed, I think) gets a bit wonky.
Check out this link for more info: https://www.rabbitmq.com/management.html
There is also a JSON API that can be used to programmatically determine, for example, how many items are in a particular queue.
There's also a cmdline tool, called rabbitmqadmin (https://www.rabbitmq.com/management-cli.html) which can come in really handy for things like setting up test RabbitMQ test environments via a bash script and things of that nature.
check JXM.io sources (open source messaging backend for node.js / JXcore) that uses RabbitMQ for multi server integration and there is a nice article showing how to cluster RabbitMQ http://jxm.io/multi-server-messaging-backend-installation/
Related
For example, if i have main application (backend) and some microservice, e.g for image cropping.
User loads an image, making request to backend, backend using rabbitmq posts new task in the queue, then image cropping service pickup a task, completes it and i need somehow notify backend.
What is options for this? I need another microservice for such notifications?
so... there are reaaaaaaly many ways to do that.
On the high level, what you want to achieve is to produce an event that 1 or more services can react to. Now depending on what you have available, you can produce the event in a number of different ways.
if you want to be completely platform independent, you can use Apache Kafka. It's a popular service specifically for what we need -> publishing events and processing them at mass-scale. Kafka can be clustered, partitioned, have multiple parallel consumers of the same type (like multiple instances of your main backend service) or different types (3 different microservices that happen to be interested in a specific event). This bad boy just has it all and is famous for that. You can set up a cluster yourself or use one that comes out-of-the-box with some of the cloud platforms (like AWS for instance), but this might be more expensive and difficult to use compared to some cloud-specific fully-managed solutions.
if you're running your stuff on the google cloud, you can make it easier and cheaper by using the PubSub service. PubSub is a fully managed service that is scaled out-of-the-box (welcome to the cloud! you don't need to scale or cluster anything by yourself!).
if you're running on AWS, you can use SNS, or a more recent alternative - EventBridge (kinda like SNS, but booooooy what can it not do?). Yeah... I would recommend EventBridge. It can just do more... with the target filtering rules, payload transformations, it can automatically trigger more things...
Azure... ehm... Event Hub... but I haven't worked with this one yet... I'm not much of an Azurer... because you know... nobody uses azure for this kind of stuff...
I'm looking for something to make my 2 running apps to communicate with each other.
I have two apps - main.js and events.js and I want to send event from events.js to main.js. I don't even know what to google for that coz everything I can find seems a little bit outdated and I'm looking something different then redis. I found out I can use uuid to communicate between different node.js processes but I don't know how. Any information would be great!
Your best bet is to use a message queue system similar to Kue.
This way you will be able to make your apps communicate to each other even if they are on different servers.
If you want to work without the redis backend you can skip the filesystem entirely and move to Sockets/TCP communication, which is one of the ways of getting around semaphores. This is a common method of low latency communication used by applications on the same machine or across the network. I have seen and used this method in games as well as desktop/server applications.
Depending on how far down the rabbit hole you want to go there are a number of useful guides about this. The Node.js TCP API is actually pretty great.
Ex.
https://www.hacksparrow.com/tcp-socket-programming-in-node-js.html
I'm looking for an efficient way to subscribe to events in riak from node. I would like to be able to be notified of changes on an entry from riak.
For example when one node.js server updates an entry, another server using and watching that entry receives the updated entry or a notification about its update automatically.
If this is impossible is there an efficient messaging system that can be efficiently used across node.js servers?
Riak implements what are called Pre and Post commit hooks. Post-commits, which will be triggered when a write successfully occurs (and is presumably what you want) can only be written in Erlang code and Riak needs to be configured to trigger your custom Erlang function, as a property on the appropriate bucket.
Depending on your needs and the scale of your application, there can be several options for your Erlang setup to notify your Node.js server(s). It would be relatively easy to write an Erlang function that would send a HTTP request to your Node.js server, but that carries quite a lot of overhead, that may very well be inappropriate for your application. A lot Better, but slightly more complicated, would be to use a pub/sub system like those offered by Redis or ZeroMQ (just to name a couple), that are battle-tested and proven to perform very well under heavy load. If you want to go with ZeroMQ, see this guide on how to implement very reliable pub/sub.
Both of these messaging tools, as well as many others, can notify your Node.js instance of updates to watched entries from either Riak or the Node.js instance that's effectively modifying the data. The second option (Node.js to Node.js) might be simpler since it wouldn't require you to learn Erlang if you're not familiar with it. Both of these tools have node.js libraries that are very well-tested:
Zeromq.node
redis-node
And if you were to use them to send out notifications from within Riak as post-commit hooks, here are the corresponding erlang drivers:
Erlzmq2
Eredis
Is there an easy way to manage offline data with a web app, and synchronize with a server when there is a connection? I have been looking at Meteor, CouchDB and the likes, but still not sure what would be the least painfull way.
I could of course implement it myself with sockets or something similar, but if something is already made for the purpose, I don't see a reason to do it again.
I'm planning to work with Node as the server.
Thanks
You're talking about two things; 1) How to store/persist data if/when offline (storage mechanism), and 2) How to synchronize with a server when online (communication mechanism). The answer to 1 is some kind of local storage, and there any several ways of doing that (localstorage, websql, filesystem APIs etc) depending on your platform. The answer to 2 really depend on how urgent your synchronization needs are, but in general you can use HTTP itself with periodic (long-) polling, websockets and similar.
On top of both storage and communication mechanisms there are numerous libraries that make the job simpler, like Meteor (communication) and CouchDB (storage), but also many many more. There are even libraries that take care of the actual synchronization mechanism (with possible conflict resolution as well), but this very much depends on your actual application.
Updated: This framework looks promising, but I haven't tested it myself:
http://blog.nateps.com/announcing-racer-experimental-realtime-model
You might want to look at cloud services as well. These are best if you are developing a new application as they push you more to a serverless model, and of course you have to be happy using a service.
Simperium (simperium) is an interesting cloud service - the only one I can find today that does syncing (unlike Firebase and Spire.io who are similar in other respects), and for iOS it includes offline storage, while for JavaScript clients you'd need to cover the local storage yourself using HTML5 features. Backbone.js seems to have some support for this, and Simperium can integrate with Backbone, using a similar API style.
For non-cloud services, Derbyjs (derbyjs) is an open source project that includes Racer, a data synchronization library (mentioned by the earlier answer) - both are under rapid development and not yet complete, but look interesting if your timescales allow, and don't require a cloud service. There is a comparison of Derbyjs to Meteor that is useful - although it's written by the Derbyjs developers it's not too biased.
I also looked at CouchDB, which has some interesting built-in replication features, but I didn't like its use of indexes that are updated lazily when a query needs them (or by a batch process), and I wasn't happy with exposing the server DB directly to clients to enable replication/sync. Generally I think it's best to decouple the client side local storage from the server side DB, and of course for a web app it would be hard to use CouchDB on the client.
I'm writing a piece to a project that's responsible for processing tasks outside of the main application facing data server, which is written in javascript using Node.js. It needs to handle tasks which are scheduled in the future and potentially handle tasks that are "right now". The "right now" just means the next time a worker becomes available it will operate on that task, so that bit might not matter. The workers are going to all talk to external resources, an example job would be to send an email. We are a small shop and we don't have a ton of resources so one thing I don't want to do is start mixing languages at this point in the process, and I already see that Node can do this for us pretty easily, so that's what we're going to go with unless I see a compelling reason not to before I start coding, which is soon.
All that said, I can't tell if there is a compelling reason to use an AMQP based server, like OpenAMQ or RabbitMQ over something like Kue or Beanstalkd with a node client. So, here we go:
Is there a compelling reason to use an AMQP based server over something like beanstalkd or redis with Kue? If yes, which AMPQ based server would fit best with the architecture that I laid out? If no, which nosql solution (beanstalkd, redis/Kue) would be easiest to set up and fastest to deploy?
FWIW, I'm not accepting my answer yet, I'm going to explain what I've decided and why. If I don't get any answers that appear to be better than what I've decided, I'll accept my own later.
I decided on Kue. It supports multiple workers running asynchronously, and with cluster it can take advantage of multicore systems. It is easily extended to provide security. It's backed with Redis, which is used all over for this exact thing, so I know I'm not backing my job process server with unproven software (that's not to say that any of the others are unproven.)
The most compelling reasons that I picked Kue is that it provides a JSON api so that the client applications (The first client is going to be a web based application, but we're planning on making smartphone apps also) can add jobs easily without going through the main application facing node instance, so I can be totally out of the way of the rest of my team as I write this. I don't need a route, I don't need anything, and it's all provided for me so I don't need to write anything to support this. This has another advantage, with an extention to provide l/p security only authorized clients can add jobs, so I don;t have to expose my redis server to client applications directly. It also has a built in web console and the API allows the client to pull back lists of jobs associated with a given user very easily, so we can show the user all of their scheduled tasks in a nifty calendar view with 0 effort on my part.
The other compelling reason is the lack of steep learning curve associated with getting redis and Kue going for me. I've set up redis before, and Kue is simple and effective.
Yes, I'm a lazy developer, but I'm the good kind of lazy developer.
UPDATE:
I have it working and doing jobs, the throughput is amazing. I split out the task marshaling logic into it's own node instance, basically all I have to do is deploy my repo to a new machine and run node task-server.js to scale out my workers. I may need to add in some more job searching calls to Kue, because of how I implimented a few things, but that will be easy.