I am quite new to Service Discovery and clustered systems. I started experimenting with Mesos and Marathon for the deployment of Docker containers, the Marathon REST API and UI seem to do a good job.
My problem is the actual discovery of deployed services. For testing purposes I deployed a Kafka Cluster scaled to 3 instances via Marathon, as so I did with a MongoDB test-cluster. The Mesos-DNS client gives me a record like kafka.marathon.mesos and mongo.marathon.mesos which implies the dynamically mapped port from the host to the container. The problem is, that my client explicitly needs information about the target port. Is there a general way to get those port information from the service automatically and dymanically? What about apps exposing multiple ports?
My thougts so far:
- Doing a REST call to get a status about the deployed app and somehow extract the relevant data
- Doing a DNS SRV lookup and somehow extract the relevant data
- Having some kind of "master", statically bound to a port, with dynamic "clients".
I searched a lot for those informations but in the end most of the tutorials ended with a manual lookup which is not what I aim for.
You're spot on. I recently gave a presentation at XebiCon around this topic and plan to publish a blog post with details about the setup incl. GitHub repo. For starters you could have a look at a Python implementation for the HTTP API consumption part.
UPDATE: the blog post is now available here.
Related
I'm learning containers and how they work and started learning Kubernetes. I was wondering if there is a way I can pull or extract data from a Kubernetes cluster? What I want to do is to pull data from a cluster and monitor the information that is coming from it. I googled some and came up with some clients that work on nodejs and just wanted to know if I'm heading in the right direction.
what I'm trying to do is pull data from a cluster and put it on a front end website for me to see. (Thinking of making a compare CPU usage of different clusters site)
the link in question: https://github.com/tenxcloud/node-kubernetes-client
Look into heapster (https://github.com/kubernetes/heapster) with InfluxDB and Grafana as backends. Grafana is great for the type of monitoring and comparison that you seem to be looking for. This is sometimes included. With AKS in Azure heapster is there out of the box, but InfluxDB and Grafana must be added. I think that the entire stack is present in Google's offering.
I have a question regarding the Node Red dashboard. I've got my dashboard all set up and working. Now, I want to be able to access the dashboard outside of my local network. Right now I do this through a VNC server. What needs to happen next is that clients need to able to access the dashboard, but they are not getting access to my VNC server of course. I have done my fair amount of Google work. I (somewhat) understand that a service like ngrok (ngrok.com) or dataplicity (dataplicity.com) is what I am looking for. What would be the best way of setting this up safely?
Might be useful to clarify: I'm using a raspberry Pi!
Thanks in advance!
If you want to give the outside world access to your dashboard, you can also consider to host your node-red application in the cloud. See links at the bottom-left of page https://nodered.org/docs/getting-started/
Most of those services have a free tier - so it might you cost nothing.
If you cannot deploy your complete node-red in the cloud (e.g. because it is reading local sensors) then you can split your node-red application into 2 node-red applications: one running locally and one (with the dashboard) running in the cloud. Of course then the 2 node-red applications need to exchange messages: for this the cloud services mentioned on that page also provides a secure way to send and receive events from the node-red cloud application that you can use.
I am building a micro-services project using docker.
one of my micro-services is a listener that should get data from various number of sources.
What i'm trying to achieve is the ability to start and stop getting data from sources dynamically.
For example in this drawing, i have 3 sources connected to 3 dockers.
My problem starts because i need to create another docker instance when a new source is available. In this example lets say source #4 is now available and i need to get his data (I know when a new source became available) but i want it to be scaled automatically (with source #4 information for listening)
I came up with two solutions, each has advantages and disadvantages:
1) Create a docker pool of a large number of docker running the listener service and every time a new source is available send a message (using rabbitmq but i think less relevant) to an available docker to start getting data.
in this solution i'm a little bit afraid of the memory consumption of the docker images running for no reason - but it is not a very complex solution.
2) Whenever a new source is becoming available create a new docker (with different environment variables)
With this solution i have a problem creating the docker.
At this moment i have achieved this one, but the service that is starting the dockers (lets call it manager) is just a regular nodejs application that is executing commands on the same server - and i need it to be inside a docker container also.
So the problem here is that i couldn't manage create an ssh connection from the main docker to create my new Docker.
I am not quite sure that both of my solutions are on track and would really appreciate any suggestions for my problem.
Your question is a bit unclear, but if you just want to scale a service horizontally you should look into a container orchestration technology that will allow you that - For example Kubernetes. I recommend reading the introduction.
All you would need to do for adding additional service containers is to update the number of desired replicas in the Deployment configuration. For more information read this.
Using kubernetes (or short k8s) you will benefit from deployment automation, self healing and service discovery as well as load balancing capabilities in addition to the horizontal scalability.
There are other orchestration alternatives, too ( e.g. Docker Swarm), but I would recommend to look into kubernetes first.
Let me know if that solves your issue or if you have additional requirements that weren't so clear in your original question.
Links for your follow up questions:
1 - Run kubectl commands inside container
2 - Kubernetes autoscaling based on custom metrics
3 - Env variables in Pods
I want to have three different Neo4J instances running (each with a different database). Then I need three different Neo4J browser visualizers (think project1.domainname.com / project2.domainname.com / project3.domainname.com) with each one mapping to a specific database instance
I've managed to get the three different database instances running on a single Azure VM - so far so good.
But I'm not sure how to create and map those browsers. I'd like to run them each as Azure websites as that would help with some other problems I've forseen.
1) Where is this browser HTML etc. so I can load it to the Azure Website?
2) Where in that code would I specify the IP Address and Port that browser should be talking to?
I've also heard some people talking about an Azure for Neo4J project but that is nearly five years old and the Neo4J guys said to put the database instances on VM's. Were they right>
The Neo4j Browser is an angular app that you can find in the neo4j source code at https://github.com/neo4j/neo4j/tree/2.3/community/browser
You can set the host that the browser app can talk to:
:config host:"http://host:port"
This is an undocumented feature and might be removed.
For 3.0 the intent is to decouple browser and Neo4j anyway.
Currently I have two servers which I have deployed node.js/Express.JS based web services API. I am using Redis for caching the JSON strings.
What will be the best option deploying this setup in to production? I see here it advices to go with a dedicated server redis. OK. I take it and use a dedicated server for running redis master. Can I use existing app servers as slave nodes? Note : these app servers are running an Node/Express application.
What other other options do I have?
You can.
It all depends on the load that those other servers have, it's a problem of resource sharing. To be honest my main issue with your architecture is not the dedicated vs the non-dedicated servers, it's the fact that you are placing a Redis server (master or not) on a host that most likely will be facing the internet (expressJS app), meaning, it's quite exposed.
If you can simulate HTTP load into your Node/Express JS servers, see the difference between running some benchmark tests on your dedicated server vs the non dedicated ones:
On a running redis server type in:
redis-benchmark -q -n 100000
If the app servers are being hammered and using all cores frequently you should see a substantial difference in the benchmarks.
My suggestion is, go ahead with your first setup and add monitoring for the redis response times, and only act when you have to, which might be now if the benchmarks show very poor results.
As a side note, consider the option of not sharing hosts for services that you expose to the internet with services that perform internal functions to your application.