I need to create a Pulsar cluster in a completely automatic way. First, the machines are created, then the software is installed. I am looking for such a solution, but the examples available on the internet relate to aws. In the book https://play.google.com/store/books/details?id=eYBTEAAAQBAJ there are links to git repo with an example that has been removed (https://github.com/josep2/pulsar-ias). Maybe someone has these files, or knows a tutorial about this topic. I'm a beginner in Azure and containers so forgive me if the question is silly.
Related
I'm with a dilema here about which SE site to ask this question so please help me out if it should be somewhere else.
I've been looking into Infrastructure as Code solutions.
Didn't like Terraform too much. The lack of intellisense makes discoberability harder than programmers have been used to.
I've been considering ARM templates. I like it that the templates are made available as we create resources in the portal but it seems way less readable and harder to maintain afterwards.
Then I found out Pulumi and love their idea compared to Terraform. The way I see it, they're approach is also declarative like the above options but we can use decent programming languages to get the job done.
The for loops is a must.
Cool, I like that! But since we like using C# (or other alternatives), then why don't we SDKs to manage our infrastructure as code?
Pulumi has compared themselves with cloud SKDs by positioning their solution as much safer advocating that, if we just use a cloud SDK ourselves, then our solution wouldn't be that reliable.
To what extent is this really true, I wonder?
Last year, I wrote some libraries that used Azure service bus queues/topics. There were several integration tests that would run in parallel and I needed to isolate them by creating new queues/topics and used Microsoft.Azure.ServiceBus.Management.ManagementClient to do this.
It really didn't seem like I had to learn anything at all.
Going to the point now. Not discarding Pulumi's innovation which I think is great:
Will Pulumi's really add that much benefit compared to using Azure SDKs?
What's been your experience with it?
A Pulumi developer here, so I'm definitely biased. I suspect the SO community may find your question violating some of the guidance, but I hope my answer survives :)
One upside of using Pulumi is that you get access to multiple providers with consistent developer experience. You may be using exclusively Azure, but you might at some point start combining it with things like building and publishing Docker images, deploying Kubernetes applications, or Datadog dashboards. All can be done from the same program or solution.
Now, the biggest difference with imperative SDKs is the notion of desired-state configuration. A Pulumi program describes the graph of resources and dependencies between them (what), not the steps to provision them (how). When you have an environment that lives for months and years, there's a big difference between evolving a single definition with baby steps and applying incremental changes (Pulumi) and writing a bunch of update scripts/programs to bring each environment to the new state (SDK).
How do you maintain multiple environments that may be similar but still different? (production vs staging vs test vs dev) How do you make sure that your short-lived infra that you created for nightly tests reflects the reality of production? What happens when an SDK program fails in the middle - can you retry running it again or will it create duplicate resources/fail with another error? How do you get a simple overview of changes over time in git? Concurrency control? Change history?
All the things above are baked into Pulumi and require manual consideration with a cloud SDK.
I´m planning a new web-project and want to implement node.js microservices. Now I´ve read a lot and found out that it will be a recommended way to do that with docker/kubernetes for availability and scaling.
It will be my first project and a have a lot of question now. Hopefully the community helps me out a bit with it:
Kubernetes-Cluster:
One of my first questions is the cluster. What is the best way to start with the cluster? Azure AKS is mostly used in web-tutorials, but isn´t it possible to start with an own server installing Docker and Kubernetes via a webhosting?
My biggest problem is that I don´t know to to calculate prices on Azure. For me it is clear that the cluster hasn´t any costs, but how do I calculate VM and Traffic.
And what about having a lot of API´s, should be each API an own AKS-Service on Azure = 1 VM which is 24 hours up and running?
It should be a webproject, all API´s must be available 24 hours.
Has anyone experiences with using Azure AKS with an similar project and has some explanations for me.
If it is possible with an webhosting package, has someone a tutorial how to start with it?
Thanks to all for help, hopefully there is someone to clarify some points for me.
According to #Bricktop -Please start using Kubernetes to get practice:
Locally:
Minikube
Kubeadm
In cloud (please create free account) you can find more details below:
Gcloud
Azure
AWS
After getting some knowledge about your solution, how it works what resources are necessary (related to this project - please use monthly billings provided by the Cloud-provider of your choice. You can use also "Pricing Calculator" from the official websites.
Hope this help
I am using HF for some time and trying different things regarding business network specification and configuration.
But, I have couple of question regarding best practices (if there are any yet) in using HF in production.
When we talk about using HF in production, should we use docker-compose-base.yaml, docker-compose-cli.yams, cofigtx.yaml.... etc. as files used to setup and configure our business network, and if not, can you please specify what is the best practice use-case?
Thank you for your answers.
You could use Docker Swarm/Compose with derivatives of the sample compose files you referenced, or you could use Kubernetes to manage a network (or subset of same). Project Cello is working on delivering such capability. The Ansible driver in particular has been demonstrated to work effectively - though it is far from a 1.0 level of maturity.
The reality is that you'll want to manage (likely) more than just four peer nodes all on the same VM or host, but manage multiple peers on multiple VMs/hosts even across multiple networks for a production deployment.
Further, you will obviously need to add management and monitoring to the deployed containers for a true production experience. The Hyperledger chat and mailing lists can be good sources of help and insight.
I have a few questions regarding the integration of the two tools. Not technical questions and how to setup( i will have my fun with that later ) but more on the course of the project and the direction, seeing that JanusGraph is still very young.
I am starting a new project and already decided to use Cassandra for storage and using a graph on top sounds very appealing to me.
A couple of things that i would like to know in advance before i take that road.
JanusGraph is very young and it picks up from where Titan left about a year or so ago. There is gap there but the fact that is part of the Linux Foundation and all the big players are going to support it sounds promising. Is it safe to assume at this point that JanusGraph is here to stay? Would it be safe to depend on Janus as a startup project? And follow development of course and be up to date as much as possible.
Cassandra. Titan/JanusGraph integrates with Cassandra 2.1.9 using the thrift api which will be deprecated eventually in Cassandra 4. I know that work is being done at the moment to make janus work with Cassandra 3 and eventually work with CQL as well. Is it safe to start with existing janus and Cassandra 2.1.9 and deal with the migration later on? Will it be a huge task for a startup to handle?
Production ready JanusGraph.(This question relates to any kind of software in it's early stages and whether it's safe for a start up to use). As i understand it, it will take some time for JanusGraph to be production ready and catch up with the rest of the tools it integrates with( although work is being done as we speak:)). Again would it be safe to start using Janus at this point and follow development and finally migrate to a production ready version? What is the overall roadmap for JanusGraph?
My concern in general is whether the combination of the tools is a safe choice for a start up. The whole stack is already new to us and we are excited to try and learn but we will hit a migration period pretty quickly. Is it something that you would do/recommend? Is it a suicide?
Please share your thoughts and keep in mind that it doesn't have to be about the stack i am talking about. It could be any startup company dealing with any kind of software in its early stages.
Cheers
Full disclosure, I'm a developer for JanusGraph on Compose.
It's as safe as any other OSS software project with a large amount of backers. Everyone could jump on some new toy tomorrow, but I doubt it. Companies are putting money into it and the development community is very active.
There is a CQL backend for Janus that's compatible with the Thrift data model. Migration to CQL should be simple and pretty painless when 0.2.0 is released.
I know there are already people using Titan for production applications. With JanusGraph being forked from Titan, I think it's pretty reasonable to start in with JanusGraph from everything I've seen. As far as a roadmap, I'd check out the JanusGraph mailing list (dev/users) and see what's going on and what's being talked about.
Disclosure: I am one of the co-founders of the JanusGraph project; I am also seeking out and adding production users to our GitHub repo and website, so I may be slightly biased. :)
Regarding your questions:
Is it safe to use?
The project is young, but it is built on a foundation of Titan, a very popular graph database that's been around since 2012 and has already been running in production. We have contributors from a number of well-known companies, and several companies are building their business-critical applications directly on JanusGraph, e.g.,
GRAKN.AI is building their knowledge graph on JanusGraph
IBM's Compose.io has built a managed JanusGraph service
Uber is already running JanusGraph in production (having previously run Titan)
several other companies run JanusGraph as a core part of their production environment
We are also starting to identify companies who will provide consulting services around JanusGraph in case someone needs production-level support for their own self-managed deployments.
So as you can see, there is significant interest in and support for this project.
Cassandra upgrade
#pantalohnes answered this question; I won't repeat it here.
Production readiness
As I linked above (GitHub repo and website), we already have production users of JanusGraph which you can find there. Those are just the companies that are publicly willing to lend their name/logo to the project; I'm sure there are more. Also, Titan has been running in many production environments for several years; JanusGraph is a more up-to-date version of Titan, despite the low version number.
I am also speaking with other companies who are planning to migrate to JanusGraph soon; look for announcements via the #JanusGraph Twitter handle to learn about more production deployments.
I'm writing an application in Node.js for a spare-time, bootstrap project. I have a Windows background and Windows Azure with three-month free trial currently seems like the simplest way to develop, deploy and host the project.
However Windows Azure appears to get expensive after the free trial expires, and in any case I'd like the option to host on non-MS platforms, so I have a couple of questions:
I can see from the tutorial that I need some Windows-specific code to import the port number at which the app should listen - are there many more examples of Windows or Azure specific code requirements further down the line?
I'd like to take a NoSQL approach to data storage since I'm more interested in flexibility and performance than in referential integrity or structural consistency - would it be difficult to wrap Azure Tables in a data access layer that would be reasonably portable to other NoSQL databases such as MongoDB or the various cloud offerings?
Finally, the catch-all question - is there anything else I should be looking out for?
Tackling your second question: there are modules in the NPM registry that can help you here.
Firstly Microsoft have recently released the Azure SDK for node as an NPM installation module. This has a rich API that will help you interface into Azure Tables.
There are also NoSQL clients available in the NPM registry for most solutions (including MongoDB).
If you keep your data access simple, you should be able to make use of the various NoSQL clients that are available and create a nice little module layer that sits above all the ones you need to support.
You could even create a public github repository and submit your hard work into the NPM registry for other people to help you develop.
I have built an app on Windows Azure's node.js support as well and there is virtually no lock in if you stick to npm modules and open platforms.
You should also check into Microsoft's Bizspark program - you get two years of 2 reserved instances for free + storage. Its a great program.