Typically, access to Azure workers is done via endpoints that are defined in the service definition. These endpoints, which must be TCP or HTTP(S), are passed through a load balancer and then connected to the actual IP/port of the Azure machines.
My application would benefit dramatically from the use of UDP, as I'm connecting from cellular devices where bytes are counted for billing and the overhead of SYN/ACK/FIN dwarfs the 8 byte packets I'm sending. I've even considered putting my data directly into ICMP message headers. However, none of this is supported by the load balancer.
I know that you can enable ping on Azure virtual machines and then ping them -- http://weblogs.thinktecture.com/cweyer/2010/12/enabling-ping-aka-icmp-on-windows-azure-roles.html.
Is there anything preventing me from using a TCP-based service (exposed through the load balancer) that would simply hand out an IP address and port of an Azure VM address, and then have the application communicate directly to that worker? (I'll have to handle load balancing myself.) If the worker gets shut down or moved, my application will be smart enough to reconnect to the TCP endpoint and ask for a new place to send data.
Does this concept work, or is there something in place to prevent this sort of direct access?
You'd have to run your own router which exposes an input (external) endpoint and then routes to an internal endpoint of your service, either on the same role or a different one (this is actually how Remote Desktop works). You can't directly connect to a specific instance by choice.
There's a 2-part blog series by Benjamin Guinebertière that describes IIS Application Request Routing to provide sticky sessions (part 1, part 2). This might be a good starting point.
Ryan Dunn also talked about http session routing on the Cloud Cover Show, along with a follow-up blog post.
I realize these two examples aren't exactly what you're doing, as they're routing http, but they share a similar premise.
There's a thing called InstanceInputEndpoint which you can use for defining ports on the public IP which will be directed to a local port on a particular VM instance. So you will have a particular port+IP combination which can directly access a particular VM.
<InstanceInputEndpoint name="HttpInstanceEndpoint" protocol="tcp" localPort="80">
<AllocatePublicPortFrom>
<FixedPortRange max="8089" min="8081" />
</AllocatePublicPortFrom>
</InstanceInputEndpoint>
More info:
http://msdn.microsoft.com/en-us/library/windowsazure/gg557552.aspx
Related
In the world of microservices endpoints should not (must not) be hardcoded. One of the best ways to do this is to have a DNS and let each microservice register while starting. By doing this whenever microservice A wants to communicate with microservice B it just asks DNS for endpoints where B currently listens.
What I do not understand is: How microservices know where the DNS lives?
Basically DNS is just a 'special' service and I can have one or multiple instances of it right? So I should not hardcode it's endpoint too or should I? And let's say I do - what if DNS instnace is moved to different location? Do I have to manually change it's location in configuration?
Does anyone happen to know how to design this? (or can anyone just point me to any document where this is explained since although there are many information about microservices and dns I can not find this particular information anywhere - maybe it's just too trivial and I am the only one who does not get it)
Manual setup of DNS is possible, as stated by the other answers, but I would recommend to use an infrastructure that supports the service discovery in all respects. For example kubernetes has built in DNS support and makes it very easy to expose a service that can consist of any number of Pods.
An infrastructure technology like kubernetes will also make many other respects of the microservices architectural style easier to implement, including high availability and scalability.
Please see the official docs for some more information.
DHCP solves this problem. When a host boots it sends a broadcast DHCP message. The DHCP server responds with many values, one of which is the location of DNS servers.
In the case of micro services, the host OS (or container host) will be configured for DNS via DHCP. The microservice code uses the OS DNS functions to resolve addresses.
https://en.m.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol
You can use your local network to discover services, via Dhcp and whatnot. But that requires that all services are already "registered" within that DNS server.
Microservices can find each other via service discovery, server or client side. If you choose client side service discovery, you can use tools like Consul, which provides a bunch of great features. One of which is a DNS endpoint which allows queries via SRV records with <serviceName>.consul.service domain names.
Consul has it's own DNS endpoint, you can configure your services to use that (usually on port 8600 locally, as Consul agents run locally).
But you can also configure an actual DNS server to forward questions to Consul, so that you can easily mix service discovery drive by Consul with manually setup services within a Bind instance or similar...
Known hostname solution. The fixed part would be the service domain name, for instance xservice.com. You can query this host using standard DNS tools (e.g., dig in your shell, etc).
Finally, in the DNS bound to xservice.com you then add a SRV record with further details.
A SRV record lists all the service details, including:
the symbolic service name;
the canonical hostname of the machine providing the service;
the TCP (or UDP) port on which the service is available.
There are many other info as well. Please see Wikipedia for the complete list.
Please keep in mind this is a somewhat static solution. If you are looking for a more dynamic one, then Oswin answer might be a better fit :-)
We've got an API micro-services infrastructure hosted on Azure VMs. Each VM will host several APIs which are separate sites running on Kestrel. All external traffic comes in through an RP (running on IIS).
We have some API's that are designed to accept external requests and some that are internal APIs only.
The internal APIs are hosted on scalesets with each scaleset VM being a replica that hosts all of the internal APIs. There is an internal load balancer(ILB)/vip in front of the scaleset. The root issue is that we have internal APIs that call other internal APIs that are hosted on the same scaleset. Ideally these calls would go to the VIP (using internal DNS) and the VIP would route to one of the machines in the scaleset. But it looks like Azure doesn't allow this...per the documentation:
You cannot access the ILB VIP from the same Virtual Machines that are being load-balanced
So how do people set this up with micro-services? I can see three ways, none of which are ideal:
Separate out the APIs to different scalesets. Not ideal as the
services are very lightweight and I don't want to triple my Azure VM
expenses.
Convert the internal LB to an external LB (add a public
IP address). Then put that LB in it's own network security
group/subnet to only allow calls from our Azure IP range. I would
expect more latency here and exposing the endpoints externally in
any way creates more attack surface area as well as more
configuration complexity.
Set up the VM to loopback if it needs a call to the ILB...meaning any requests originating from a VM will be
handled by the same VM. This defeats the purpose of micro-services
behind a VIP. An internal micro-service may be down on the same
machine for some reason and available on another...thats' the reason
we set up health probes on the ILB for each service separately. If
it just goes back to the same machine, you lose resiliency.
Any pointers on how others have approached this would be appreciated.
Thanks!
I think your problem is related to service discovery.
Load balancers are not designed for that obviously. You should consider dedicated softwares such as Eureka (which can work outside of AWS).
Service discovery makes your microservices call directly each others after being discovered.
Also take a look at client-side load balancing tools such as Ribbon.
#Cdelmas answer is awesome on Service Discovery. Please allow me to add my thoughts:
For services such as yours, you can also look into Netflix's ZUUL proxy for Server and Client side load balancing. You could even Use Histrix on top of Eureka for latency and Fault tolerance. Netflix is way ahead of the game on this.
You may also look into Consul.io product for your cause if you want to use GO language. It has a scriptable configuration for better managing your services, allows advanced security configurations and usage of non-rest endpoints. Eureka also does these but requires you add a configuration Server (Netflix Archaius, Apache Zookeeper, Spring Cloud Config), coded security and support accesses using ZUUL/Sidecar.
What do you suggest as the best way to protect your web servers IP address for outgoing requests? I'm already using Cloudflare for inbound requests but if my web server (nodejs) is making outbound connections for sending webhooks or something, I would prefer not to expose my origins IP. I have a firewall set up to prevent any connections inbound not coming from Cloudflare but I don't want my IP to expose where I'm hosted only to have my datacenter receive a DDoS.
There actually aren't any good articles I can find anywhere regarding protecting your IP with outbound connections.
Two thoughts:
1) Set up a second datacenter containing proxy servers and route outbound web server traffic through the proxy servers.
2) Set up a webhook queue, send webhooks to the queue and have servers in a 2nd datacenter work the queue.
Ideas?
I have worked at my company with a number of models over the years, including both ones that you listed. We started out using a queue that were available to web hook processors on remote data centers, but we transitioned over to a model that had less emphasis on queues, and instead simplified it; an originating server chooses one of the available notification/web hook senders, that in turns calls the web hook subscriber. The sender also takes care of buffering, resending, alerting and aging of messages.
For the purpose of protecting your IP address, it depends on a number of variables. In our case, we acquire additional IP address ranges for the senders, but you can achieve your goal by having the proxy hosted on AWS or similar.
Why would you want to do this? Your inbound requests are already dropped if they aren't from cloudflare.
In this link (Azure: security between web roles) the OP asks: "In Azure, if you choose to use internal endpoint (instead of input endpoint), https is not an option. http & tcp are the only options. Does it mean internal endpoint is 100% secure and you don't need encryption"
the answer he gets is: No, a web/worker role cannot connect to an internal endpoint in another deployment
My question is possible at all to deploy such a solution?
Thanks
Joe
There are two separate things you brought up in your question.
Internal endpoints are secure in that the only other VM instances that can access these are within the same deployment. If, say, a web app needs to talk to a WCF service on a worker role instance, it can direct-connect with a tcp or http connection, with no need for encryption. It's secure.
Communication between deployments requires a Virtual Network, as internal endpoints are not accessible outside the boundary of the deployment. You can connect two deployments via Virtual Network, and a that point each of the virtual machine instances in each deployment may see each other. The notion of endpoints is moot at this point, as you can simply connect to a specific port on one of the server instances.
The Azure documentation says that internal endpoints on a web role will not be load balanced. What is the practical ramification of this?
Example:
I have a web role with 20 instances. If I define an internal endpoint for that web role, what is the internal implementation? For example, will all 20 instances still service this end point? Can I obtain a specific endpoint for each instance?
We have a unique callback requirement that could be nicely served by utilizing the normal load balancing behavior on the public endpoint, but having each instance expose an internal endpoint. Based on the published numbers for endpoint limits, this is not possible. So, when defining an internal endpoint, is it "1 per instance", or what? Do all of the role instances service the endpoint? What does Microsoft mean when they say that the internal endpoint is not load balanced? Does all the traffic just flow to one instance? That does not make sense.
First let's clarify the numbers and limitations. The limitations for EndPoints is for Roles, not for Instances. If you are not sure, or still confusing Roles and Instances terms, you can check out my blog post on that. So, the limit is Per Role(s).
Now the differences between the EndPoints - I have a blog post describing them here. But in a quick round, Internal EndPoint will only open communication internally within the deployment. That's why it is Internal. No external traffic (from Internet) will be able to go to an Internal Endpoint. In that terms, it is not load balanced, because no traffic goes via/through a load balancer! The traffic of internal EndPoints only goes between Role Intances (eventually via some internal routing hardwere) but never lives a deployment boundaries. Having said that, it must already be clear that no Internet traffic can be sent to an Internal EndPoint.
A side note - InputEndpoint however is discoverable from Internet and from Inside the deployment. But it is LoadBalanced, since the traffic to an InputEndpoint comes via/through the LoadBalancer from the Internet.
Back to the Numbers. Let's say you have 1 WebRole with 1 Input EndPoint and 1 Internal EndPoint. That makes a total of 2 EndPoints for your deployment. Even if you spin up 50 instances, you sill have just 2 EndPoints that count toward the total EndPoints limit.
Can you obtain a specific EndPoint for Specific Instace - certainly yes! via the RoleEnvironemnt class. It has Roles enumeration. Each Role has Instances, and each Instance has InstanceEndpoints.
Hope this helps!
The endpoints are defined at the role level and instantiated for each instance.
An input endpoint has a public IP address making it accessible from the internet. Traffic to that input endpoint is load-balanced (with a round-robin algorithm) among all the instances of the role hosting the endpoint.
An internal endpoint has no public IP address and is only accessible from inside the cloud service OR a virtual network including that cloud service. Windows Azure does not load balance traffic to internal endpoints - each role instance endpoint must be individually addressed. Ryan Dunn has a nice post showing a simple example of implementing load balanced interaction with an internal endpoint hosting a WCF service.
The Spring Wave release introducted a preview of an instance input endpoint, which is a public IP endpoint that is port forwarded to specific role instance. This, obvously, is not load balanced but provides a way to directly connect to a specific instance.
Just trying to make things more concise and concrete:
// get a list of all instances of role "MyRole"
var instances = RoleEnvironment.Roles["MyRole"].Instances;
// pick an instance at random
var instance = instances[new Random().Next(instances.Count())];
// for that instance, get the IP address and port for the endpoint "MyEndpoint"
var endpoint = instance.InstanceEndpoints["MyEndpoint"].IPEndpoint;
Think of internal endpoints as a discovery mechanism for finding your other VMs.