Suppose I have a service, and elasticsearch running on multiple hosts. If I ensure that each host contains the complete elasticsearch index (by having replicas >= numberOfHosts-1), is it possible to make sure that a service call on a particular host only searches in the index of the same host (i.e. no elasticsearch calls are made across hosts)?
Using shard allocation filtering, you can ensure that shards of an index will only reside on a single host. Note, however, that by doing so, if you host goes down, you'll lose data.
PUT test/_settings
{
"index.routing.allocation.include._ip": "192.168.1.1",
"index.routing.allocation.exclude._ip": "192.168.1.2,192.168.1.3"
}
As a result, if you query the test index directly on 192.168.1.1 you can be sure that there won't be any chatter between 192.168.1.1 and the nodes to fulfill your request.
If you send your query to a client node or another kind of load balancer node, your request is guaranteed to be routed directly to 192.168.1.1 since the cluster will be aware that the test index is only available on that host.
It is not a good practice to localize an index on a single host as if that host fails or gets corrupted, you're guaranteed to lose data.
Related
I am trying to set-up a slave\standby as below:
Normally, the clients connect to the primary, a hostname (for example) updates.mysite.com which resolves to an Ingress LoadBalancer IP say A behind which is my actual (primary) application.
I have setup another slave application - which is basically a copy (in a different data center) , which the client would need to connect to, in case the primary site goes down. The clients would then connect to the LoadBalancer IP B behind which is the slave application.
Options I considered:
When the primary goes down, bring up the slave so that now updates.mysite.com points to the IP B. We set the TTL to be very low, say 1 minute. For a maximum of 1 minute, clients may try to connect to the primary after which it would be switched to the new IP (same hostname). The problem here is it leads to unnecessary calls to fetch the DNS record again - for something that happens not very frequently.
Setup updates2.mysite.com, let clients know that on disconnection or when updates.mysite.com is not available - they need to connect to updates2.mysite.com
Setup updates2.mysite.com which would point to the IP B. Expose an API that would give the hostname to connect to, clients should get the hostname from here - before trying to connect - and also on disconnections. We make sure that the API gives the correct hostname to connect to.
What would be the best approach and are there any other effective ways I can achieve this? I prefer the Option 1 due to the least amount of work and simple setup - but I might be missing something.
Take Google.com for example. If it ultimately resolves to a single IP at any point of time, the packets will land on a single server. Even if all it does is send a redirect response (for transferring load other servers), it still has to be capable of handling hundreds of thousands of requests per second.
I can think of a number of non standard ways to handle this. For example the router may be programmed to load balance the packet across multiple servers. But it still means that google.com is dependent on a single physical facility as IP addresses are not portable to another location.
I was hoping internet fabric itself has some mechanism to handle such things.Multiple A records per domain is one such mechanism. But while researching this I found that google.com's DNS entry has only one A record and the IP value is different depending on which site you query it from.
How is it done? In what ways is it better and why has Google chosen to do it this way instead of having multiple A records?
Trying to lookup A record of google.com yields different results from different sites:
https://www.misk.com/tools/#dns/google.com Resolves to 216.58.217.142
https://www.ultratools.com/tools/dnsLookupResult resolves to 172.217.9.206
This is generally done using dynamic DNS/round robin DNS/ DNS load balancing.
Say your have 3 web servers at 3 different locations. When the lookup is done the DNS server will respond with a different IP for each request. Some DNS servers also allow a policy based config... wherein it can return a certain IP 70% of time and some other IP 30% of the time.
This document provides reference on how to do this with Windows 2016.
I was unable to find existing answers on this topic.
I am running a redis client that connects to a remote redis server (not on the same host).
I can connect either via the domain name or via the server's IP, i.e. I can launch the client either via redis-cli -h 123.123.123.123 or redis-cli -h my.domain.com. It is more convenient to use the domain name.
Speed is important for my use case, therefore I would like to know whether the "costly" DNS lookup happens only once at startup or multiple times throughout the life of the client.
Thanks!
The overhead will be paid only when a connection is established.
If you make sure that your application is keeping permanent connections to the Redis instances instead of systematically connecting/disconnecting, I would say the overhead is negligible.
I'm using Memcached on each of my EC2 web server instances. I am not sure how to configure the various hostnames for the memcache nodes at the server level.
Consider the following example:
<?php
$mc = new Memcached()
$mc->addServer('node1', 11211);
$mc->addServer('node2', 11211);
$mc->addServer('node3', 11211);
How are node1, node2, node3 configured?
I've read about a few setups to configure the instance with hostname and update /etc/host with these entries. However, I'm not familiar enough with configuring such things.
I'm looking for a solution that scales - handles adding and removing instances - and automatic.
The difficulty with this is keeping an updated list of hosts within your application. When hosts could be added and removed, keeping this list up to date may be a challenge. You may be able to use some sort of proxy which would help by giving you a constant endpoint for your application.
If you can't use a proxy, I have a couple ideas.
If the list of hosts is static, assign an elastic ip to each memcached host. Within ec2 region, this will resolve to the local IP address of the host its associated with. With this idea, you have a constant list of hosts that your application can use.
If you are going to add/remote hosts on a regular basis, you need to be able dynamically update the lists of hosts your application will use. You can query the EC2 api for instances with a certain tag, then get the IP addresses for all of those instances. Cache the list in memory or on disk and load it with your application. If you run this every minute, any host changes should propagate within 1 minute, unless the EC2 api is being slow to update.
After Amazon's failure and reading many articles about what redundant/distributed means in practice, DNS seems to be the weak point. For example, if DNS is set to round-robin among data centers, and one of the data centers fails, it seems that many browsers will have cached that DNS and continue to hit a failed node.
I understand time-to-live (TTL), but of course this may be set to a long time.
So my question is, if a browser does not get a response from an IP, is it smart enough to refresh the DNS in the hope of being routed to another node?
Round-robin DNS is a per-browser thing. This is how mozilla does it:
A single host name may resolve to multiple ip addresses, each of which is stored in the
host entity returned after a successful lookup. Netlib preserves the order in which the dns
server returns the ip addresses. If at any point during a connection, the ip address
currently in use for a host name fails, netlib will use the next ip address stored in the
host entity. If that one fails, the next is queried, and so on. This progression through
available ip address is accomplished in the NET_FinishConnect() function. Before a url load
is considered complete because it's connection went foul, it's host entity is consulted to
determine whether or not another ip address should be tried for the given host. Once an ip
address fails, it's out, removed from the host entity in the cache. If all ip addresses in
the host entity fail, netlib propegates the "server not responding" error back up the call
chain.
As for Amazon's failure, there was NOTHING wrong with DNS during Amazon's downtime. The DNS servers correctly reported the IP addresses, and the browsers used those IP addresses. The screw-up was on Amazon's side. They re-routed traffic to an overwhelmed cluster. The DNS was dead-on, but the clusters themselves couldn't handle the huge load of traffic.
Amazon says it best themself:
EC2 provides two very important availability building blocks: Regions and Availability
Zones. By design, Regions are completely separate deployments of our infrastructure.
Regions are completely isolated from each other and provide the highest degree of
independence. Many users utilize multiple EC2 Regions to achieve extremely-high levels of
fault tolerance. However, if you want to move data between Regions, you need to do it via
your applications as we don’t replicate any data between Regions on our users’ behalf.
In other words, "remember all of that high-availability we told you we have? Yeah it's really still up to you." Due to their own bumbling, they took out both the primary AND secondary nodes in the cluster, and there was nothing left to fail over to. And then when they brought it all back, there was a sudden "re-mirroring storm" as the nodes tried to synchronize simultaneously, causing more denial of service. DNS had nothing to do with any of it.