I'm fairly new to Proxy servers and how they work exactly.
I recently span up an AWS EC2 instance to act as a proxy server using tiny proxy. Everything seems to work just fine however i am curious about something. Is it possible to configure tiny proxy to use a different public IP each time it makes a request ? I looked into AWS Elastic IP's but don't quite understand how those might fit in this scenario.
Not possible. Public IPs are allocated to the instance during launch. You can allocate multiple IPs using Elastic IPs like you mentioned but you can't get IPs per request like you asked. What's your use case?
Related
I have successfully set haproxy on my server cluster. I have run into one snag that I can't find a solution for...
TESTING INDIVIDUAL CLUSTER COMPUTERS
It can happen that for one reason or another, one computer in the cluster gets a configuration variation. I can't find a way to tell haproxy that I want to use a specific computer out of a cluster.
Basically, mysite.com (and several other domains) are served up by boxes web1, web2 and web3. And they round-robin perfectly.
I want to add something to the URL to tell haproxy that I specifically want to talk to web2 only because in a specific case, only that server is throwing an error on one web page.
Anyone know how to do that without building a new cluster with a URI filter and only have one computer in that cluster? I am hoping to use the cluster as-is but add something to the URI that will tell haproxy which server to use out of the cluster.
Thanks!
Have you thought about using different port for this? Defining new listen section with different port, because, as I understand, you can modify your URL by any means?
Basically, haproxy cannot do what I was hoping. There is no way to add a param to the URL to suggest which host in the cluster to use.
I solved my testing issue by setting up unique ports for each server in the cluster at the firewall. This could also be done at the haproxy level.
To secure this path from the outside world, I told the firewall to only accept traffic from inside our own network.
This lets us test specific servers within the cluster. We did have to add a trap in our PHP app to deal with a session cookie that is too large because we have haproxy manipulating this cookie to keep users on the server they first hit. So when the invalid session cookie is detected, we have the page simply drop the session and reload the page.
This is working well for our testing purposes.
I have a local service fabric cluster which has 6-7 custom http endpoints exposed. I use fiddler to redirect these to my service like so:
127.0.0.1:44300 identity.mycompany.com
127.0.0.1:44310 docs.mycompany.com
127.0.0.1:44320 comms.mycompany.com
etc..
I've never deployed a cluster in azure before, so there's some intricacies that i'm not familiar with and I can't find any documentation on. I've tried a multiple times to deploy and tinker with the load balancers/public ips with no luck.
I know DNS CNAMES can't specify ports, so I guess that I have to have separate public IP for each hostname I want to use and then somehow internally map that to the port. So i end up with something like this:
identity.mycompany.com => azure public ip => internal redirect / map => myservicefabrichostname.azure.whatever:44300
my questions are:
1) is this the right way to go about it? or is there some fundamental method that i'm missing
2) do I have to specify all these endpoints (44300, 44310, 44320...) when creating the cluster (it appears to set up a load of load balancer rules/probes) or will this be unnecessary if I have multiple public IPs), i'm unsure if this is for internal or external access.
thanks
EDIT:
looks like the azure portal is broken :( been on phone with microsoft support and it looks like it's not displaying the backendpools in the load balancer correctly, so you can't set up any new nat rules.
Might be able to write a powershell script to get round this though
EDIT 2:
looks like Microsoft have fixed the bug in the portal, happy times
Instead of using multiple ip addresses you can use a reverse proxy. Like HAProxy, IIS (with rewriting), the built-in reverse proxy, or something you build yourself or reuse. The upside of that is that is allows for flexibility in adding and removing underlying services.
All traffic would come in on one endpoint, and then routed in the right direction (your services running on various ports inside the cluster). Do make sure your reverse proxy is high available.
I have 2 server AWS EC2 (EC2-A, EC2-B).
Now, I have 1 domain http://my-domain.com call to website in EC2-A (/var/www/html/website1).
I want setup domain category http://my-domain.com/category call to website in EC2-B (/var/www/html/website2).
Can I do it?
I am not a master on this subject, but I would most likely use one of the following approaches:
If possible, use category.my-domain.com and then make the DNS server point to the correct EC2 instance; or
Use a NAT instance that redirects the traffic to the correct instance based on the url pattern. That is, all requests would be made to the NAT instance, which in turn would forward it to the correct instance. To achieve that I would probably choose haproxy and do something like explained here.
I have a couple EC2 instances behind an Elastic Load Balancer. These instances serve HTTP requests for a single web site. I recently started looking at the HOST header of the traffic, because I am planning to split my app into virtual hosts.
With some regularity (dozens of times a day), I log a request for a host name that is totally unrelated to my servers. As a couple examples, today I saw requests with the host names ad.adserverplus.com and r1---sn-upfn-hp5e.c.youtube.com. I looked these up and the IP addresses are not the same as any of my servers, nor of the ELB, so I am trying to develop a theory as to how this happens.
I realize that someone could be spoofing the host header, but it happens often enough that I am pretty sure this is not what is going on. My other idea is that somehow there is stale DNS data that just happens to resolve one of those hosts to my IP address, but again this seems like it could happen once in a great while but not regularly. What are some other possibilities, and how might I verify / discredit them?
EDIT
I looked at some of the unexpected host names today, and it seems that they actually do resolve to an IP that is one of the possible IPs that my domain apex resolves to. I use Route 53 for DNS, and I have the zone apex pointed to the ELB, so when I query the IP address for my domain, I get different answers depending on when I ask. So this makes me very curious, how do these IP addresses get assigned to me and how does EC2 make sure they are not co-opting an IP address that someone else is already using.
There are any number of reasons for this. First you should understand that the public host name for your EC2 instances and load balancers have likely been used before. If you have an elastic IP associated with your load balancer, it has also probably been used before.
As such you can get traffic to your servers that is intended for a previous tenant of that hostname of IP address that you are currently using.
One thing you can do is to configure your web servers to reject traffic (respond with 403) to traffic that is not arriving with the proper hostname specified or that comes from a specific external host.
Your IP or your ELBs IP may have at one point in time been an open proxy. meaning that someone is hoping that you would forward the requests on to their intended destination.
but in general open port 80 to the internet and all kinds of bots and zombies will visit you with a pretty constant flow of dodgy requests. I would imagine though that the \ec2 IP ranges would be a particularly juicy range to search for poorly patched websites to exploit.
I'm using Memcached on each of my EC2 web server instances. I am not sure how to configure the various hostnames for the memcache nodes at the server level.
Consider the following example:
<?php
$mc = new Memcached()
$mc->addServer('node1', 11211);
$mc->addServer('node2', 11211);
$mc->addServer('node3', 11211);
How are node1, node2, node3 configured?
I've read about a few setups to configure the instance with hostname and update /etc/host with these entries. However, I'm not familiar enough with configuring such things.
I'm looking for a solution that scales - handles adding and removing instances - and automatic.
The difficulty with this is keeping an updated list of hosts within your application. When hosts could be added and removed, keeping this list up to date may be a challenge. You may be able to use some sort of proxy which would help by giving you a constant endpoint for your application.
If you can't use a proxy, I have a couple ideas.
If the list of hosts is static, assign an elastic ip to each memcached host. Within ec2 region, this will resolve to the local IP address of the host its associated with. With this idea, you have a constant list of hosts that your application can use.
If you are going to add/remote hosts on a regular basis, you need to be able dynamically update the lists of hosts your application will use. You can query the EC2 api for instances with a certain tag, then get the IP addresses for all of those instances. Cache the list in memory or on disk and load it with your application. If you run this every minute, any host changes should propagate within 1 minute, unless the EC2 api is being slow to update.