Currently my Lambda is served by an API Gateway (HTTPApi) behind a CloudFront distribution. The API's only stage is $default. I have a custom domain example.com which directs traffic to the CloudFront distro and to the API. I am using CloudFormation for my whole infrastructure.
I want to improve my staging by introducing dev and prod stages. I want example.com to point to the prod stage, and dev.example.com to point to the dev stage.
In my api_stack I have created the two stages:
self.http_api.add_stage(
"dev",
stage_name="dev",
auto_deploy=True,
)
self.http_api.add_stage(
"prod",
stage_name="prod",
auto_deploy=False,
)
I only have one CloudFront distribution and it points to the $default stage (the invoke URL of which is stored in origin_url in the following code). So I extend my cloudfront_stack to point to the prod stage (having manually deployed at least once).
self.distribution = cloudfront.Distribution(
self,
construct_id,
default_behavior=cloudfront.BehaviorOptions(
origin=cloudfront_origins.HttpOrigin(
origin_url,
origin_path="/prod",
),
...
certificate=certificate,
domain_names=domain_names,
certificate is a DnsValidatedCertificate for example.com. domain_names=["example.com"].
At this point, my API (website, Flask app) breaks. static/ filepaths fail, as per this issue.
Ignoring that issue... I don't see how I can extend my CloudFront distribution to serve <invokeURL>/dev at dev.example.com. I expect I need:
A new certificate for dev.example.com
???
It almost feels like I need a second CloudFront distribution, which feels extraneous. I would like to solve this problem while keeping the invoke URLs disabled (as per HttpApi: disable_execute_api_endpoint).
Have I got it the wrong way round; should I have multiple CloudFront distributions, each with their own singular API deployment (Stage)? Does Stage describe something far more encompassing than a Stack? I would love for cloudfront.BehaviorOptions param origin to be origins and for some mapping to be available elsewhere: dev.example.com/: example.com/dev/. Please educate me! :)
<small Might imply I make 2 distros />
Related
Background:
I have a Javascript hosted on S3 bucket with static website + Cloudfront distribution + Route 53 setup. I need Cloudfront because I need custom domain with SSL support. This works fine. As an example, the script can be accessed at https://app.example.org/myscript.js. This URL is given to clients to embeded on their webpages and I can not change it.
The setup in Route 53 is like this:
app.example.org => Cloudfont Distribution (s3://app.example.org)
What I want:
I want to set up a staging environment for new features. I would like to direct 10% of production request to another version of the script.
What I tried
I've tried setting up another S3 bucket with static webiste + cloudfront with different alternative domain name (e.g. app-beta.example.org).
I need to use a different alternative subdomain because Cloudfront does not allow same alternative domain name with multiple distributions.
In Route 53, I set up alias A records like this:
app-beta.example.org => Alias Cloudfont Distribution (s3://app-beta.example.org)
app.example.org => Alias app-beta.example.org (weighted 10)
app.example.org => Alias Cloudfont Distribution (s3://app.example.org) (weighted 100)
Why didn't it work
It turned out that this won't work:
Because the requested domains is the same app.example.org, cloudfront distribution (s3://app.example.org) will pickup the request regardless the record of app.example.org => Alias app-beta.example.org.
I think here Route 53 do try routing the traffics to cloudfront according to the weight rules but eventually when requests reach cloudfront, cloudfront respects the 'alternative domain name' and use the one with (app.example.org) regardless.
How about not using cloudfront?
I've thought of using EC2 or EBS for an nginx server serving the static javascript files. This could work. But:
the javascript file will be requested worldwide. Using EC2 or EBS means for some users it will have higher latency. This is not ideal.
this means extra resources to manage and cost.
How about lambda#Edge?
Technically lambda#Edge is a perfect solution if it is cheaper. Yes, it is already cheap ($0.0000006 per request) but in our case, we will be paying more than USD 10k for it. ( the script is requested > 10 billion times per month, and 2 lambda functions are required for the set up)
Question
Is there any other way I can achieve what I want ? (i.e. gradually roll out new version without changing the URL)
Can somebody explain to me the architecture of this website (link to a picture) ? I am struggling to understand the different elements in the front-end section as well as the fields on top, which seem to be related to AWS S3 and CDNs. The backend-section seems clear enough, although I don't understand the memcache. I also don't get why in the front end section an nginx proxy is needed or why it is there.
I am an absolute beginner, so it would be really helpful if somebody could just once talk me through how these things are connected.
Source
Memcache is probably used to cache the results of frequent database queries. It can also be used as a session database so that authenticated users' session work consistently across multiple servers, eliminating a need for server affinity (memcache is one of several ways of doing this).
The CDN on the left caches images in its edge locations as they are fetched from S3, which is where they are pushed by the WordPress part of the application. The CDN isn't strictly necessary but improves down performance by caching frequently-requested objects closer to where the viewers are, and lowers transport costs somewhat.
The nginx proxy is an HTTP router that selectively routes certain path patterns to one group of servers and other paths to other groups of servers -- it appears that part of the site is powered by WordPress, and part of it node.js, and part of it is static react code that the browsers need to fetch, and this is one way of separating the paths behind a single hostname and routing them to different server clusters. Other ways to do this (in AWS) are Application Load Balancer and CloudFront, either of which can route to a specific server based on the request path, e.g. /assets/* or /css/*.
I want to setup fake URLs or honeytoken to trick an attacker to follow those links, and have a script to auto block the attacker IPs using AWS WAF.
Security is a big thing these days, and our web infrastructure has already been a target of massive bruteforce and DDOS attempts. I want to setup tracks so attacker who are using directory traversing attacks can be found. e.g A common directory listing attacks traverse URLs like ../admin, ../wp-admin etc while scanning a target website. I want to setup a mechanism to get alerted when any of these non-existent URLs get browsed.
Question:
1. Is it possible to redirect part of web-traffic e.g www.abc.com/admin to API gateway and remaining www.abc.com to my existent servers?
2. How will I setup DNS entries for such, if it is possible?
3. Is there a different/easy to achieve this.
Any suggestion is welcome, as I am open to Ideas. Thanks
You can first setup a Cloudfront distribution with WAF ip blacklisting. Then setup a honey pot using API Gateway and Lambda which becomes a origin for urls like /admin, /wp-admin.
Following example with bad bot blocking using honeypots can provide you with a head start.
I'm moving from Apache on Linux to Azure Web Apps and I have a specific url (mysite.com/blog and everything under it) that is configured with a reverse proxy so the end user doesn't know that the content is actually coming from another service.
I'm sure I can do this within Web Apps (which runs on IIS) but I can't find any documentation on how to do this. As a backup I'm open to putting another service in front of my Web App.
How can I accomplish this in Azure?
Update: I did try using another service - Functions. My architecture looks like this:
This works in production but I'm hitting snags in development. /blog may or may not work depending on the entry point. In prod, our DNS will be configured so mysite.com points to mysite-proxy.azurewebsites.net and, therefore, any URI the user hits will work. In dev, however, we may want to browse to hit /blog from the Traffic Manager which will route us to /blog on the webapp which doesn't exist. Same problem, of course, if we go to /blog directly on the webapp. I tried to show these examples on the right side of the diagram.
I would like to find a solution so the webapp itself can handle the /blog proxying and then we can determine whether it's worth the speed and cost tradeoff compared to the existing solution.
You might want to checkout Azure Functions Proxies: https://learn.microsoft.com/en-us/azure/azure-functions/functions-proxies
Sounds like you want an Application Gateway (caution, costs like $15/day)
The AGW can have multiple listeners against multiple hostnames, including path-based routing.
You will want to add two backends, one for the /blog endpoint and one for the non-/blog stuff. The backends just take the IP or FQDN of the target resource, in this case you will have:
blogBackend: myblog.com
defaultBackend: myWebapp.azurewebsites.net
Then you need to add listeners for your public-facing domain, it would be like:
myHttpListener: port 80, hostname=mywebsite.net
myHttpsListener: port 443, hostname=mywebsite.net
Then you need an HTTP setting
myHttpSetting: protocol=HTTPS, port=443, useWellKnownCACert=Yes, HostnameOverride=Yes,pick from backend target
Then you need rules, one for http=>https redirect, and the other for handling the pathing
myRedirectRule: type=basic, listener=myHttpListener, backendtargettype=redirection, targettype=listener, target=myHttpsListener
myRoutingRule: type=path-based, listener=myHttpsListener, targettype=backendpool, target=defaultBackend, httpSetting=myHttpSetting, pathRules=
path=/* name=root backendpool=defaultBackend
path=/blog name=blog backendpool=blogBackend
You can create additional http settings and assign them to the path rules to change the behaviour of the reverse proxy. For example, you can have the user's URL be https://mywebsite.net/blog, but have the path stripped on the request to the blog so the request looks like myblog.com instead of myblog.com/blog
There's a lot of tiny parts, but the app gateways can handle multiple applications at scale. Biggest thing is to watch out for the cost since this is more of an enterprise solution.
I'm migrating over a test site to AWS from another company. They've been nothing but unhelpful in giving up the necessary credentials and info to make this a seamless transition.
My test site, now, has everything it needs to be a perfect test site. Looks exactly like the current up and running site, has all the databases, necessary bells and whistles. The only issue is that my AWS public DNS is redirecting to the live server.
I've tried removing all .htaccess files from the EC2 instance and the S3 buckets. I've tried searching for any and all files that would cause this redirect. The live server has nothing on it that would cause this as well.
The IT department of the client only knew that there was some code injection in some file to help redirect every URL the client owns to the same site. I'm at my wits end with non-cooperative dev shops and don't want to spend more time digging through endless files for some few lines of code.
Am I forgetting / missing / overlooking something here? Before I go crazy.
What do you mean unhelpful giving credentials and information? AWS is a IaaS company, you are responsable for the setup and configuration of your servers. They do offer a paid support plan if you will like to purchase it but it's pretty straightforward to get your access keys when you create an EC2 or RDS instance.
Why don't you fix you problem at the dns level? Simply create a subdomain where you will host the temporary test site on the testing server, see if everything works and then change the dns configuration to the live server.