Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I have a Linux web server farm with about 5 web servers, web traffic is about 20Mbps.
We currently have a Barracuda 340 Load Balancer (keep away from this device - piece of crap!) that is acting as a firewall. I want to put in a dedicated firewall and I'd like to know what peoples opinions are on building versus buying a dedicated firewall.
Main requirements:
Dynamically block rouge traffic
Dynamically rate limit traffic
Block all ports except 80, 443
Limit port 22 to a set of IPs
High availability setup
Also if we go for the build route, how do we know what level traffic the system can handle.
As they say - "there are more than one way to skin a cat":
Build it yourself, running something like Linux or *BSD. The benefit of this, is that it makes it easy to do the dynamic part of your question, it's just a matter of a few well-placed shell/python/perl/whatever scripts. The drawback is that your ceiling traffic rate might not be what it would be on a purpose-built firewall device, although you should still be able to achieve data rates in the 300Mbit/sec range. (You start hitting PCI bus limitations at this point) This may be high enough to where it won't be a problem for you.
Buy a dedicated "firewall device" - Possible drawbacks of doing this, is that doing the "dynamic" part of what you're trying to accomplish is somewhat more difficult - depending on the device, this could be easy (Net::Telnet/Net::SSH come to mind), or not. If you are worried about peak traffic rates, you'll have to carefully check the manufacturer's specifications - several of these devices are prone to the same traffic limitations as "regular" PC's, in that they still run into the PCI bus bandwidth issue, etc. At that point, you might as well roll your own.
I guess you could read this more as a "pro's and con's" of doing either, if you want.
FWIW, we run dual FreeBSD firewalls at my place of employment, and regularly push 40+Mbit/sec with no noticeable load/issues.
Definitely build. I help manage an ISP and we have two firewalls built. One is for fail over and for redundancy. We use a program called pfsense. I couldn't recommend this program more. It has a great web interface for configuring it and we actually run it off a compact flash card.
in my current startup, we have used PFSense to replace multiple routers/firewalls, and it has throughput which replaces much more expensive routers.
Maybe that is why Cisco is having trouble? :)
Related to high availability: OpenBSD can be configured in a failover / HA way for firewalls. See this description. I've heard that they've done demos where such setups done as well (if not better) as high-end Cisco gear.
Over the last 8 years we maintained a small development network with about 20 to 30 machines. We had one computer dedicated to be the firewall.
Actually, we never run into serious problems we are now replacing it with a dedicated router/firewall solution (though we haven't decided yet which). Reasons for that are: simplicity (the goal is the firewall, not to maintain the linux for running it as well), less space and less power consumption.
Don't know much about this field, but maybe an Astaro security gateway?
Hi I would go for a dedicated firewall product in this scenario. I have used the Checkpoint firewall range of products for many years and I have always found them to be easy to setup and manage and they have great support. Using Checkpoint or one of their competitors is a fairly expensive option, especially if you're comparing it to open source software, so it depends on your budget.
I've also used Cisco's line of PIX and ASA firewalls. These are also good, but in my opinion are more difficult to manage
Related
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
It's more a general question in terms of what one has to look out for when running an Erlang system. For example, I know of atom exhaustion attacks. What are other possible attacks and how to make your system more secure?
Running a cluster means they are sharing a cookie, and if one knows the cookie than they can attach to any of your nodes (assuming they are attached to your network) and execute any arbitrary Erlang command or program.
So my thought is that clustered means that there are at least two files (and some number of people) who know what the cookie is (or where to find it).
I would be afraid of bugs in applications deployed in your system. Good example from otp is SSL app, which was completely re-written 3 years ago. The next would be http client - memory leaks. Xmerl was never a strong part of the system.
Also, be careful with 3rd party Erlang apps: new web servers (probably better than inets, but if you do not need all the performance consider stable Yaws), ejabberd - number of techniques hitting directly OS, Riak - interaction with filesystem, ulimit, iostats etc.
First of all, you want to have your Cluster in a closed VPN (if they are far apart and parhaps communicate over a WAN). Then, you want to run them atop hardened UNIX or LINUX. Another strong idea is to close all epmd connections to your cluster even if one has got the cookie by using net_kernel:allow(Nodes). One of the main Weaknesses of Erlang Vms (i have come to realise) is memory consumption. I think that if an Erlang Platform providing service to many users and its NOT protected against DOS attacks, its left really vulnerable. You need to limit number of allowed concurrent connections for the Web Servers so that you can easilly block out some script boys in the neighbourhood. Another situation is having distributed/replicated Mnesia database across your cluster. Mnesia replicates data but i ain't sure if that data is encrypted. Lastly, ensure that you are the sole administrator of all the machines in your cluster.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
i have a asp.net mvc website and the volume of traffic is increasing. I have the site pointing to a backend sql server 2008 database.
at what point, do i need to figure out what the bottleneck of the system and look to review if i need to load balance machines, or change the way i am doing database connection management.
are there specific tools and thresholds that are indicators that the current model isn't scalable or is hitting a breaking point (besides just observations of a slow site.
When you start noticing performance issues.
There are some very easy things you can do to increase performance with so little work, it's easier to do them that see if you need to yet ;)
First and foremost is putting all static images and other media on a separate server. That eliminates a whole lot of queries on the boxen running the dynamic parts of the web server.
Next in line is make sure you are using as many hard drive spindles as possible. Of course you want your database on a separate machine, let alone a separate hard drive, but you also want your web server logs written to a separate hard drive. That prevents a lot of jumping around of the hard drive heads.
As far as "how do you know when you need to performance tune", I will give a different answer than George Stocker: When there is a cost associated with your performance that outstrips the cost of looking into it. I say it this way because your customers may be a little unhappy if your website is a little sluggish, but if it doesn't prevent anyone from using it, or recommending it to others, then it may not be worth looking into. People put up with sub-optimal performance all the time.
There are a plethora of tools available to address the plethora of possible bottlenecks. A decent performance tuning strategy starts with measurement and consistent instrumentation of the given system.
But performance tuning requires precious time and resources, and should only be pursued when it gives you the most bang for the buck, i.e. it provides the greatest improvement to achieving your website's objectives given the work required. If your website supports (or is) a business or organization, you must continuously evaluate the business landscape and plan the next allocation of resources. This is entirely dependent on the particular industry.
An engineer might focus on continual refinement of an existing system, but the project commissioners (be they an external client, or your company's management) must weigh the costs and benefits of all types of development, from improving an existing featureset, to adding new features, to addressing technical limitations affecting product usability (including performance issues). That's not to say engineers have no say in resource allocation, but their perspective is just one of many contributing to success.
When you have doubts that the website would survive a doubling in max usage. One common line of thought where I am from is that you should have the performance capacity to support at least 2x the number of users you expect.
Determining whether or not you can support 2x is something better left to load testing though, rather then speculation. One comment from your other comment though: chances are a website performance problem is going to affect everyone using the web site, including you on a local machine... unless it's a bandwidth problem and you're connected to a local network. Barring cable cuttings, it's not going to be 'just the people in Asia'.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
I'm going to build a high-performance web service. It should use a database (or any other storage system), some processing language (either scripting or not), and a web-server daemon. The system should be distributed to a large amount of servers so the service runs fast and reliable.
It should replicate data to achieve reliability and at the same time it must provide distributed computing features in order to process large amounts of data (primarily, queries on large databases that won't survive being executed on a single server with a suitable level of responsiveness). Caching techniques are out of the subject.
Which cluster/cloud solutions I should take for the consideration?
There are plenty of Single-System-Image (SSI), clustering file systems (can be a part of the design), projects like Hadoop, BigTable clones, and many others. Each has its pros and cons, and "about" page always says the solution is great :) If you've tried to deploy something that addresses the subject - share your experience!
UPD: It's not a file hosting and not a game, but something rather interactive. You can take StackOverflow as an example of a web-service: small pieces of data, semi-static content, intensive database operations.
Cross-Post on ServerFault
You really need a better definition of "big". Is "Big" an aspiration, or do you have hard numbers which your marketing department* reckon they'll have on board?
If you can do it using simple components, do so. The likes of Cassandra and Hadoop are neither easy to setup (especially the later) or develop for; developers who are going to be able to develop such an application effectively will be very expensive and difficult to hire.
So I'd say, start off using your favourite "Traditional" database, with an appropriate high-availability solution, then wait until you get close to the limit (You can always measure where the limit is on your real application, once it's built and you have a performance test system).
Remember that Stack Overflow uses pretty conventional components, simply well tuned with a small amount of commodity hardware. This is fine for its scale, but would never work for (e.g. Facebook), but the developers knew that the audience of SO was never going to reach Facebook levels.
EDIT:
When "traditional" techniques start failing, e.g. you reach the limit of what can be done on a single database instance, then you can consider sharding or doing functional partitioning into more instances (again with your choice of HA system).
The only time you're going to need one of these (e.g. Cassandra) "nosql" systems is if you have a homogeneous data store with very high write requirement and availability requirement; even then you could probably still solve it by sharding conventional systems - as others (even Facebook) have done at times.
It's hard to make specific recommendations since you've been a bit vague, but I would recommend Google Appengine for basically any web service. It's reliable, easy to use, and is built on the google architecture so is fast and reliable.
i'd like to recommend stratoscal symphony. it's a private cloud service that does it all. everything you just mentiond - this service provides perfectly. their symphony products deliver the public cloud experience in you enterprise data center. if that's what you're looking for, i suggest you give it a shot
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have a lot of spare intel linux servers laying around (hundreds) and want to use them for a distributed file system in a web hosting and file sharing environment. This isn't for a HPC application, so high performance isn't critical. The main requirement is high availability, if one server goes offline, the data stored on it's hard drives is still available from other nodes. It must run over TCP/IP and provide standard POSIX file permissions.
I've looked at the following:
Lustre (http://wiki.lustre.org/index.php?title=Main_Page): Comes really close, but it doesn't provide redundancy for data on a node. You must make the data HA using RAID or DRBD. Supported by Sun and Open Source, so it should be around for a while
gfarm (http://datafarm.apgrid.org/): Looks like it provides the redundancy but at the cost of complexity and maintainability. Not as well supported as Lustre.
Does anyone have any experience with these or any other systems that might work?
check also GlusterFS
Edit (Aug-2012): Ceph is finally getting ready. Recently the authors formed Inktank, an independent company to sell commercial support for it. According to some presentaions, the mountable POSIX-compliant filesystem is the uppermost layer and not really tested yet, but the lower layers are being used in production for some time now.
The interesting part is the RADOS layer, which presents an object-based storage with both a 'native' access via the librados library (available for several languages) and an Amazon S3-compatible RESP API. Either one makes it more than adequate for adding massive storage to a web service.
This video is a good description of the philosophy, architecture, capabilities and current status.
In my opinion, the best file system for Linux is MooseFS , it's quite new, but I had an opportunity to compare it with Ceph and Lustre and I say for sure that MooseFS is the best one.
Gluster is getting quite a lot of press at the moment:
http://www.gluster.org/
Lustre has been working for us. It's not perfect but it's the only thing we have tried that has not broken down over load. We still get LBUGS from time to time and dealing with 100TB + file systems is never easy but the Lustre system has worked and increased both performance and availability.
If not someone forces you to use it, I would also highly recommend using anything else than Lustre. From what I hear from others and what also gave myself nightmares for quite some time is the fact that Lustre quite easily breaks down in all kinds of situations. And if only a single client in the system breaks down, it puts itself into an endless do_nothing_loop mode typically while holding some important global lock - so the next time another client tries to access the same information, it will also hang. Thus, you often end up rebooting the whole cluster, which I guess is something you would try to avoid normally ;)
Modern parallel file systems like FhGFS (http://www.fhgfs.com) are way more robust here and also allow you to do nice things like running server and client components on the same machines (though built-in HA features are still under development, as someone from their team told me, but their implementation is going to be pretty awesome from what I've heard).
Ceph looks to be a promising new-ish entry into the arena. The site claims it's not ready for production use yet though.
I read a lot about distributed filesystems and I think FhGFS is the best.
http://www.fhgfs.com/
It worth a try. See more about it at:
http://www.fhgfs.com/wiki/
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
I have a server behind a firewall. It runs a web application (Java servlets under Apache Tomcat) and responds only to port 443 (HTTPS). There is no scripting code in the pages served - the forms use HTTP POST to receive the form, process the data (with appropriate input filtering) and then output an HTTP result page.
I am currently using an appliance firewall, but it is 'hardware-flakey'. I have been looking at upgrading to a more 'industrial strength' solution, but the vendor is quite insistant that I purchase a subscription to their "deep packet inspection" software. He claims that even web servers need this kind of protection.
I am not convinced, but do not have the security background to be certain. The firewall would sit between the "world" and my server, and use "port forwarding" to allow only ports 443 and 22 (for maintenance) to reach the server.
So - do I really need this deep packet inspection, or not?
Given that the only protocols that you're interested in (ssh and https) are "negotiate encryption on connect" there's little that a standard firewall will be able to inspect after that point. Once the SSL/SSH session is established the firewall will only see encrypted packets. Ask your vendor what their product examines in this context.
Alternatively, it is possible that the device acts more like a proxy -- that it acts as the server-side end-point for the connection before relaying on to your real server -- in which case it is possible that the product does something deeper, although this isn't the case if the firewall really is "port forwarding" as you say. Again, your vendor should be able to explain how their device operates.
Also you may want to ask what vulnerabilities/risks the inspection system is intended to protect against. For example: Does it look out for SQL injection? Is it targetted to a particular platform? (If your web server runs on a SPARC CPU, for example, then there's little point inspecting URLs for x86 shellcode).
As a network security professional, this sounds like overkill to me.
Martin Carpenter's answer is 100% on target. Anytime you're considering security, you need to understand
What you're securing,
What you're securing it against,
The likelihood of an attack, and
Your risk if an attack succeeds.
For your application, which allows only encrypted, authenticated communication on only 2 ports, I can see only a few vulnerabilities:
Denial-of-service (DOS) is always a threat unless your firewall blocks those attacks.
You might have other applications listening on other ports, but you can detect them with any simple port scanning program.
You may want to restrict outbound communication to prevent a user or rogue application from initiating communication to an unauthorized server.
I also agree that it's a good idea to ask the vendor what "deep packet inspection" means to him and why your particular situation requires it. Unless you get a specific, knowledgeable answer, in layman's terms, that makes sense to you, I'd go elsewhere. There's nothing about network security that can't be explained simply, without buzzwords.
Update on several fronts...
First - I now have reason to believe that part of the flakiness of the OTS hardware product is a combination of low-powered CPU and insufficient buffer memory. In weeks of logging and a few crashes, there are no entries in the logs before the crash, yet I'm logging everything according to the log control. Talking with another firewall vendor, it was indicated that may suggest the buffer is filling faster than it can empty during heavy use. This corresponds with findings - the most used IP is the one crashing the most often.
So I checked, and the firewall did have some deep packet inspection stuff turned on. I've turned it off to see if things improve.
The firewall's main purpose in my network scenario is "gate keeper". That is, I want the firewall to prevent all traffic EXCEPT http, https and some ssh from ever getting beyond the WAN port. Since there are no users inside the firewall, any traffic generated from the inside comes from my application and can be allowed out.
Further talks with the one vendor indicated that they no longer thing deep packet inspection is necessary - the other fellow was just trying to "upsell" me on the unit in question. I also found out their hardware won't really do all that I want without spending a ton of money.
I'm now seriously exploring the use of OpenBSD and a PF firewall to do what I reauire in a cost-effective manner.