How can we analyze the software reliability? How to check the reliabilty of any application or product?
First try to define "software reliability" and the way to quantify it.
If you accomplish this task, you will probably be able to "check" this characteristic.
The most effective way to check reliability is going to be to run your software and gather statistics on its actual reliability. There are too many variables in play, both at the hardware and software levels, to realistically analyze reliability prior to execution, with the possible exception of groups with massive resources like NASA.
There are various methods for determining whether a piece of software meets a specification, but most of the really productive ones do this by construction, i.e., by constraining the way in which the software is written so that it can be easily shown to be correct. Check out VDM, Z and the B toolkit for schemes for doing this sort of thing. Note that these tend to be expensive ways to program if you're not in a safety-critical systems environment.
Proving the correctness of the specification itself is really non-trivial!
Reliability is about continuity of correct service.
The best approach to assess reliability of a software is by dynamic analysis, in other words: testing.
In order to reduce your testing time you may want to apply input profiles different from operational one.
Apply various input distributions, measure how much time your software runs without failure. Then find out how far your input distributions are from operational profile and draw your conclusion about how much time the software would have run with operational profile.
This involves modeling techniques such as Markov chains or stochastic Petri nets.
For further digging, useful keywords are: fault forecasting and statistical testing.
Related
Recently while doing an analysis on a practice project I noticed some features were too highly correlated after I engineered them. To my understanding, if features are too highly correlated, they might hinder a model’s performance (since they look more like dependent than independent variables).
Since learning about feature engineering, I’ve had the intuition somehow there’s a way feature engineering can cause collinearity between the created variables and the existing ones if certain principles aren’t considered. But I haven’t been able to identify under which circumstances.
The background or question itself might be wrong, but: is there a way feature engineering could cause collinearity? If so, how can this happen and how to avoid it?
I am working on an application which supports many-core MIMD architectures (on consumer/desk-computers). I am currently worrying about the scaling behaviour of the application. Its designed to be massively parallel and addressing next-gen hardware. That's actually my problem. Does anyone know any software to simulate/emulate many-core MIMD Processors with >16 cores on a machine-code level? I've already implemented a software based thread sheduler with the ability to simulate multiple processors, by simple timing techniques.
I was curious if there's any software which could do this kind of simulation on a lower level preferably on an assembly language level to get better results. I want to emphasize once again that I'm only interested in MIMD Architectures. I know about OpenCL/CUDA/GPGPU but thats not what I'm looking for.
Any help is appreciated and thanks in advance for any answers.
You will rarely find all-purpose testing tools that are ALSO able to target very narrow (high-performance) corners - for a simple reason: the overhead of the "general-purpose" object will defeat that goal in the first place.
This is especially true with paralelism where locality and scheduling have a huge impact.
All this to say that I am affraid that you will have to write your own testing tool to target your exact usage pattern.
That's the price to pay for relevance.
If you are writing your application in C/C++, I might be able to help you out.
My company has created a tool that will take any C or C++ code and simulate its run-time behavior at bytecode level to find parallelization opportunities. The tool highlights parallelization opportunities and shows how parallel tasks behave.
The only mismatch is that our tool will also provide refactoring recipes to actually get to parallelization, whereas it sounds like you already have that.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
Developers can learn a lot from other industries. As a thought exercise, is it possible to build a passenger aircraft using agile techniques?
Forgetting cost for now; how feasible is it to use iterative and incremental development for both the hardware (fuselage, wings, etc) as well as software, and still come out with a working and safe product which meets the customer’s requirements at the time of delivery?
Does it make sense to refactor a plane?
Agile in software and Agile in manufacturing are really quite different, although they share similar principals and values.
Agile in manufacturing emerged in Japan in the 1950s. Read up on W.E. Deming and the Toyota Production System to find out more. It's all about constantly improving the process whereby a product is reproduced.
Agile in software evolved in the early 1990s as a rapid development model. It's all about constantly improving the product.
You can certainly build a plane using Agile manufacturing methods, I've no doubt that some already are. Anything built in Japan definitely will be as Agile manufacturing is very well established there (it's taught in primary schools).
You couldn't build a plane using Agile software methods because you can't afford to rapidly change the product - in software changes and mistakes are cheap and reproduction is free. This isn't the case for aviation.
You could design a prototype plane using something like Agile software methods - but it would have to be standardised in order to be reproduced (a design task in itself).
How would you work using Test Driven Development? Would you automatically build and test a plane every iteration? Would you be able to make a ten minutes build? How easy is to make changes to the airplane? Even if you are really flexible in your desing the building some components need to be sent to special factories so there is not inmmediate feedback.
From de design using CAD software you need to make a mould, take the piece of fiber, put it in the plane. Etc. So here a trivial change has a non trivial cost. In Agile you can make a very little change and have it tested, built and an ready to ship in 20 minutes. If small changes are expensive then the short development cycle and refactoring won't be so usefull. Your feedback can take longer than a week so there is a strong reason for thinking in advance like in the waterfall model. And every attempt has a cost in physical materials unless you are programming. The Idea is not new. Carpenters measure twice. Programmers just first code and then test.
In summary. There may be some similarities but it will definitively be the same.
I'm going to say "kind of". In fact there's one example right now that I think is pretty close to answering this question.
Boeing is attempting to do this now with the new 787 - see following: Boeing 787 - Specification vs. Collaboration (From the 777 to the 787, the initial specifications document supposedly went from 2500 pages to 20 pages with the change in technique.) Suppliers from around the world are working independently to develop the components for this aircraft. (We'll call this the "teams".)
So, I want to say yes, but at the same time, iterations in creating the aircraft has resulted in delays of 2+ years and has resulted in stories like this one - (787 Delayed for 5th Time)
Will the airplane ever get built? Yes, of course it will. But when you look at the rubber hitting the road here, it seems like "integration test" is having one heck of a time.
Edit: At the same time, this shift in technique has resulted in building a new breed of aircraft built out of entirely new materials that will arguably be one of the most advanced in the world. This may be a direct result of the more Agile approach. Maybe that's actually the question - not a "can you?" but a "if Agile delays complex systems, does it provide a more innovative product in the payoff?"
Toyota pioneered Lean Production which Agile methodoligies followed on from. For the building of the hardware of the aircraft lean production would be the way to go and for the software an agile methodology would be the way to go.
Pick the right tools for the job.
A great book following how TPS was created and works
http://www.amazon.com/Machine-That-Changed-World-Production/dp/0060974176
http://en.wikipedia.org/wiki/Toyota_Production_System
I think in this case you are thinking too big. Agile is about breaking things down into more managable pieces and then working against that. The whole idea of Agile (XP in particular) is that you do your testing first so that you cut the number of bugs out and because plane software needs to have a very high code coverage for its testing it fits in quite neatly I think.
You aren't going to 'refactor' the mechanics of the plane but you will tweak them if they are unsafe and thats the whole iterative approach for you.
I have heard of Air Traffic Control software written with Agile Methodologies pushing it forward.
This is taken from http://requirements.seilevel.com/blog/2008/06/incose-2008-can-you-build-airplane-with.html
***Actually, that’s not true,***
My first guess - this probably relates to some of the core differences between systems and software engineering. I am going to over simplify this and just say that it is about scale. Systems projects are just a superset of software and hardware projects, integrating and deploying some combination of these. The teams of people deploying systems projects are quite large. And many of the projects discussed here are for government or regulated systems where specification and traceability is necessary. I could see how subsets of systems projects could in fact be developed using agile (pure software components), but I’m not convinced that an entire end-to-end project can. To put this in context, imagine you are building an airplane - a very commonly referenced type of systems engineering projects. Can you see this working using agile?
All skeptism aside, I do think that iterative development most certainly could work well on systems projects, and most people here would not argue that. In fact, I would love it if we could find examples of agile working on systems projects, because the number one feeling I get at systems engineering conferences is a craving for lighter processes.
I decided to do a little research outside the conference walls, and low-and-behold, I found a great article on this exact topic – “Toward Agile Systems Engineering Processes” by Dr. Richard Turner of the Systems and Software Consortium. The article is very well laid out, and I highly recommend reading it. He defines what agile is and what he believes the issue why most systems engineering projects are not agile. For example, he suggests that executives and program managers tend to believe that the teams involved have perfect knowledge about systems we are building, so we can plan them out in advance and work to a perfect execution against a perfect schedule.
Agile Can Work With Complex Systems
He talks to how to the agile concepts can work in systems projects. Here are a few examples summarized from his article:
Learning based: The traditional V-model implies a one-time trip through, implying one time to re-learn. However, perhaps the model can be re-interpreted to allow multiple iterations through it to fulfill this.
Customer focus: Typically systems engineering processes do not support multiple interactions with the customer throughout the project (just up front to gather requirements). That said, he references a study indicating the known issues with that on systems projects. Therefore, perhaps processes should be adapted to allow for this, particularly allowing for them to help prioritize requirements throughout projects.
Short iterations: Iterations are largely unheard of because the V-model is a one-time pass through the development lifecycle. That said, iterations of prototyping through testing could be done in systems engineering in many cases. The issue is in delivering something complete at the end of each iteration. He suggests that this is not as important to the customer in large deployments as is reducing risk, validating requirements, etc. This is a great point to rememember the airplane example! Could we have even a working part of an airplane after 2 weeks? What about even the software to run a subsystem on the aircraft?
Team ownership: Systems engineering is very process driven, so this one is tricky. Dr. Turner puts the idea out that perhaps allowing the systems engineers to drive it instead of the process to drive them, while more uncomfortable for management, might produce more effective results.
There is this story of an aircraft engine plant (September 1999). Their methods seem quite agile:
http://www.fastcompany.com/magazine/28/ge.html
Yes, you could do it. If you followed Agile Software Development techniques too closely however, it would be astronomically expensive, because of the varying costs of activities.
Consider the relative costs of design and build. If we include coding as part of the software design process, then design is definitely the expensive part and build is ridiculously easy and cheap. Most Agile projects would plan to release every few iterations at least. So we can work in small iterations with a continuous build process. Not so easy when you have to assemble a plane once a fortnight. Worse if you actually plan on "releasing" it. You'd probably need to get the airworthiness & safety people on to an Agile process too.
I'd truly love to see it tried.
Yes, you can use agile techniques for building complex systems, but I don't know if I'd use it for this particular system.
The problem with aircraft is the issue of safety. This means every precaution needs to be taken, from correctly identifying and interpreting the requirements to verifying and validating each and every single line of code.
Additionally, formal methods should probably be used to make sure that the system is truly safe by making sure the programming logic is sound and satisfies its conditions properly.
I'm fairly certain the answer is irrelevant. Even if you could, you would not be allowed to. There are too many safety requirements. You would not even be allowed to develop the flight software using Agile. For instance, you do not have a Software Requirements Specification (SRS) in Agile. Yet, for any avionics software onboard an airplane that can affect flight safety you will need an SRS.
Of course one can refactor a plane.
When one refactor, one modifies the source code, not the binaries. With a plane it's exactly the same thing: one modifies the blueprints, not the plane itself.
Well, I know it's not all about speed and memory usage.
But I would like to know what you think will happen to most of the high-level programming languages. As far as I know, Java is much faster than it was in the past, what about python, php etc.
Speed has more to with Moore's law than the language itself. So if you are looking in absolute terms, you'll get more bangs for more buck by just upgrading your machine on a regular basis.
In terms of memory footprint, I expect most languages to continue gathering functionality thus increasing their footprint.
High level programming languages will continue to get more abstractions that make it easier for developers to specificy what they want a computer to do, without having to get their hands dirty with difficult underlying details that a compiler and/or runtime system is better at optimizing anyway than any developer might be able to do a priori.
Think about:
support for multi-threaded execution (like Parallel Extentions in latest .NET)
specifying structure and functional outcome instead of manually telling computer exactly how and in what order to shuffle which sets of bits around
Those kinds of things.
Parallelism, given that increasing the number of processing units (cores) is the principal way of gaining speed nowadays. To make it manageable to humans, software transactional memory seems to be one of the most promising real-world solutions.
We are currently setting up the evaluation criteria for a trade study we will be conducting.
One of the criterion we selected is reliability (and/or robustness - are these the same?).
How do you assess that software is reliable without being able to afford much time evaluating it?
Edit: Along the lines of the response given by KenG, to narrow the focus of the question:
You can choose among 50 existing software solutions. You need to assess how reliable they are, without being able to test them (at least initially). What tangible metrics or other can you use to evaluate said reliability?
Reliability and robustness are two different attributes of a sytem:
Reliability
The IEEE defines it as ". . . the
ability of a system or component to
perform its required functions under
stated conditions for a specified
period of time."
Robustness
is robust if it continues to operate despite abnormalities in input, calculations, etc.
So a reliable system performs its functions as it was designed to within constraints; A robust system continues to operate if the unexpected/unanticipated occurs.
If you have access to any history of the software you're evaluating, some idea of reliability can be inferred from reported defects, number of 'patch' releases over time, even churn in the code base.
Does the product have automated test processes? Test coverage can be another indication of confidence.
Some projects using agile methods may not fit these criteria well - frequent releases and a lot of refactoring are expected
Check with current users of the software/product for real world information.
It depends on what type of software you're evaluating. A website's main (and maybe only) criteria for reliability might be its uptime. NASA will have a whole different definition for reliability of its software. Your definition will probably be somewhere in between.
If you don't have a lot of time to evaluate reliability, it is absolutely critical that you automate your measurement process. You can use continuous integration tools to make sure that you only ever have to manually find a bug once.
I recommend that you or someone in your company read Continuous Integration: Improving Software Quality and Reducing Risk. I think it will help lead you to your own definition of software reliability.
Talk to people already using it. You can test yourself for reliability, but it's difficult, expensive, and can be very unreliable depending on what you're testing, especially if you're short on time. Most companies will be willing to put you in contact with current clients if it will help sell you their software and they will be able to give you a real-world idea of how the software handles.
As with anything, if you don't have the time to assess something yourself, then you have to rely on the judgement of others.
Reliability is one of three aspects of somethings' effectiveness.. The other two are Maintainability and Availability...
An interesting paper... http://www.barringer1.com/pdf/ARMandC.pdf discusses this in more detail, but generally,
Reliability is based on the probability that a system will break.. i.e., the more likely it is to break, the less reliable it is... In other systems (other than software) it is often measured in Mean Time Between Failure (MTBF) This is a common metric for things like a hard disk... (10000 hrs MTBF) In software, I guess you could measure it in Mean Time between critical system failures, or between application crashes, or between unrecoverable errors, or between errors of any kind that impede or adversely affect normal system productivity...
Maintainability is a measure of how long/how expensive (how many man-hours and/or other resources) it takes to fix it when it does break. In software, you could add to this concept how long/how expensive it is to enhance or extend the software (if that is an ongoing requirement)
Availability is a combination of the first two, and indicates to a planner, if I had a 100 of these things running for ten years, after figuring the failures and how long each failed unit was unavailable while it was being fixed, repaired, whatever, How many of the 100, on average, would be up and running at any one time? 20% , or 98% ?
Well, the keyword 'reliable' can lead to different answers... When thinking of reliability, I think of two aspects:
always giving the right answer (or the best answer)
always giving the same answer
Either way, I think it boils down to some repeatable tests. If the application in question is not built with a strong suite of unit and acceptance tests, you can still come up with a set of manual or automated tests to perform repeatedly.
The fact that the tests always return the same results will show that aspect #2 is taken care of. For aspect #1 it really is up to the test writers: come up with good tests that would expose bugs or imperfections.
I can't be more specific without knowing what the application is about, sorry. For instance, a messaging system would be reliable if messages were always delivered, never lost, never contain errors, etc etc... a calculator's definition of reliability would be much different.
My advice is to follow SRE methodology around SLI, SLO and SLA, best summarized in free ebooks:
Site Reliability Engineering which provides principal introduction
The Site Reliability Workbook which comes with concrete examples
Looking at the reliability more from tool perspective you need:
monitoring infrastructure (I recommend Prometheus)
alerting (I recommend Prometheus AlertManager, OpsGenie or PagerDuty)
SLO computation tooling for instance slo-exporter
You will have to go into the process by understanding and fully accepting that you will be making a compromise, which could have negative effects if reliability is a key criterion and you don't have (or are unwilling to commit) the resources to appropriately evaluate based on that.
Having said that - determine what the key requirements are that make software reliability critical, then devise tests to evaluate based on those requirements.
Robustness and reliability cross in their relationship to each other, but are not necessarily the same.
If you have a data server that cannot handle more than 10 connections and you expect 100000 connections - it is not robust. It will be unreliable if it dies at > 10 connections. If that same server can handle the number of required connections but intermittently dies, you could say that it is still not robust and not reliable.
My suggestion is that you consult with an experienced QA person who is knowledgeable in the field for the study you will conduct. That person will be able to help you devise tests for key areas -hopefully within your resource constraints. I'd recommend a neutral 3rd party (rather than the software writer or vendor) to help you decide on the key features you'll need to test to make your determination.
If you can't test it, you'll have to rely on the reputation of the developer(s) along with how well they followed the same practices on this application as their other tested apps. Example: Microsoft does not do a very good job with the version 1 of their applications, but 3 & 4 are usually pretty good (Windows ME was version 0.0001).
Depending on the type of service you are evaluating, you might get reliability metrics or SLI - service level indicators - metrics capturing how well the service/product is doing. For example - process 99% of requests under 1sec.
Based on the SLI you might setup service level agreements - a contract between you and the software provider on what SLO (service level objectives) you would like with the consequences of not them not delivering those.