I've been looking into Spock and I've had experience with FitNesse. I'm wondering how would people choose one over the other - if they appear to be addressing the same or similar problem space.
Also for the folks who have been using Spock or other groovy code for tests, do you see any noticeable performance degradation? Tests are supposed to give immediate feedback - as we know that if the tests take longer to run, the developer tends to run them less frequently - so I'm wondering if the reduction in speed of test execution has had any impact in the real world.
Thanks
I am no FitNesse guy, so please take what I say with a grain of salt. To me it seems what FitNesse is trying to do is to provide a programming language independent environment to specify tests. They use it to have a more visual interface with the programmer. In Spock a Groovy ast transform is used to transform the table into a groovy program.
Since you basically stay in a programming language it is in Spock more easy to realize more complicated test setups. As a result you often seem to have to write fixture code in FitNesse.
I personally don't need a test execution button, I like the direct approach. I like not having to take of even more classes, only to enable testing and I like looking at the code directly. For example I want to just execute my test from the command line, not from a web interface. That is surely possible in FitNesse too, but as a result the whole visual thing FitNesse is trying to give the user is just ballast for me. That's why I would choose Spock over FitNesse.
The advantage of the language agnostic approach is of course, that a lot of test specifications can be used for Java and for .Net. so if that is a requirement for you, you may want to judge different. It usually is not to me.
As for performance, I would not worry too much about that part.
Related
I am now working on a project where we are using cucumber-jvm to drive acceptance tests.
On previous projects I would create internal DSLs in groovy or scala to drive acceptance tests. These DSLs would be fairly simple to use such that even a non-techie would be able to write tests with a little bit of guidance.
What I see is that BDD adds another layer of indirection and semantic sugar to the tests, but I fail to see the value-add, especially if the non-techies can use an internal DSL.
In the case of cucumber, stepDefs seem to scatter the code that drives any given test over several different classes, making the test code difficult to read and debug outside the feature file. On the other hand putting all the code pertaining to one test in a single stepDef class discourages re-use of stepsDefs. Both outcomes are undesirable, leaving me asking what is the use of natural language worth all this extra, and unintuitive indirection?
Is there something I am missing? Like a subtle philosophical difference between ATDD and BDD? Does the former imply imperative testing whereas the latter implies declarative testing? Do these aesthetic differences have intrinsic value?
So I am left asking what is the value add to justify the deterioration in the readability of the actual code that drives the test. Is this BDD stuff actually worth the pain? Is the value add more than just aesthetic?
I would be grateful if someone out there could come up with a compelling argument as to why the gain of BDD surpasses the pain of BDD?
What I see is that BDD adds another layer of indirection and semantic sugar to the tests, but I fail to see the value-add, especially if the non-techies can use an internal DSL.
The extra layer is the plain language .feature file and at the point of creation it has nothing to do with testing, it has to do with creating the requirements of the system using a technique called specification by example to create well defined stories. When written properly in the business language, specification by example are very powerful at creating a shared understanding. This exercise alone can both reduce the amount of rework and can find defects before development starts. This exercise is otherwise known as deliberate discovery.
Once you have a shared understanding and agreement on the specifications, you enter development and make those specifications executable. Here is where you would use ATDD. So BDD and ATDD are not comparable, they are complimentary. As part of ATDD, you drive the development of the system using the behaviour that has been defined by way of example in the story. the nice thing you have as a developer is a formal format that contains preconditions, events, and postconditions that you can automate.
Here on, the automated running of the executable specifications on a CI system will reduce regression and provide you with all the benefits you get from any other automated testing technique.
These really interesting thing is that the executable specification files are long-lived and evolve over time and as you add/change behaviour to your system. Unlike most Agile methodologies where user stories are throw-away after they have been developed, here you have a living documentation of your system, that is also the specifications, that is also the automated test.
Let's now run through a healthy BDD-enabled delivery process (this is not the only way, but it is the way we like to work):
Deliberate Discovery session.
Output = agreed specifications delta
ATDD to drive development
Output = actualizing code, automated tests
Continuous Integration
Output = report with screenshots is browsable documentation of the system
Automated Deployment
Output = working software being consumed
Measure & Learn
Output = New ideas and feedback to feed the next deliberate discover session
So BDD can really help you in the missing piece of most delivery systems, the specifications part. This is typically undisciplined and freeform, and is left up to a few individuals to hold together. This is how BDD is an Agile methodology and not just a testing technique.
With that in mind, let me address some of your other questions.
In the case of cucumber, stepDefs seem to scatter the code that drives any given test over several different classes, making the test code difficult to read and debug outside the feature file. On the other hand putting all the code pertaining to one test in a single stepDef class discourages re-use of stepsDefs. Both outcomes are undesirable, leaving me asking what is the use of natural language worth all this extra, and unintuitive indirection?
If you make the stepDefs a super thin layer on top of your automation testing codebase, then it's easy to reuse the automation code from multiple steps. In the test codebase, you should utilize techniques and principles such as the testing pyramid and the shallow depth of test to ensure you have a robust and fast test automation layer. What's also interesting about this separation is that it allows you to ruse the code between your stepDefs and your unit/integration tests.
Is there something I am missing? Like a subtle philosophical difference between ATDD and BDD? Does the former imply imperative testing whereas the latter implies declarative testing? Do these aesthetic differences have intrinsic value?
As mentioned above, ATDD and BDD are complimentary and not comparable. On the point of imperative/declarative, specification by example as a technique is very specific. When you are performing the deliberate discovery phase, you always as the question "can you give me an example". In that example, you would use exact values. If there are two values that can be used in the preconditions (Given) or event (When) steps, and they have different outcomes (Then step), it means you have two different scenarios. If the have the same outcome, it's likely the same scenario. Therefore as part of the BDD practice, the steps need to be declarative as to gain the benefits of deliberate discovery.
So I am left asking what is the value add to justify the deterioration in the readability of the actual code that drives the test. Is this BDD stuff actually worth the pain? Is the value add more than just aesthetic?
It's worth it if you are working in a team where you want to solve the problem of miscommunication. One of the reasons people fail with BDD is because the writing and automation of features is lefts to the developers and the QA's, and the artifacts are no longer coherent as living specifications, they are just test scripts.
Test scripts tell you how a system does a particular thing but it does not tell you why.
I would be grateful if someone out there could come up with a compelling argument as to why the gain of BDD surpasses the pain of BDD?
It's about using the right tool for the right job. Using Cucumber for writing unit tests or automated test scripts is like using a hammer to put a screw into wood. It might work, but it's never pretty and it's always painful!
On the subject of tools, your typical business analyst / product owner is not going to have the knowledge needed to peek into your source control and work with you on adding / modifying specs. We created a commercial tool to fix this problem by allowing your whole team to collaborate over specifications in the cloud and stays in sync (realtime) with your repository. Check out Simian.
I have also answered a question about BDD here that may be of interest to you that focuses more on development:
Should TDD and BDD be used in conjunction?
Cucumber and Selenium are two popular technologies. Most of the organizations use Selenium for functional testing. These organizations which are using Selenium want to integrate Cucumber with selenium as Cucumber makes it easy to read and to understand the application flow. Cucumber tool is based on the Behavior Driven Development framework that acts as the bridge between the following people:
Software Engineer and Business Analyst.
Manual Tester and Automation Tester.
Manual Tester and Developers.
Cucumber also benefits the client to understand the application code as it uses Gherkin language which is in Plain Text. Anyone in the organization can understand the behavior of the software. The syntax's of Gherkin is in the simple text which is readable and understandable.
Is it a must to know Groovy to start learning Ofbiz? The one book i can find is for version 4 of ofbiz and it uses Beanshell. I don't know if later versions of ofbiz have added support for groovy.
Objective: to create workflows as necessary.
Thanks.
EDIT: Found this. From what i understand, one can use any compliant Java scripting language.However, Groovy will be supported OOTB. (Groovy is not just a scripting language, but it is one of the roles)
Later versions have added support for Groovy.
AFAIK, you can still use Beanshell if you want
No, you shouldn't, unless you want use not java but groovy to develop something - e.g, event, service - for OFBiz.
If you familiar with java, that's enough for you to read and understand the source code written in groovy, and it's enough in most case to write some simple in-line groovy script used in OFBiz.
"Know groovy", it's nice-to-have but not must to "start learning OFBiz".
Groovy is used a lot in screen actions, which is the data preparation code that is part of generating UI output. There is a lot of code in the project like this.
It can also be used to implement services and request events (used for processing input), and is a popular tool for custom extensions to OFBiz even though not used a lot in OFBiz itself.
As stated in other answers if you know Java it's easy to read most Groovy code (some closure syntax can be confusing at first), but it's worth learning about more to reduce code size and effort, and make your code cleaner and easier to maintain. In other words, Groovy has a lot of extensions beyond plain Java that are very useful, especially for business logic in applications like those built with Apache OFBiz.
As a case in point, the next generation framework based on the ideas in OFBiz (Moqui Framework, www.moqui.org) is written largely in Groovy and supports Groovy for everything whereas OFBiz also uses JUEL for expressions (and even Beanshell still in a couple places). I should note that both frameworks support a number of other scripting languages for business logic if you have other strong preferences, but it is nice to standardize on one so that developers have less to learn and can more easily work with existing business logic and (as applicable) framework code.
I often listen people talk about unit test. But I don't know the exact apperance of it, especially in Linux C code. I know that for VisualStudio this kind of tool will provide some template? to do unit test, but how to make one for Linux general C code. Could anyone show me an example? Or maybe show me how to do TDD for Linux C code? It's mysterious for me. Thanks!
Unit Testing is a type of automated test which tests individual units of functionality. It's especially useful during initial development and for ensuring that changes don't break your program, which is invaluable when you refactor. You can read more here:
http://en.wikipedia.org/wiki/Unit_testing
Unit testing became a big deal with Extreme Programming and Test Driven Development. With that, a large number of unit testing frameworks sprung up.
Which answers your second question. To do unit testing in C, you'll need a unit testing framework for the language. Fortunately, there are a variety available that you can just use, so you needn't worry about creating your own:
http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C
If you choose to do your own, you'll need to find a way to attach tests with units of functionality, ensure they are all run when you run your program, and have a way to remove them easily for deployment. Macros will likely be a key.
First, I have to say that I really like Groovy and all the good stuff it is bringing to the Java dev world. But since I'm using it for more than little scripts, I have some concerns.
In this Groovy help page about dynamic vs static typing, there is this statement about the absence of compilation error/warning when you have typo in your code because it could be a call to a method added later at runtime:
It might be scary to do away with all of your static typing and
compile time checking at first. But many Groovy veterans will attest
that it makes the code cleaner, easier to refactor, and, well, more
dynamic.
I'm pretty agree with the 'more dynamic' part, but not with cleaner and easier to refactor:
For the other two statements I'm not sure: from my Groovy beginner perspective, this is resulting in less code, but in more difficult to read later and in more trouble to maintain (can not rely on the IDE anymore to find who is declaring a dynamic method and who is using one).
To clarify, I find that reading groovy code is very pleasant, I love the collection and closure (concise and expressive way of tackle complicated problem).
But I have a lot of trouble in these situations:
no more auto-completion inside 'builder' using Map (Of Map (of Map))
everywhere
confusing dynamic methods call (you don't know if it is a typo or a
dynamic name)
method extraction is more complicated inside closure (often resulting in code duplicate: 'it is only a small closure after all')
hard to guess closure parameters when you have to write one for a method of a subsystem
no more learning by browsing the code: you have to use text search instead
I can only saw some benefits with GORM, but in this case the dynamic method are wellknown and my IDE is aware of them (so it is more looking like a systematic code generation than dynamic method for me)
I would be very glad to learn from groovy veteran how they can attest of these benefits.
It does lead to different classes of bugs and processes. It also makes writing tests faster and more natural, helping to alleviate the bug issues.
Discovering where behavior is defined, and used, can be problematic. There isn't a great way around it, although IDEs are getting better at it over time.
Your code shouldn't be more difficult to read--mainline code should be easier to read. The dynamic behavior should disappear into the application, and be documented appropriately for developers that need to understand functionality at those levels.
Magic does make discovery more difficult. This implies that other means of documentation, particularly human-readable tests (think easyb, spock, etc.) and prose, become that much more important.
This is somewhat old, but i'd like to share my experience if someone comes looking for some thoughts on the topic:
Right now we are using eclipse 3.7 and groovy-eclipse 2.7 on a small team (3 developers) and since we don't have tests scripts, mostly of our groovy development we do by explicitly using types.
For example, when using service classes methods:
void validate(Product product) {
// groovy stuff
}
Box pack(List<Product> products) {
def box = new Box()
box.value = products.inject(0) { total, item ->
// some BigDecimal calculations =)
}
box
}
We usually fill out the type, which enable eclipse to autocomplete and, most important, allows us to refactor code, find usages, etc..
This blocks us from using metaprogramming, except for Categories which i found that are supported and is detected by groovy-eclipse.
Still, Groovy is pretty good and a LOT of our business logic is in groovy code.
We had two issues in production code when using groovy, and both cases were due bad manual testing.
We also have a lot of XML building and parsing, and we validate it before sending it to webservices and the likes.
There's a small script we use to connect to an internal system whose usage is very restricted (and not needed in other parts of the system). This code i developed using entirely dynamic typing, overriding methods using metaclass and all that stuff, but this is an exception.
I think groovy 2.0 (with groovy-eclipse coming along, of course) and it's #TypeChecked will be great for those of us that uses groovy as a "java++".
To me there are 2 types of refactoring:
IDE based refactoring (extract to method, rename method, introduce variable, etc.).
Manual refactoring. (moving a method to a different class, changing the return value of a method)
For IDE based refactoring I haven't found an IDE that does as good of a job with Groovy as it does with Java. For example in eclipse when you extract to method it looks for duplicate instances to refactor to call the method instead of having duplicated code. For Groovy, that doesn't seem to happen.
Manual refactoring is where I believe that you could see refactoring made easier. Without tests though I would agree that it is probably harder.
The statement at cleaner code is 100% accurate. I would venture a guess that good Java to good Groovy code is at least a 3:1 reduction in lines of code. Being a newbie at Groovy though I would strive to learn at least 1 new way to do something everyday. Something that greatly helped me improve my Groovy was to simply read the APIs. I feel that Collection, String, and List are probably the ones that have the most functionality and I used the most to help make my Groovy code actually Groovy.
http://groovy.codehaus.org/groovy-jdk/java/util/Collection.html
http://groovy.codehaus.org/groovy-jdk/java/lang/String.html
http://groovy.codehaus.org/groovy-jdk/java/util/List.html
Since you edited the question I'll edit my answer :)
One thing you can do is tell intellij about the dynamic methods on your objects: What does 'add dynamic method' do in Groovy/IntelliJ?. That might help a little bit.
Another trick that I use is to type my objects when doing the initial coding and remove the typing when I'm done. For example I can never seem to remember if it's .substring(..) or .subString(..) on a String. So if you type your object you get a little better code completion.
As for your other bullet points, I'd really need to look at some code to be able to give a better answer.
We're writing a large production system in Java, and I'm considering whether or not we can write some of the components in one of the JVM-based dynamic languages. Groovy appears to be the best choice from the Java interoperability standpoint. But is the Groovy implementation reliable enough to use in production (I would assume so), and is the Groovy language specification itself stable enough so that we aren't going to have to revise our production code substantially in a year or two? What are your experiences?
Summary (5/30/09): Good comments, the sense I get is that you should be cautious in adopting Groovy for mission-critical production use, it's fine for ancillary usages like putting together test cases, and there's a middle ground where it's probably fine but do your homework first. Performance is an issue, which needs to be balanced against the increase in developer productivity. Bill and Ichorus have equally helpful answers based on Groovy use, so it was a coin toss.
Update (12/3/09): More recently I've been taking a serious look at Scala, another JVM language. It was designed and implemented by Martin Odersky, the original author of the current javac compiler and the co-designer of Java Generics. Scala is a strongly typed, but uses type inferencing to strip out a lot of boilerplate. It's a nice blend of object-oriented and functional programming. James Gosling likes it. James Strachan, the author of Groovy, likes it too. And Odersky's experience writing javac means Scala's raw performance is not far from Java's, which is impressive.
Update (4/26/11): Take a look at Groovy++, a statically typed extension of Groovy, which has performance equivalent to Java. Looks very interesting.
Edit: Here almost four years later and Groovy has become much more solid.
I can wholeheartedly recommend it for production grade projects.
I've been using Groovy to support production applications for a while and for that purpose it is stable enough. As for actually having Groovy in bona fide production code; I don't think I would do that. Groovy does too many surprising things. It has gotten much better in this regard over the past year or so, but every once in awhile I will run into a bug that is a bit difficult to track down because of the generated code (my issues seem to have revolved around scoping).
I have gotten away from Groovy (though the stuff that we use that is simple and solid is still around) and have used Python (jython implementation), which has been far more predictable in my opinion. Also, python trumps Groovy in readability.
You can write some very interesting code in Groovy with closures and operator overloading and whatnot.
These languages are used for convenience and speed on ancillary code...stuff that can be switched out on the fly if need be. None of it is in production. I don't think I would put either in production unless it was as a stopgap to get something critical out of the door in a major hurry or as a proof of concept or prototype.
And in the case of putting it in actual production, it would have to be in the most dire of circumstances and I would assign someone to rewrite it in pure Java for the next release. I am 98% sure that either would be fine in production but that 2% is too much unnecessary risk.
We've got several production apps running on Grails (using Groovy as the language). So far, no issues have resulted. As for JVM compatibility, take a look at how little the JVM byte-code has changed in the last 5 years... like 1 instruction was added, and none were made obselete.
Will new versions of Groovy come out in the next year? Yes. Will you be required to change to them? No. Though you might want to, 1.6 is a huge speed improvement.
Which brings us to the major drawback of Groovy, the speed issue. Obviously, Groovy is slower than straight Java. The current number are up to 10 TIMES slower, for certain actions. That said, is your CPU the bottleneck in your app? For us, it's mostly DB access and latency. If it's the same for you, what matter if the CPU spends 200ms processing the page request instead of 35ms?
Is that the only problem with Groovy? Nope. Dynamic languages have refactoring difficulties, since there isn't necessarily a complete class specification anywhere in the code. However, this is partially balanced by the smaller code-size which makes it easier to find the places to modify the code.
Anyway, Groovy is a perfectly fine language for production uses. Mix it with Java for your "critical" code, if you fear the reliability. That's the BEST part of Groovy... how easy it mixes with Java classes.
I'm currently exploring using Groovy for only writing unit tests. This has the effects of
Allowing the potentially tedious part of writing tests to be done in a syntax that is a bit simpler than Java.
Keeps the Groovy code out of production.
Allows a large portion of the code base to be written in a non-Java language.
Of course, we haven't started yet, but this is at least my way of attempting to introduce alternate JVM languages to our new projects (and possibly existing ones). I have the same concerns you do, and even more so around performance than stability.
Scripting languages evolve "too fast" in the aspect of syntactic features.
If you want a language for the JVM that will stay compatible for
many years,
Java is your only choice ;)
By the way, I don't think that the readability of code is
ensured by a scripting language automatically.
We used Grails/Groovy as our primary backend at my previous company, and from that experience I'd say that I would choose Groovy over Java in most circumstances I am likely to encounter, since it interoperates with Java seamlessly and is otherwise more fun and expressive. Additionally I would expect the database would almost always be the bottleneck of my application rather than language performance, and we didn't encounter any stability issues/bugs with groovy as far as I recall.
But personally it's usually not about Groovy vs Java for me in most cases -- it's about Groovy/Java + available libraries vs. other languages like Python/Jython/JavaScript/Ruby + available libraries. And there are a lot of other considerations there such as strength of community, maturity of relevant technologies for your particular application, etc. In particular, for web development, Grails was decent, but the community seemed lacking. My overall opinion is that I would use python or Node.js going forward. If I needed the JVM I'd use a jython-compatible python web development environment.
I've been playing with Groovy for a month or so. The simplicity is awesome, and the dynamic language features are really cool too. However, speed is definitely an issue. Also, the groovy console really sucks. You cannot do things that you can do e.g. in python. Every once in a while I have to restart the console, reimport, things, etc. It also keeps forgetting the values I put in the variables while in the console mode; somehow mystically they go out of scope. (Is it because of the JVM? I don't know.) I cannot come up with an example from the top of my head, but the behavior I get in the Groovy console is different from the behavior I get in the Grails console, and is different from what I get by just writing code in a script.
A few more warnings. Note that Groovy is almost, but not 100% compatible with Java. For example, this will not compile:
public class HelloWorld {
public static void main(String args[]) {
System.out.println( "Hello, world!\n");
}
}
Also take a look at How to get classpath in Groovy?