I have an Alloy model. The model is of some decision making logic in software I wrote. In that model I have a few predicates that create examples. The predicate creates instances that are expected behavior and outside expected behavior. I would love to take those examples as inputs to a unit test for my code.
Does anyone have an example of some software that interacts with Alloy to dump many examples generated to a single file? I would love to run a program, get a file with many instances in it, and then use that file as input to my test program.
This interests me because the examples and counter examples created are often not what I would think to do when hand writing my test inputs.
Thoughts?
You can export an instance in the File/Export To menu.
If you can work in Java, then maybe it is interesting to know we're setting up an open source repo on github: https://github.com/AlloyTools/
I think it is quite easy to link your code with this code and generate your test cases or provide them from proper files.
I am extremely interested in this kind of applications for Alloy so please keep us posted on https://groups.google.com/forum/#!forum/alloytools
Related
I have a reasonably complex structure of data types and records, which is not so easy to make sense of by just looking at production code for someone not familiar with the codebase.
To make sense of it, I created this kind of dummy functions that have two advantages: 1. they're checked at compile-time and 2. they serve as some kind of documentation, showcasing an example of how the overall structure of data types and records mixes together:
-- benefit 1: a newcomer can quickly make sense of the type system
-- benefit 2: easier to keep track of how the types evolve because of compile-time checking
exampleValue1 :: ApiResponseContent
exampleValue1 = ApiOnlineResponseContent [
OnlineResultRow (EntityId 10) [Just (FvInt 1), Just (FvFloat 1.5), Nothing],
OnlineResultRow (EntityId 20) [Just (FvInt 2), Nothing, Just (FvBool True)]
]
The only thing that bothers me a bit is that they feel a bit awkward put within the production code, as they're clearly dead code. However they're not tests either, they're just compiled-time-checked examples of how values can be assembled together from complex nested types. Therefore, they clearly don't belong to production code, but they don't quite belong to tests either.
Is it common practice to have this kind of compile-time examples? And where should they be placed within the codebase?
Have you considered including the examples as part of Haddock documentation?
Otherwise, there's a school of thought that try to reframe tests as examples. You can find a brief mention of this in Gerard Meszaros's work on unit testing. Dan North has also repeatedly used similar language, but it can often be difficult to track down his many iterations of such ideas. He tends to 'think in public' - here's one such reflection:
"I call this development, using example-guided design"
I understand that the kind of example being asked about isn't quite like that. Still, I think that it better belongs with test code than with production code.
If you put the examples in the production code, you essentially make it part of the API of your library (if you're shipping a library). This means that changing the examples would constitute a breaking change. That doesn't seem right to me.
In the BDD/DDD community, there's a lot of emphasis on tests as examples, also in the sense that automated tests serve as documentation. If Haddock documentation isn't an option, I'd consider putting the examples in the test code, as documentation. I sometimes do that by simply putting such 'vacuous tests' in a file called examples, perhaps with a little comment at the top to explain that the code in that file assists learning rather than verify behaviour.
It dodges the risk of introducing redundant breaking changes in the production code, and seems conceptually like a better fit.
I disagree that these aren't tests. Even “does this example compile” could be seen as a test, but probably you could also use them to actually test some functions, while you're at it.
So, put these definitions in your test suite, and use them for unit testing the functions that will actually be dealing with such values.
The main convention I’ve seen for housing such examples is to include ….Tutorial modules, such as Dhall.Tutorial and Clash.Tutorial, or parallel …-tutorial packages such as lens-tutorial.
These contain Haddock documentation, organised so as to read in linear order, alongside example definitions like yours.
With good examples, I find this convention very helpful for understanding and experimenting with a new package in addition to types and reference docs. It’s also fairly discoverable when browsing packages on Hackage, especially if you ensure that the reference and README link to the tutorial for longer explanations.
These modules can be just compiled as basic validation, but having more machine validation for docs is incredibly valuable, so I think it’s even better to link them into the test suite (e.g. hspec) as examples for unit tests (HUnit) or property tests (QuickCheck/hedgehog), or organise them as documentation tests (doctest).
In addition to GHC’s code coverage tools, weeder can help identify examples that aren’t being tested.
I'm using Alloy to model a system. I would like to check the implemented system matches the Alloy model by comparing log traces from a concrete execution of the actual system with the model.
The way I see this working is:
Add logs to the implemented system at points that correspond to the high level concepts modelled in Alloy, such as "Receptionist checks in guest G1"
Pre-process these into a form understood by Alloy
Give this to Alloy (or some other tool) and say 'Does this model admit this trace?' (this question)
This would be run over the operational logs of the system (or maybe subsets if performance is a problem) and continuously validate that the system was operating 'to spec'.
Is that possible / reasonable?
Possible yes.
Reasonable I'm not quite sure.
To me, Alloy shine at finding unknown unknowns, i.e. pitfalls, in your specifications.
Once the specification is fool-proofed using Alloy analysis, I don't see the point of encumbering your program with unnecessary translations and analysis steps. It's not only error prone, but you might also find yourself limited with the scalability of the analyzer if the traces you want to validate are substantial...
But again, it's doable. So if you want it, sure, do it ... :-)
I'm using Alloy to model a system. I would like to check the implemented system matches the Alloy model by comparing log traces from a concrete execution of the actual system with the model.
Yes, I think that is a bit of work but it should be doable. I would be very interested in getting this to work. I have been thinking about this for a long time.
Loïc argues correctly that Alloy shines in finding solutions but to keep this manageable, Alloy must keep the scope small. Although this is true, Alloy is also a specification language. The timing issue is only in finding a solution. However, the problem you sketch is different, you already have the solution in the log. Each event specifies a transition in the state.
If you're familiar with the Alloy Evaluator then you should be aware that once you have a solution, you can run any Alloy code on that instance. Inside Alloy, there is a full set of classes to simulate an instance and run Alloy code against it.
So I think you can start with an initial instance and use your log event to create a secondary instance and then use Alloy to verify this is a valid transition. This will be very fast and I do not see why this could not handle a very large number of objects. Surely thousands and with a bit of caching wizardry millions.
We are currently working hard on Alloy 6, which will be integrate Electrum, where we will have full temporal logic that will make the rules easier to express.
I've been looking for a customer for a long time that would like to develop the necessary code to bridge Alloy & the trenches. If this works as I think it can work, it would be very interesting for the software industry.
I am now working on a project where we are using cucumber-jvm to drive acceptance tests.
On previous projects I would create internal DSLs in groovy or scala to drive acceptance tests. These DSLs would be fairly simple to use such that even a non-techie would be able to write tests with a little bit of guidance.
What I see is that BDD adds another layer of indirection and semantic sugar to the tests, but I fail to see the value-add, especially if the non-techies can use an internal DSL.
In the case of cucumber, stepDefs seem to scatter the code that drives any given test over several different classes, making the test code difficult to read and debug outside the feature file. On the other hand putting all the code pertaining to one test in a single stepDef class discourages re-use of stepsDefs. Both outcomes are undesirable, leaving me asking what is the use of natural language worth all this extra, and unintuitive indirection?
Is there something I am missing? Like a subtle philosophical difference between ATDD and BDD? Does the former imply imperative testing whereas the latter implies declarative testing? Do these aesthetic differences have intrinsic value?
So I am left asking what is the value add to justify the deterioration in the readability of the actual code that drives the test. Is this BDD stuff actually worth the pain? Is the value add more than just aesthetic?
I would be grateful if someone out there could come up with a compelling argument as to why the gain of BDD surpasses the pain of BDD?
What I see is that BDD adds another layer of indirection and semantic sugar to the tests, but I fail to see the value-add, especially if the non-techies can use an internal DSL.
The extra layer is the plain language .feature file and at the point of creation it has nothing to do with testing, it has to do with creating the requirements of the system using a technique called specification by example to create well defined stories. When written properly in the business language, specification by example are very powerful at creating a shared understanding. This exercise alone can both reduce the amount of rework and can find defects before development starts. This exercise is otherwise known as deliberate discovery.
Once you have a shared understanding and agreement on the specifications, you enter development and make those specifications executable. Here is where you would use ATDD. So BDD and ATDD are not comparable, they are complimentary. As part of ATDD, you drive the development of the system using the behaviour that has been defined by way of example in the story. the nice thing you have as a developer is a formal format that contains preconditions, events, and postconditions that you can automate.
Here on, the automated running of the executable specifications on a CI system will reduce regression and provide you with all the benefits you get from any other automated testing technique.
These really interesting thing is that the executable specification files are long-lived and evolve over time and as you add/change behaviour to your system. Unlike most Agile methodologies where user stories are throw-away after they have been developed, here you have a living documentation of your system, that is also the specifications, that is also the automated test.
Let's now run through a healthy BDD-enabled delivery process (this is not the only way, but it is the way we like to work):
Deliberate Discovery session.
Output = agreed specifications delta
ATDD to drive development
Output = actualizing code, automated tests
Continuous Integration
Output = report with screenshots is browsable documentation of the system
Automated Deployment
Output = working software being consumed
Measure & Learn
Output = New ideas and feedback to feed the next deliberate discover session
So BDD can really help you in the missing piece of most delivery systems, the specifications part. This is typically undisciplined and freeform, and is left up to a few individuals to hold together. This is how BDD is an Agile methodology and not just a testing technique.
With that in mind, let me address some of your other questions.
In the case of cucumber, stepDefs seem to scatter the code that drives any given test over several different classes, making the test code difficult to read and debug outside the feature file. On the other hand putting all the code pertaining to one test in a single stepDef class discourages re-use of stepsDefs. Both outcomes are undesirable, leaving me asking what is the use of natural language worth all this extra, and unintuitive indirection?
If you make the stepDefs a super thin layer on top of your automation testing codebase, then it's easy to reuse the automation code from multiple steps. In the test codebase, you should utilize techniques and principles such as the testing pyramid and the shallow depth of test to ensure you have a robust and fast test automation layer. What's also interesting about this separation is that it allows you to ruse the code between your stepDefs and your unit/integration tests.
Is there something I am missing? Like a subtle philosophical difference between ATDD and BDD? Does the former imply imperative testing whereas the latter implies declarative testing? Do these aesthetic differences have intrinsic value?
As mentioned above, ATDD and BDD are complimentary and not comparable. On the point of imperative/declarative, specification by example as a technique is very specific. When you are performing the deliberate discovery phase, you always as the question "can you give me an example". In that example, you would use exact values. If there are two values that can be used in the preconditions (Given) or event (When) steps, and they have different outcomes (Then step), it means you have two different scenarios. If the have the same outcome, it's likely the same scenario. Therefore as part of the BDD practice, the steps need to be declarative as to gain the benefits of deliberate discovery.
So I am left asking what is the value add to justify the deterioration in the readability of the actual code that drives the test. Is this BDD stuff actually worth the pain? Is the value add more than just aesthetic?
It's worth it if you are working in a team where you want to solve the problem of miscommunication. One of the reasons people fail with BDD is because the writing and automation of features is lefts to the developers and the QA's, and the artifacts are no longer coherent as living specifications, they are just test scripts.
Test scripts tell you how a system does a particular thing but it does not tell you why.
I would be grateful if someone out there could come up with a compelling argument as to why the gain of BDD surpasses the pain of BDD?
It's about using the right tool for the right job. Using Cucumber for writing unit tests or automated test scripts is like using a hammer to put a screw into wood. It might work, but it's never pretty and it's always painful!
On the subject of tools, your typical business analyst / product owner is not going to have the knowledge needed to peek into your source control and work with you on adding / modifying specs. We created a commercial tool to fix this problem by allowing your whole team to collaborate over specifications in the cloud and stays in sync (realtime) with your repository. Check out Simian.
I have also answered a question about BDD here that may be of interest to you that focuses more on development:
Should TDD and BDD be used in conjunction?
Cucumber and Selenium are two popular technologies. Most of the organizations use Selenium for functional testing. These organizations which are using Selenium want to integrate Cucumber with selenium as Cucumber makes it easy to read and to understand the application flow. Cucumber tool is based on the Behavior Driven Development framework that acts as the bridge between the following people:
Software Engineer and Business Analyst.
Manual Tester and Automation Tester.
Manual Tester and Developers.
Cucumber also benefits the client to understand the application code as it uses Gherkin language which is in Plain Text. Anyone in the organization can understand the behavior of the software. The syntax's of Gherkin is in the simple text which is readable and understandable.
I'm new to UML and I have written a Sequence diagram for a Voucher User Interface, it is a small pet project I'm doing to learn UML. I have written a decent Sequence Diagram in StarUML and now I would like to generate the C++ code, the step is very simple I just go to Tools/C++/Generate Code... But there I face a problem, which you can understand from the screen captures below. I tried various places but I could not find any help, anyone experienced in StarUML who could help me go further?
Haven't used starUML for a while so things may have changed. But:
Last time I looked, it didn't provide templates for behavioural code generation. The only codegen templates were for static structure (classes, attributes, packages). If that's still the case then I suspect the empty panel arises from it not finding any suitable model elements to generate code for. i.e. your Sequence Diagram package doesn't have any classes in it.
You might be able to work it out by looking at the codegen templates; they should be in
<STARUML_INSTALL_PATH>\modules\staruml-generator\templates
hth.
I just tried to generate the code from start uml.
With Version 5.0.3.1570. It works well.
I got the diagram instantly.
One issue, it actually did not generate the composition / aggregation stuff but, then, I did that manually.
Since its propitiatory for my organization, I cannot revel but, sequence is exactly the same as of yours.
I choose Design model & then gave file path in dialog box.
I recently inherited a large codebase and am having to read it. The thing is, I've usually been the dev starting a project. As a result, I don't have a lot of experience reading code.
My reaction to having to read a lot of code is, well, umm to rewrite it. But I need to bring myself up to speed quickly and build on top of an existing system.
Do other people have techniques they've learned to absorb a code base? At this point, I'm just reading through the code. I've tried generating UML diagrams using UModel. They're so big they won't print cleanly and when I zoom in, I really do lose the perspective of seeing all the relationships.
How have other people dealt with this problem?
Wow - I literally just finished listening to a podcast on reading code!!!
http://www.pluralsight-training.net/community/blogs/pluralcast/archive/2010/03/01/pluralcast-10-reading-code-with-alan-stevens.aspx?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+pluralcast+%28Pluralcast+by+Pluralsight%29
I would recommend listening to this. One interesting point that was made that I found radical and may be something you could try (I know I'm going to!). Download the entire source code base. Start editing and refactoring the code then...throw that version away!!! I think with all the demands that we have with deadlines that doing this would not even occur to most developers.
I am in a similar position to you in my own work and I have found the following has worked for me:
- Write test cases on existing code. To be able to write the test case you need to be able to understand the cde base.
- If it is available, look at the bug\issues that have been documented through the life cycle of the product and see how they were resolved.
- Try and refactor some of the code - you'll probably break it, but that's fine you can throw it away and start again. By decomposing the code into smaller problems you'll understand it bettter
You don't need to make drastic changes when refactoring though. When your reading the code and you understand something, rename the variable or the method names so the better reflect the problem the are trying to solve.
Oh and if you can, please get a copy of Working Effectively with Legacy Code by Michael C. Feathers - I think you'll find it invaluable in your situation.
Good luck!
This article provides a guideline
Visualization: a visual representation of the application's design.
Design Violations: an understanding of the health of the object
model.
Style Violations: an understanding of the state the code is currently
in.
Business Logic Review: the ability to test the existing source.
Performance Review: where are the bottlenecks in the source code?
Documentation: does the code have adequate documentation for people
to understand what they're working on?
In general I start at the entry point of the code (main function, plugin hook, etc) and work through the basic execution flow. If its a decent code base it should be broken up into decent size chunks, and you can then go through and figure out what each chunk of the code is responsible for. Looking back at when in the execution flow of the system its called.
For the package/module/class exploration I use whatever doxygen will generate once its been run over the sources. It generates some nice class relation diagrams, inheritance hierarchies and file dependencies graphs. The benefit of these is they are each focused on a single class and how it ties it with its neighbors, siblings and parents, so the graphs are usually of manageable size and easy to understand.
As you understand what different, classes, functions and sub-systems do I like to add comments to fill what sounds like obviously missing documentation. This helps you when you re-read through the code the second time.
I would recommend another podcast and resources:
SE-Radion episode on Software Archeology