I just pushed a small change to JUnit integration support for FitNesse to github; hopefully this should be merged with Uncle Bob’s master branch soon. The change enables you to output test run stats in Junit XML files, which can then be picked up by Hudson, TeamCity and other CI servers and integrated into their test history statistics.

To enable this feature, override the default test listener in the JUnitHelper instance you use for test runs (third constructor argument), and supply a JUnitXMLTestListener instance passing the folder where you want the XML files in the constructor argument. For example:

 String htmlOutputDir=new File(System.getProperty("java.io.tmpdir"),
                                  "fitnesse").getAbsolutePath();
  String xmlOutputDir=new File(System.getProperty("java.io.tmpdir"),
                                   "fitnesse-xml").getAbsolutePath();
  String fitNesseRootDir=".";
  JUnitXMLTestListener xmlTestListener=new JUnitXMLTestListener(xmlOutputDir);
  JUnitHelper helper =
      new JUnitHelper(fitNesseRootDir,htmlOutputDir,xmlTestListener);
    helper.assertTestPasses(".....");
 

Grab the source from my github repository. Also, check out the full example.

Here’s a video from a joint workshop that David Evans, Mike Scott and I organised yesterday at Skills Matter. We talked about strategies to get the most out of acceptance tests (especially with FitNesse) and organised a group workshop to review some good and bad examples of acceptance tests – taken from my Hands On Acceptance Testing workshop.

While doing research for my new book, I was very surprised to find out that Jim Shore gave up on acceptance testing. I use his “describe-demonstrate-develop” process description all the time in my workshops, so I guess I better stop doing that. Jim Shore wrote:

My experience with Fit and other agile acceptance testing tools is that they cost more than they’re worth. There’s a lot of value in getting concrete examples from real customers and business experts; not so much value in using “natural language” tools like Fit and similar.

The two failure patterns that Shore describes in his post are falling back on testers to write everything and merging acceptance and integration tests. I’ve experienced both of these myself, and it seems that they are common in general. We discussed both during the top 10 ways to fail with acceptance testing openspace session at CITCON Europe last year. However, there are good ways to solve both problems.

I never really expected customers to write anything themselves, but I was relatively successful in persuading them to participate in specification workshops that led to examples which were then converted to acceptance tests later. Another idea I discovered while doing the research for my new book is discussing the key examples with customers and then going off to write detailed test pages, which then get verified by the customers. The third good idea is doing ad-hoc specification writing sessions when a developer needs information, by involving a tester and a business analyst. This is a lot less formal than a specification workshop and gives you similar benefits if you have all the knowledge in the room (or readily available) most of the time.

Not preserving acceptance tests as a separate group and mixing quick and slow tests is something that most people, at least according to my ongoing research, get burned with at some point but again teams learn from that and improve.

One of the biggest benefits from acceptance testing for me was that the teams finally get a source of information on what goes on in the system as reliable as the code itself. Without acceptance tests, code is the only thing you can really trust and any other documentation gets outdated very quickly. And such tests are much easier to read and understand than the code because they are on a higher level and in a natural language. Having a live specification helps me quite a lot when change requests come in later. It also helps with handing over and taking over code. Acceptance tests stay relevant throughout the project because they are automated, and automated tests are kept up to date in order for them to pass. Automation, and consequently a tool, are necessary to get this benefit. With informal agreements and on-site reviews that Jim Shore describes, I guess something else needs to be in place to facilitate this.

I agree with Shore that it takes a while for the problems with tools such as FIT to surface, but I’m not sure whether that is tool related or not. Most people I spoke to so far said that it took them between six months and a year to discover that acceptance testing isn’t about the tools but about communication, and that the biggest benefit is in the examples as Shore wrote. A notable exception to six months to a year rule was Lisa Crispin’s team who generally started out knowing what they need (but that’s because she had done it before). Clear examples and improved communication are the biggest benefits of the process, but using a tool brings some additional nice benefits as well. A tool gives us an impartial measure of progress. Ian Cooper said during the interview for my new book that “the tool keeps developers honest”, and I can certainly relate to that. With tests that are evaluated by an impartial tool, “done” is really “what everyone agreed on”, not “almost done with just a few things to fill in tomorrow”. I’m not sure whether an on-site review is enough to guard against this completely.

On March 2nd, Dave Evans and I are running a FitNesse clinic in central London starting at 6:30 PM. This is a unique opportunity to get your FitNesse tests reviewed for free and get help with writing and maintaining FitNesse tests and suites.

Dave and I will discuss some of the common pitfalls faced by teams in getting to grips with Fitnesse. We will show examples of good and bad acceptance tests, illustrating how different styles of fixtures lend themselves to different types of tests. We also highlight some of the features of Fitnesse that allow you to keep your tests expressive, useful and easy to maintain.

Make this interactive and bring your own test examples to be discussed, critiqued and improved upon in a group workshop.

For more information and to register, see
http://skillsmatter.com/podcast/agile-testing/david-evans-gojko-adzic-interactive-agile-acceptance-testing-clinic

Effective prioritisation is key to delivering value with a software project. The MoSCoW model, splitting features into Must have, Should have, Could have and Would like (really Won’t have), theoretically does the trick. However, I found it very hard to apply in practice – too many things end up in the Must category. I recently came across an alternative model that seems to solve that problem.

The MoSCow model, at least when I tried to implement it, suffered from the Priority 1 Paradox:

The more we ask customers to narrow down priority 1 features the more they’ll be scared to leave anything out of that category.

The complaint is always the same: everything must be there – especially all the old functionality when replacing a legacy system. Prioritising so that most of the stuff ends up as ‘Must’ pretty much defeats the point of the exercise.

I am currently reading Stand Back and Deliver: Accelerating Business Agility by Niel Nickolaisen and colleagues from Accelinova. Among other interesting ideas, I found a real gem. This book advises aligning business processes (equally applicable to software features) along two axes: mission critical and market differentiation. Nickolaisens calls this the Purpose Alignment Model:

In a case study, the authors describe a discussion with a stakeholder who claimed that their IP protection was a differentiating factor. The authors then asked the VP of sales ‘When was the last time you sealed a deal by touting Lifebrands’ excellence in IP protection? When was the last time you felt you should include the IP protection process in sales presentations and collateral?‘ (pp 36) They also offer a simpler version of this rule of thumb as a Billboard filter:

Would you put this feature on a billboard?

If so, the feature is obviously differentiating and should be invested in. If not, this feature falls into the parity category at best. The introduction of the parity category which, as the authors repeatedly point out, holds mission-critical features which are not billboard material, seems as a great way to avoid the Priority 1 paradox. This is to intentionally avoid the feeling that anything not Priority 1 won’t be delivered. The system must have these things too offer a full service, but it just isn’t worth investing too much or being too creative there. Nickolaisen and co-authors suggest that the Parity category is where companies should apply industry best practices and streamline processing.

Overdoing it in this category might actually be damaging to the bottom line. In another case study, the authors describe a project to automate complicated pricing rules. After it was identified that complex pricing isn’t really billboard material, the project was completely turned on its head. Instead of implementing a complex rule engine, the pricing system was streamlined using industry best practices. It turned out that customers were actually happier with the new model, so simplifying a parity process increased the revenue. This was possible because the stakeholders agreed that while the company must have an automated pricing system, it wasn’t a core differentiating factor.

Strategic design ideas of Eric Evans also suggest some good guidelines what to do with such software: if the thing is available off the shelf (generic domains), buy it. If not, possibly get a third party to develop and maintain it. Evans calls these domains supporting, and they are prime candidates for outsourcing.

This article is part of my Building software that matters series. See other articles here

Btw, I’ve created a mailing list for this topic so rather than commenting here, post your comments to the Building Software That Matters mailing list and let’s discuss them!

Image credits:Steve Rhodes

Here’s the video from my talk on Behaviour Driven Development and Acceptance Testing with Cucumber last week. In this talk, I go over the basics of Cucumber scenario structure and integration and show examples of using it with Ruby, .NET and Java applications. (scroll down to see the slides and links from the presentation) If you’re interested Cucumber or Behaviour-Driven development, also check out my new Introduction to BDD and Hands-on BDD with Cucumber workshops.

Finally, here are the links I mentioned during the talk.

• Main Cucumber web site
• Richard Lawrence’s blog (.NET Cuke4Nuke integration)
• Cuke4Duke (Java integration)
• Visual Studio plugin for Cuke4Nuke
• My BDD in .NET with Cucumber article series: part 1, part 2, part 3
• Introduction to BDD and BDD with Cucumber workshops

I will run a three day hands-on workshop on agile acceptance testing and specification by example in Berlin, Germany on April 19-21.

This three day workshop immerses the participants into a project driven by Specification by Example and Agile Acceptance Testing. Through facilitated exercises and discussion, you will learn how to bridge the communication gap between stakeholders and implementation teams, build quality into software from the start, design, develop and deliver systems fit for purpose.

This workshop is aimed at testers, business analysts and developers. It combines an introduction to Specification by Example and Agile Acceptance Testing, a set of exercises to help you get started with FitNesse – the most popular tool for agile acceptance testing – and a full day of working on realistic domain examples taken from your recent projects or a future phases of projects. This ensures that you gain real-world experience, enabling you to kick-start internal adoption of these practices in your team.

Click here for more information and to register.

During the last two years, I was involved in several projects deployed on the Amazon cloud. Being a relatively early adopter was a fantastic experience that provided lots of opportunities to burn my fingers and learn from mistakes. It also seriously challenged my view of scalable software architectures. I spoke about key lessons learned at CloudCamp London last week – here is the summary of that presentation.

Before I start, I’d like to point out that judging from this post it might seem that I have a negative view of cloud deployments, but nothing could be further from the truth. I have many nice things to say about the cloud, but lots of other presenters at CloudCamp do that all the time. I wanted to play the devil’s advocate a bit and expose some of the things that you won’t necessarily find in marketing materials.

First fundamental rule of cloud deployment: No single machine on the cloud is going to be any more reliable than any other machine there

Before the cloud, I was used to investing more in machines which were more important. Database boxes would have better power supplies than web servers, ideally redundant. Content servers got better disks and lots of them. A nice Cisco appliance would balance requests to web servers, and was infinitely more reliable than them. Web servers, for all I cared, could crash and burn at any time, as long as they did not all decide to do it at the same time. With the cloud, this isn’t possible. No matter how many virtual cores or memory you rent, all the boxes are running on very similar hardware. Or, putting it in another way:

Cynical version of the first rule: All your cloud boxes are equally unreliable

A very healthy way to look at this is that all your cloud applications will run on a bunch of cheap web servers. It’s healthy because planning for that in advance will help you keep your mental health when glitches occur, and it will also force you to design for machine failure upfront making the system more resilient.

A few months ago, one of the load balancing appliance providers tried to pitch their software for the cloud. Choose a box to act as a load balancer, install their software, and hey presto you have the equivalent of a balancer appliance in the cloud. Yeah, right. That machine is as probable to go down as any web server, and when it does your entire server cluster will be cut off from the Internet.

Second fundamental rule of cloud deployments: All machines will be affected by the same networking and IO constraints

Amazon lets people get pretty big virtual boxes in terms of processor or memory power. However, IO and networking are a completely different issue. For in-house data grid deployments, getting a separate set of network cards and putting them on a dedicated VLAN or even their own switch is a really good idea, because of the broadcast traffic between the nodes. You can’t do that on the cloud. Putting a card with hardware TCP offloading is not an option (and broadcast is also not an option at least on Amazon, but that’s another story). So the architecture has to work around this. Bottlenecks can’t be just solved by getting better hardware. Beware of this while designing an architecture that depends on all traffic going through a single file server, database machine or load balancer. If all the traffic goes through a single point, the entire capacity of the cluster will be limited to that machine’s IO or network constraints (which is probably shared with who knows how many other virtual machines on the same physical box).

Third fundamental rule of cloud deployment: networking is unreliable

When I started using the cloud, it took me a while to fully grasp the fact that it’s a really really big data centre. Even if you get only two machines there, they still run in a huge data centre, can be separated with lots of cable and switches and might not even be in the same building. That introduces a new level of complexity that I simply did not have to deal with earlier. It’s not often that I had problems with the network in the cloud, but strange things do happen. I watched my grid control machine disappear from the network and come back in ten minutes without any idea what happened. A few months ago I was completely puzzled by the fact that a web server can’t talk to an application server, although I was connected to both of them at that same time. They were working fine, but they simply did not see each other for a while.

Murphy’s law guarantees that these issues, although rare, will occur at the most inconvenient moment. So choose infrastructure and clustering software wisely. I used an opensource messaging system that worked perfectly inhouse, but the cluster simply couldn’t recover from network glitches in the cloud. Put in lots of monitoring scripts to check if machines can see each other, and plan for parts of the cluster being inaccessible for short periods of time.

There is no fast shared storage

All these things together contribute to the last big challenge I had to get my head around. There simply is no equivalent of a nice network attached shared storage on Amazon. If you’re planning to use a hot-standby database by sharing data files, think again. Elastic Block Clouds can only be mounted to a single machine. You can get a machine and expose an elastic block using NFS, but that falls under the effects of all three rules explained above. SimpleDB is slow. PersistentFS allows you to mount S3 as a file system to multiple machines, but read-only. There is no good solution for this, apart from designing around it.

How to keep your sanity

It took me a while to understand that just deploying the same old applications in the way I was used to isn’t going to work that well on the cloud. To get the most out of cloud deployments, applications have to be designed up-front for massive networks and running on cheap unstable web boxes. But I think that is actually a good thing. Designing to work around those constraints makes applications much better – faster, easier to scale, cheaper to operate. Asynchronous persistence can significantly improve performance but I never thought about that before deploying to the cloud and running into IO issues. Data partitioning and replication make applications scale better and work faster. Sections of the system that can work even if they can’t see other sections help provide a better service to customers. This also makes the systems easier to deploy, because you can do one section at a time.

To conclude, there are three key ideas to keep in mind:

• Partition, partition, partition: avoid funnels or single points of failure. Remember that all you have is a bunch of cheap web servers with poor IO. This will prevent bottlenecks and scoring an own-goal by designing a denial of service attack in the system yourself.
• Plan on resources not being there for short periods of time. Break the system apart into pieces that work together, but can keep working in isolation at least for several minutes. This will help make the system resilient to networking issues and help with deployment.
• Plan on any machine going down at any time. Build in mechanisms for automated recovery and reconfiguration of the cluster. We accept failure in hardware as a fact of life – that’s why people buy database servers with redundant disks and power supplies, and buy them in pairs. Designing applications for cloud deployment simply makes us accept this as a fact with software as well.

I recently came across two books that fit the Building software that matters theme perfectly, and deserve to be read by anyone managing a software project, running a development team or generally serious about delivering software. Both books tackle topics so difficult that development teams often just push the responsibility for them to the customer, expecting some kind of magical resolution.

Developers prefer to talk in terms of value generation without developing related revenue projections. Business stakeholders usually don’t realize that delivery sequences and architectural options have a significant impact on project level ROI. Clearly, software development requires a more financially responsible approach […]. This situation clearly calls for a close collaboration between developers and business stakeholders.

The previous paragraph pretty much sums up the basic idea of Software by Numbers: Low-Risk, High-Return Development by Mark Denne and Jane Cleland-Huang. Iterative delivery and fast roll-out to production makes perfect sense and I do not think that anyone can doubt that, but can you actually put a value on it? Can you compare two alternative delivery plans for the same project? Denne and Cleland-Huang set to do exactly that. By developing a financial model to evaluate and compare delivery plans, the authors show how to make an informed decision about prioritisation on a project so that it maximises the desired financial effect, either reducing risk, optimising cash-flow or return on investment.

Denne and Cleland-Huang explain how to identify and evaluate the financial impact of Minimum Marketable Features, including “intanglibles” such as brand value or customer retention. Then they develop a system of heuristics that combines this information with technical and architectural constraints, prerequisites and timing constraints to come up with the best possible delivery plan, depending on what you want to achieve. Through several case studies, they show how just choosing the next most important thing isn’t as nearly as effective as considering the financial impact of the entire delivery plan. Somewhere in the middle, they also come up with a way to estimate the financial impact of the project by forces other than black magic which helps us answer a key question: should we do this at all or should we do something else?

Effect Managing It, by Mijo Balic and Ingrid Ottersten, deals with things that happen even earlier in the process – coming up with a list of features to achieve the desired business effect. Instead of focusing on functionality, Balic and Ottersten look at software projects as agents of business transformation. The authors argue that the ultimate goal of software projects is to achieve a business change, and that one of the key reasons for failure of so many projects is that the focus is often on the wrong things. They explain a model of structuring requirements analysis in a way that ensures achieving the desired business effect. That focus on business impact, not functionality or technology, is what they call “Effect Managing”.

Effect Managing shares many key ideas with behaviour-driven development and agile acceptance testing, but deals with things that are much earlier in the development cycle. Balic and Ottersten base their model on first considering who can deliver the desired change (stakeholders in the BDD lingo), then thinking about how these people can deliver it and what the software needs to do to support that that. Organising all these ideas in a hierarchy which they call an “Effect Map”, this model provides visibility similar to the Goals-Features-Requirements model, but in my opinion much more effective because it is visual. After trying out effect maps on two projects, I’m astonished how they open up a discussion and provide structure to the process of selecting the features for a project and probably equally important throwing away features that are less important.

Be warned, though, that both books are relatively hard to read. Although they are short, less than 200 pages each, the flow of ideas in both books is a bit hard to follow. Effect Managing IT at times reads like a literal translation (from what I understand, the original book was written in Swedish) and Software by Numbers is full of financial tables and formulas. Nevertheless, I think that both books have such great value that getting through them is well worth it.

Next agile testing user group evening in London will be on the 4th February, at the new Skills Matter offices.

I’ll be presenting Cucumber a tool for behaviour-driven development and agile acceptance testing. I will demonstrate how to use Cucumber for Java, .NET and Ruby applications, talk about new Cucumber features and best practices for writing and maintaining Cucumber scenarios.

The event is free, but up-front registration is required for capacity planning. For more information and to register click here

I got this question from a reader today:

How would you effectively define a sufficient set of If-When-Then scenarios to test for correctness what is potentially an extremely large set of transformations?

BDD scenarios (and agile acceptance tests in general) are not about full regression testing – they work best as a set of examples that demonstrate business rules which need to be developed, and as the software matures become live documentation of what the system does. So they typically should contain:

• A representative example demonstrating each important aspect of a business rule. Business users, analysts or customers will typically lead defining these.
• An example demonstrating each important technical edge case (technical boundary conditions) developers will typically suggest cases here. Business users, analysts or customers will define the correct expected behaviour.
• An example demonstrating each particularly troublesome area of the expected implementation (such as cases that caused bugs in the past, boundary conditions that might not be explicitly demonstrated by previous examples etc). Testers will typically suggest these and business users, analysts or customers will define the correct behaviour.

I would start by asking business users to define important examples, then developers and testers to comment on these examples and suggest important conditions and cases not covered by existing examples. I would do this in a specification workshop to allow everyone to benefit from a discussion and build a shared understanding of the problem. At the point where everyone is happy that they have enough information to work, we have enough examples. Then I would capture the examples discussed during the workshop, identify potential duplicates, remove information not directly related to the business rules for the cases and formalise them as acceptance tests/BDD scenarios.

This is not to say that additional examples cannot be added later to provide a more complete functional regression check. This can even be automated using the same scenario step format and automation tools and helpers as the other examples. But I like to keep the two separate so that there is a small and focused specification that can be easily read and understood later, when things change. When using tools such as FitNesse, you can link the two sets of examples with an “For Additional Examples, See …” link to point people to a full regression set later. Having them separate also allows you to quickly run the entire specification and get faster feedback.

In part 2 of this tutorial we used tables to organise scenario information efficiently. This works fine for processing lists of objects, but it requires additional code to work with Cucumber tables and it isn’t the best solution when things go wrong. In this part, I’ll show you how to save even more time and effort when working with repetitive scenarios.

What’s wrong with plain tables?

Try this yourself: modify the expected scenario from the previous part to make it fail and re-run Cuke4Nuke. You’ll see that there’s something wrong, but what exactly? “Tables don’t match” isn’t the most helpful error message, especially when you’re dealing with a long list of objects.

Using scenario outlines

Luckily, Cucumber allows us to specify a scenario template and then a list of parameters for the template in a table. Instead of Scenario, we use the Scenario outline header. Parameters in the template should be enclosed in in the less-than < and greater-than > symbols. After the scenario, we list the examples in the table under the Examples header.

	Scenario Outline: Hello World (with examples)
		Given The Action is <action>
		When The Subject is <subject>
		Then The Greeting is <greeting>.

		Examples:
		|action|subject|greeting|
		|Hello|Mike|Hello, Mike|
		|Jump|Tom|Jump, Tom|
		|Shout|Jim|Shout, Jim|

Cucumber will execute the scenario once for each row of the Examples table. This is effectively the same as specifying the same scenario three times with different data, but a lot less verbose and easier to understand. As we’re using the same scenario steps as in part 1 of this tutorial, there is no new C# code and we can just run the whole thing again.

Try to break it again – modify one scenario example to fail. You’ll see a much better error report than with table arguments. The NUnit exception stack trace isn’t something I’d especially like to show to a business user, but Cuke4Nuke points us to the exact row and describes the failure in detail:

Scenario outlines allow us to re-run the same scenario steps several times for different parameter values, without special code to handle tables and with better error reporting. For further examples, see Richard Lawrence’s post on removing duplication using scenario outlines.

You can get the source code for all the examples mentioned in this tutorial from GitHub Cuke4NukeExample repository.

If you’re interested in learning more about Behaviour-Driven Development and Cucumber, check out my new BDD workshops.

The BDD Given/When/Then scenario structure is great for workflows, but too verbose when a specification includes lots of similar cases or when complicated objects come into play. This is where Cucumber tables come in. Tables provide just enough structure to group related data together so that it can be efficiently manipulated and easy to understand in a scenario. After setting up Cucumber for .NET and going through a basic example, it’s time to learn how to use tables to make complex scenarios easier to read and manage.

Handling lists of objects

In addition to matching regular expressions, Cucumber steps can also receive tables. Cucumber uses the standard wiki syntax for tables, with cells separated by the pipe symbol. Instead of a single greeting which we’ve described in part 1 of this tutorial, let’s create a scenario for a group of greetings. Add this to your feature file:

	Scenario: Hello World with tables
		Given the mailing list is
			|action|subject|
			|Hello|Mike|
			|Jump|Tom|
		When the list is processed
		Then the following messages have gone out
			|message|
			|Hello, Mike|
			|Jump, Tom|

The table API

Running Cuke4Nuke without the -q flag will tell us which missing methods it expected to find (in the You can implement step definitions for undefined steps with these snippets section):

[Pending]
[Given(@"^the mailing list is$")]
public void TheMailingListIs(Table table)
{
}

[Pending]
[When(@"^the list is processed$")]
public void TheListIsProcessed()
{
}

[Pending]
[Then(@"^the following messages have gone out$")]
public void TheFollowingMessagesHaveGoneOut(Table table)
{
}

As you can see from the snippets that Cuke4Nuke generated, tables in scenarios are passed as Cuke4Nuke.Framework.Table objects. This class has three interesting members:

• The first is is the Data field which is a list of lists of strings. Effectively, this represents the table, row by row, including the header. The first table from the scenario example would be represented by the following list: [ ["action","subject"], ["Hello","Mike"],["Jump","Tom"] ]
• The second is the Hashes() method which transforms the data table into a list of Dictionary objects, creating key-value pairs from data rows (below the header) by reading keys from the header row. The first table from the scenario example would be represented by the following Dictionary: [ ["action"=>"Hello","subject"=>"Mike"], ["action"=>"Jump","subject"=>"Tom"] ]. The Hashes() method makes it a bit easier to process raw table data.
• The third is the AssertSameAs(Table) method which is a shorthand for comparing two tables. This will come in handy for checking data in batches (eg in the TheFollowingMessagesHaveGoneOut method)

Copy and paste the snippets generated by Cuke4Nuke into your step definition class, remove the Pending attributes, and let’s add some meat to it. Again, on a real project the steps would talk to our domain classes but let’s keep it simple now to demonstrate Cucumber features. The TheMailingListIs method will convert the input table to a list of our domain Message objects, TheListIsProcessed will generate string contents for each message and store it in an outgoing list, and the TheFollowingMessagesHaveGoneOut method will create a Cucumber table from that list and compare it to the list of expected values, defined in the scenario. Notice how we use the Hashes() method to process an input table and the Data property to create an output table.

class Message
{
    public String Action { get; set;}
    public String Subject {get; set; }
}

List messages = new List();

[Given(@"^the mailing list is$")]
public void TheMailingListIs(Table table)
{
    foreach (Dictionary element in table.Hashes())
    {
        messages.Add(
          new Message { Action = element["action"],
                               Subject = element["subject"] });
    }
}
List outgoing = new List();
[When(@"^the list is processed$")]
public void TheListIsProcessed()
{
    foreach (Message m in messages)
    {
        outgoing.Add(m.Action + ", " + m.Subject);
    }
}

private List list(String s)
{
    List l = new List();
    l.Add(s);
    return l;
}

[Then(@"^the following messages have gone out$")]
public void TheFollowingMessagesHaveGoneOut(Table table)
{
    Table expected = new Table();
    expected.Data.Add(list("message"));
    foreach (String s in outgoing)
    {
        expected.Data.Add(list(s));
    }
    table.AssertSameAs(expected);
}

Now you re-run Cuke4Nuke and should see the tables in a green colour, meaning that the tests passed:

There it is. That’s all you need to know about using tables with Cucumber and Cuke4Nuke. Rinse and repeat until happy. If you’re interested in learning more about Behaviour-Driven Development and Cucumber, check out my new workshops.

• BDD in .NET with Cucumber part 3: Scenario outlines and tabular templates
• Download source code

At the Oresund Developer conference in Sweden about two months ago, Aslak Hellesoy talked about recent changes to his Cucumber Behaviour-Driven Development tool, aimed at providing better support for platforms other than Ruby. Instead of using the tool through slow Ruby ports, .NET and Java developers got a chance to benefit from much quicker native integrations using a new wire protocol that allows Cucumber to talk to external systems. Richard Lawrence and Matt Wynne were very kind to work on a native .NET integration for Cucumber, called Cuke4Nuke, which recently got up to speed with Cucumber Ruby features. In this article, I’ll show you how to set up and use Cuke4Nuke to use Cucumber for .NET BDD development.

Basic Software

• If you don’t already have Ruby installed, install it using the new one-click Ruby Installer. You don’t have to know Ruby to write Cuke4Nuke tests, but you still need it to run Cucumber. Install the Ruby development kit as well. Alternatively use the old 1.86 installer which comes with the development kit.
• Add GemCutter.org to the list of Ruby Gem sources. Execute the following command from the command line:

  gem sources -a http://gemcutter.org/

• Install the Cuke4Nuke gem. Execute the following command from the command line:

  gem install cuke4nuke

• Install the Win32Console gem to get colour reports in the windows console window:

  gem install win32console

  You might also need WAC to work around console colour problems so download that Exe and put it somewhere on the disk.

• If you don’t already have NUnit, grab the latest version from NUnit.org. We’ll need this later as Cuke4Nuke uses NUnit assertions.

To verify that the system is set up, execute Cuke4Nuke from the command line. You should see a help screen with Cuke4Nuke options. If you see an error that the command Cuke4Nuke isn’t found, check if there is a Cuke4Nuke.bat executable in your Ruby bin folder (C:\Ruby\bin if you accepted the default installation options for Ruby), whether that folder is on your executable path and whether Cuke4Nuke gem exists in the gems folder (C:\Ruby\lib\ruby\gems\1.8\gems\cuke4nuke-0.3.0 on my system).

Project setup

Create a normal C# class library project and add a Features folder to it. This is where your Cucumber scenarios will go. Add a step_definitions subfolder to that and create a file called cucumber.wire, with the following content:

host: localhost
port: 3901

This tells Cucumber to connect to an external system on port 3901, the default port for Cuke4Nuke. Now let’s add a test scenario to make sure that everything is working correctly. Create a file called basic.feature in the Features folder, with the following content:

Feature: Hello World
	In order to ensure that my installation works
	As a Developer
	I want to run a quick Cuke4Nuke test

	Scenario: Hello World
		Given The Action is Hello
		When The Subject is World
		Then The Greeting is Hello, World.

At this point, your project structure should look as on the picture below:

Now let’s run Cuke4Nuke for the first time. Build the project, open the console, go to your project folder and run the following command:

Cuke4Nuke bin\debug\Cuke4NukeExample.dll -q

(replace the DLL path with your project output DLL). You should see the Cucumber output telling you that there is one scenario with three steps, all of which are undefined. If you do not see the colours, add the -c flag and pipe the output to WAC.exe, making the command similar to the following:

Cuke4Nuke bin\debug\Cuke4NukeExample.dll -q -c | d:\apps\wag.exe

In any case, the command result should be similar to one on the picture below:

Now let’s add the step definitions that help Cucumber talk to our project code. Create a C# class file (say Steps.cs) and paste the following code:

using System;
using System.Text;
using Cuke4Nuke.Framework;
using NUnit.Framework;
namespace Cuke4NukeExample
{
    public class HelloWorldSteps{
        private String action;
        private String subject;
        [Given("^The Action is ([A-z]*)$")]
        public void ActionIs(String action)
        {
            this.action = action;
        }
        [When("^The Subject is ([A-z]*)$")]
        public void SubjectIs(String subject)
        {
            this.subject = subject;
        }
        [Then(@"The Greeting is ([^\.]*).")]
        public void CheckGreeting(String greeting)
        {
            Assert.AreEqual(greeting, action + ", "+subject) ;
        }
     }
}

Add a reference to NUnit.Framework.dll (from your NUnit installation) and Cuke4Nuke.Framework.dll (you’ll find it in C:\Ruby\lib\ruby\gems\1.8\gems\cuke4nuke-0.3.0\dotnet\Cuke4Nuke.Framework.dll). Now build the project again, go to the console and re-run the Cuke4Nuke command. The output should now say that the steps passed.

When it runs the feature specification, Cuke4Nuke will look for methods marked with Given, When and Then attributes that match steps by regular expression. Any capture groups in the regular expression are passed to the method as arguments (they don’t have to be Strings, you can use other .NET types as well). So, for example, the line “When The Subject is World” matches the following method:

[When("^The Subject is ([A-z]*)$")]
public void SubjectIs(String subject)
{
     this.subject = subject;
}

Our simple script will set the subject and the action and then verify the greeting using the standard NUnit assertion Assert.AreEqual. Just to see it when it fails, modify the CheckGreeting method or feature source so that they don’t match and re-run Cuke4Nuke.

Of course, for a proper project this code would talk to our domain classes but let’s keep it simple.

Build integration

We can add Cuke4Nuke as a post-build step so that all the specifications get executed after every build. Open your project .csproj file and add this just above the closing tag:

<PropertyGroup>
<PostBuildEvent>cuke4nuke $(TargetPath) $(ProjectDir)features -q
 </PostBuildEvent>
</PropertyGroup>

You can now inspect the results of Cuke4Nuke in your output window every time a project is built.

Continuous Integration

To complete the project setup, let’s run Cucumber tests within a continuous integration environment and store test outputs next to project build results. I use TeamCity even for .NET projects, so I’ll use it in this tutorial as well. (Setting up TeamCity is outside the scope of this tutorial, but it is fairly easy to do. Grab it from jetbrains.com). TeamCity 5 should support Cucumber out of the box but I haven’t been able to make it work. Cucumber/ANT docs for TeamCity suggests that a few environment variables should do the trick for ANT/Java builds but this doesn’t work for Cuke4Nuke. The Cucumber Teamcity template uses a deprecated API and fails to build with the latest Cucumber version, and the only other blog post I found on TeamCity and Cucumber uses the same (broken) API. However, Cucumber can export a JUnit XML test report file, which is more than enough to get us nicely integrated with TeamCity (even with the previous versions). We’ll make Cucumber save the test results into the test subfolder of our project folder, and tell TeamCity to monitor that. First, let’s change the .csproj project file. Modify the PropertyGroup block you just added to the following:

  <PropertyGroup>
    <PostBuildEvent>cuke4nuke $(TargetPath) $(ProjectDir)features -q
     -f junit -o$(ProjectDir)test</PostBuildEvent>
    <PreBuildEvent>del $(ProjectDir)test\*.xml</PreBuildEvent>
  </PropertyGroup>

This will save test output in the JUnit format (-f junit) into the test subfolder of our project directory (-o$(ProjectDir)test). It will also delete any previous test results from that folder when the build starts. Now set up your TeamCity project as normal. On the Build Runner configuration screen, select “ANT Junit” in the “XML Report Processing” section and set the import path as test/*xml (if you are working without a version control system, this will probably have to be a full path to your project folder rather than a relative path):

Everything should now be set up. You should be able to run the TeamCity build and get the “Tests passed:1″ message in TeamCity.

That’s it. Now continue writing feature specifications and implementing them in nice little iterations. Good luck! If you’re interested in this topic, you might also be interested in my new BDD with Cucumber workshops.

• BDD in .NET with Cucumber part 2: Making scenarios easier to read with tables
• BDD in .NET with Cucumber part 3: Scenario outlines and tabular templates
• Download source code

I’ve reimplemented the JUnit runner of Trinidad to talk to the latest FitNesse code (and use -c command execution path). The Maven plugin is also working. I’d appreciate if people could test this and see if it still works for them as the old trinidad support was dropped from the FitNesse trunk and will not be supported in the next release.

The beta build of fitnesse trunk can be downloaded using maven from http://maven.neuri.com repository. Use

 <dependency>
  <groupId>org.fitnesse</groupId>
  <artifactId>fitnesse</artifactId>
  <version>20091221-SNAPSHOT</version>
 </dependency>

Or alternatively download the binary jar directly.

The API is almost the same but some arguments are now obsolete, such as the test engine type. Because tests are executed using FitNesse -c directly, whatever is the selected engine type for the page will be used.

See JUnit Helper examples and FitNesseSuite runner examples with nice reporting for graphical JUnit tools

The Maven plugin also works, grab it from the same maven repository and use version 20091221-SNAPSHOT. Engine type parameter is obsolete and removed. See an example of how to use the plugin or how to directly invoke the target.

At the Agile Specification, BDD and Testing Exchange last month in London, Dan North spoke about selling behaviour driven development to the business. He put forward a relatively controversial idea on estimation: treat all stories as equal. Estimate each story as size 1 and just be done with it. Although this idea might seem overly simplistic, I find it really appealing.

Story estimation is one of those things where the process of doing it is more important than the output. Planning poker is a very effective technique to discover differences in understanding. However, BDD achieves the same by getting an agreement on scenarios. Specification workshops go much further than planning poker in building a shared understanding of the tasks at hand and definition of done. Teams that practice BDD or agile acceptance testing could skip planning poker if only there was some other effective way to get a sensible feel of how long a project phase will take. That’s why I find Dan’s idea so appealing.

On a decent-size project, the estimates will average out anyway. Treating all stories as medium size and figuring out how long a medium story will take will not be any less accurate than doing more detailed estimates and adding them up. It will, however, save quite a lot of time.

As a nice side effect, this should also produce a better breakdown of stories. Any story that obviously stands out by complexity will have to be broken down into smaller pieces, as we are not allowed to give it a higher valuation. Evenly-sized stories make it easier to establish a constant pace of delivery.

I got this question today from a blog reader:

They are thinking about using Concordion here. I remember you made some comments about using it. Can it be used sensibly by a tester without lots of dev experience?

Concordion is a great tool for acceptance testing as support for development. Whether it can be used sensibly by a tester without lots of development experience, that depends on what the intended use is. Concordion tests/results are HTML files, so anyone can read them using a browser. I’ve never tried to write Concordion tests using Word or anything similar, only by hand-coding HTML, so I don’t know whether tests can be maintained with a visual tool. However, anyone with basic HTML knowledge should be able to write tests as well. In terms of running the tests, Concordion runs within JUnit/NUnit, so this should be fairly simple as well.

Concordion does not have a test management tool and intentionally doesn’t allow you to reuse or share automation components, so it requires a fair bit of cooperation between developers and testers. I think this is very good for tests that support development, but it might be a problem for retro-fitting regression test packs into an existing product if testers are expected to do the bulk of work.

I wrote a review of concordion last year, which will give you a bit more detail on how it works. I also have a ">short video about it (.NET oriented but still a good introduction to capabilities):

David Peterson, the author of Concordion, spoke about it at the agile acceptance testing tools round-up event, and that ">video is also online.

I just packaged and released Trinidad 20091121 to maven repository maven.neuri.com. this version is compiled and linked against the latest stable Fitnesse version 20091121 (which is also deployed to the maven repository with the correct version number). Trinidad is an in-memory test runner for FitNesse (Java) tests which enables you to do some nice things such as run FitNesse tests from JUnit for easier debugging or from Maven for better build system integration, without requiring a running FitNesse server.

A question for you

Here’s a question for the community.

From the next version of FitNesse, all non slim stuff will apparently be decoupled from the core. this includes FIT, FitLibrary and the part of Trinidad that was bundled with FitNesse in several most recent releases. Also, FitNesse got -c option in the most recent release making a large part of Trinidad obsolete. This means that we have two options for the future of trinidad:

1) drop the testing infrastructure. keep JUnit runners and Maven runners but link them directly with FitNesse -c option

2) keep the testing infrastructure in place, maintaining Trinidad as a separate project from FitNesse

option 1 guarantees better compatibility with upcoming fitnesse features and less inconsistencies (eg no potential for trinidad to interpret things differently, skip suite pages etc which happened in the past). However, dropping the testing infrastructure means that html reports generated by Trinidad will no longer be available and that the FitNesse JUnit Test Suite which nicely reports test names in the JUnit window will also not be available.

potentially – if Uncle Bob is OK with this, we could refactor or hack the way FitNesse stores results to enable this kind of reporting to continue.

option 2 means potential inconsistencies and problems going forward as I’d have to sync trinidad with fitnesse code after every release, similar to the early releases which were, honestly, problematic. Doing this properly would probably mean treating fitnesse as a third party system and doing lots of integration testing around it.

which would you prefer?

I’m launching two new Behaviour-Driven Development workshops in late January.

Introduction to BDD is an intensive one day workshop which introduces behaviour-driven development to developers, business analysts and testers. The optional programming module is offered in Java or .NET, with Cucumber to automate BDD scenarios.

Hands-on BDD with Cucumber is a three day workshop which immerses the participants into a project driven by Specification by Example and Behaviour-Driven Development. The workshop is adjusted to fit your business domain and particular needs, so that the participants get real-world experience and instant benefits. It is combines the Introduction to BDD, a day of working on a realistic domain example taken from your recent project or a future phase of a project, and a day of programming exercises for test automation developers. During the workshop, we use Cucumber to manage BDD Scenarios. The programming modules can be either in Java or .NET.

These two workshops are offered at a 20% discount for all bookings made before 31st January 2010. For more information on these and my other workshops, check out the training page.

Both workshops are offered as on-site training only (there are no public workshops scheduled at this time). Contact my company, Neuri Ltd for more information and to book a workshop.

Here’s a video of my presentation on challenging requirements at the Agile Testing, Specifications and BDD exchange last month. I talk about problems with requirements gathering processes established in the industry and offer ideas to improve that.