BDD with Cucumber (SpecFlow)

Shouty is a social network that allows people who are physically close to communicate, just like people have always communicated with their voices. In the real world you can talk to someone in the same room, or across the street. Or even 100 m away from you in a park - if you shout.

That’s Shouty. What you say on this social network can only be “heard” by people who are nearby.

Let’s start with a very basic example of Shouty’s behaviour. Something we might have discussed in a three amigos meeting:

Sean the shouter shouts "free bagels at Sean’s" and Lucy the listener who happens to be stood across the street from his store, 15 metres away, hears him. She walks into Sean’s Coffee and takes advantage of the offer.

We can translate this into a Gherkin scenario so that Cucumber can run it. Here’s how that would look.

You can see there are four special keywords being used here. Scenario just tells Cucumber we’re about to describe an example that it can execute. Then you see the lines beginning with Given, When and Then.

Given is the context for the scenario. We’re putting the system into a specific state, ready for the scenario to unfold.

When is an action. Something that happens to the system that will cause something else to happen: an outcome.

Then is the outcome. It’s the behaviour we expect from the system when this action happens in this context.

You’ll notice we’ve omitted from our outcome anything about Lucy walking into Sean’s store and making a purchase. Remember, Gherkin is supposed to describe the behaviour of the system, so it would be distracting to have it in our scenario.

Each scenario has these three ingredients: a context, an action, and one or more outcomes.

Together, they describe one single aspect of the behaviour of the system. An example.

Now that we’ve translated our example into Gherkin, we can automate it!

Hello! I’m Gaspar Nagy, the creator of SpecFlow and a BDD trainer nowadays. I will guide you through the SpecFlow automation topics in Cucumber School. First, let’s install SpecFlow!

SpecFlow is an open-source tool that is available as a NuGet package that you need to configure for your project. Although SpecFlow works fine even without Visual Studio, in Cucumber School we are going to use Visual Studio 2019, because it integrates nicely with SpecFlow. If you don’t have Visual Studio, you can download the Visual Studio Community edition that is free for education purposes and small teams.

In order to use the Visual Studio integration of SpecFlow, you need to install a Visual Studio extension. This is something you need to do only once. The Visual Studio extensions can be managed by opening the Manage Extensions command from the Extensions menu.

There are plenty of useful extensions in the Visual Studio marketplace and for SpecFlow there are two that you can choose from. You will find these if you type SpecFlow into the search box. Both the SpecFlow for Visual Studio 2019 and the Deveroom for SpecFlow extensions work well with SpecFlow and both of them are free and open-source. In this course I will use the Deveroom extension, but you can follow the exercises with the other one as well.

In order to install the extension you just need to click on the Download button next to the name of the extension you selected. The extension is downloaded, but it only gets installed once you close your Visual Studio. So we need to close all instances of Visual Studio 2019 and wait for the install dialog to pop up.

Here it is. Accept the installation of the extension by clicking on the Modify button , that completes the setup process. Our Visual Studio is now ready to work with SpecFlow. So are we.

Now we are going to create a Visual Studio solution for the Shouty application. As we will focus on the business logic of the application in this course, I create a .NET Standard class library project for the production code. I also remove the class that comes with the template.

We also need to add a project for the scenarios and the automation code. SpecFlow works with test execution frameworks in order to make the scenarios executable. It supports all well known test execution frameworks like MsTest, NUnit or xUnit. There is also a free dedicated runner developed by Tricentis called SpecFlow+ Runner. For the sake of simplicity in this course we are going to use xUnit, so I add a .NET Core xUnit Test Project to my solution.

I call our test project Shouty.Specs . Including Specs in the project name emphasizes that we are creating an executable specification. As we won’t have coded unit test in this project I remove the UnitTest1.cs file added by the template.

To make this project a SpecFlow project, we need to add two NuGet package references.

The first is SpecFlow.xUnit. This is going to install SpecFlow for the project and configure it to work with xUnit.

At the time of the recording this leads to an error as the xUnit version used by the xUnit Test Project template is not recent enough for the latest SpecFlow version. This is something we can easily fix by updating the xUnit related packages of the specs project. In fact we can upgrade all packages in this case.

Let’s retry. Installing the SpecFlow.xUnit package is now successful.

The second package we need to add is the SpecFlow.Tools.MsBuild.Generation package. This will instruct SpecFlow to turn our scenarios into executable tests every time we build the project.

As we automate the scenarios, we will need to create class instances and call methods from the application project. To make this possible we need to add a reference to the SpecFlow project pointing to the Shouty project.

Let’s have a quick look at the project file of the Specs project. If we did everything well, our project file should look like this. As this is a .NET Core project, we could actually achieve the same outcome just by adding these lines to the project file manually. That probably would have been easier. Maybe next time.

The SpecFlow project will contain the feature file, the automation code, and some other files necessary for the automation infrastructure. Adding all these into the root folder of the project would be quite messy. Teams that work with SpecFlow usually follow some conventions in order to structure their SpecFlow projects. If you have used any Cucumber-family tools before, these conventions will be familiar for you.

To achieve that, let’s create three folders. One, called Features where we will store…​ well the feature files, I guess. The second folder that we usually have is called StepDefinitions. This will be the container for our automation code. And finally we also create a third folder called Support where we can store any files related to the supporting infrastructure.

Nice!

Let’s verify out setup by building the solution.

The build succeeded so now we’re ready to create our first feature file.

In this course we are going to use Cucumber Expressions that are explained in detail in Chapter 3. In SpecFlow version 3.1 that we use Cucumber Expressions are not supported by default. You can enable this feature by adding the CucumberExpressions.SpecFlow.3-1 NuGet package to the project, like you can see in the project file. We also specified the exact SpecFlow package version explicitly to avoid the version compatibility warning. With later SpecFlow versions, these additions won’t be needed.

Let’s create our first feature file using the "Add / New Item…​" command. Call the file HearShout.feature

All feature files start with the keyword Feature: followed by a name. It’s a good convention to give it a name that matches the file name.

Now let’s write out our first scenario.

This is the one where the listener is within range. Given Lucy is located 15 metres for Sean, When Sean shouts "free bagels at Sean’s", , Then Lucy hears Sean’s message.

Build the solution.

As you can see our new scenario has appeared in the "Test Explorer" window.

If you don’t have a "Test Explorer" in your Visual Studio, you can open it using the "Test Explorer" command from the "Test" menu.

Run the tests.

The test execution failed with an error message that says "No matching step definition found for one or more steps". This is because some of our steps are undefined.

Undefined steps are also higlighted in Visual Studio with an orange color.

Undefined means SpecFlow doesn’t know what to do for any of the three steps we wrote in our Gherkin scenario. It needs us to provide some step definitions.

Step definitions translate from the plain language you use in Gherkin into C# code. We write a C# method, then annotate it with a pattern using the Given, When or Then attributes provided by SpecFlow.

When SpecFlow runs a step, it looks for a step definition that matches the text in the Gherkin step. If it finds one, then it executes the code in the step definition.

If it doesn’t find one… well, you’ve just seen what happens. SpecFlow prints out some code snippets that we can use as a basis for new step definitions, but you can get the same snippets from Visual Studio, using the "Define steps…​" command in the editor.

The dialog can help us to create a new class and paste the selected snippets. We’ll just call the class StepDefinitions.

The wizard detected that we had a StepDefinitions folder and saved the new class there. SpecFlow would find it anywhere within the same project, but it is better this way.

Now run the tests again.

This time we’ve got another error message. It says "One or more step definitions are not implemented yet".

The step definition snippets we have copied into the project have thrown a PendingStepException that at the end caused this error.

Now our scenario and our steps are pending, meaning that we have to implement the automation code to have a fully automated scenario. You can think of the PendingStepException as a reminder of that.

Now that we’ve wired up our step definitions to the Gherkin steps, it’s almost time to start working on our solution. First though, let’s tidy up the generated code.

We’ll rename the int parameter to something that better reflects its meaning. We’ll call it distance.

We can put a breakpoint into the method and debug the scenario to see what’s happening.

When the breakpoint hits, the value of the distance parameter is 15.

Notice that the number 15 does not appear anywhere in our C# code. The value is automatically passed from the scenario step to the step definition.

If you’re interested, that was caused by the {int} in the step definition pattern or cucumber expression. We’ll explain these patterns in detail in a future lesson.

Now that we have the step definitions matching, we can start working on our solution. We like to use our scenarios to guide our development, so we’ll start designing the objects we’ll need by sketching out some code in our step definitions.

The scenario will be failing while we do this, but we should see the error messages gradually progressing as we drive out the interface to our object model.

Our next goal is for the scenario to fail because we need to implement the actual business logic. Then we can work on changing the business logic inside our objects to make it pass.

To implement the first step, we need to create a couple of Person objects, one for Lucy and one for Sean.

Then we create the Person class into our production project to remove the errors. To make it visible for the SpecFlow project, we need to make it public.

In order to complete the step definition for the Given step, we need to specify the distance between Lucy and Sean.

To keep things simple, we’re going to assume all people are situated on a line: a one-dimensional co-ordinate system. We can always introduce proper geo-locations later. We’ll place Sean in the centre, and Lucy 15 metres away from Sean.

This might not be the design we’ll end up with once this is all working, but it’s a decent place to start.

We can implement our simple distance concept by introducing a MoveTo method like this:

We have two instances of person, one representing Lucy, and one representing Sean. Then we call a method to move Lucy to the position specified in the scenario.

As this seems to be complete like this, we can remove the pending exception.

There is no MoveTo method yet, so Visual Studio reports a compilation error. To fix it, we can create the method on the Person class, but at this stage we don’t bother with the correct implementation. It is enough if it compiles, so an empty method is just fine for now.

When we run the scenario, the first step should be passing! The easiest way to see this is to open the test output by clicking on the "Open additional output for this result" link and check the "Standard Output" section. Here you can see all steps executed by SpecFlow with their results. The first step is "done" and the two others are still pending .

We’re making progress!

We’ll keep working like this until we see the scenario failing for the right reasons.

In the second step definition, we want to tell Sean to shout something.

In order to send instructions to Sean from the "When" step, we need to store him in an instance field, so that he’ll be accessible from all of our step definitions. Let’s move both declarations up to class level together with the initializations.

In the When step, we’re capturing Sean’s message using this pattern that is mapped to the parameter p0. Let’s give it a more meaningful name. Don’t worry if the pattern sounds unfamiliar to you, we will look at that in detail in the next chapter.

And now we can tell Sean to shout the message:

We eliminate the compilation error by declaring the Shout method in the Person class.

When we run the scenario again, the second step is also passing!.

The last step definition is where we implement a check, or assertion. We’ll verify that what Lucy has heard is exactly the same as what Sean shouted.

Once again we’re going to write the code we wish we had. In that we are going to use an assertion from the Xunit library, so we need to add the necessary namespace usages.

So we need a way to ask Lucy what messages she has heard, and we also need to know what it was that Sean shouted.

We can record what Sean shouts by storing it in an instance field as the When step runs. This is a common pattern to use in SpecFlow step definitions when you don’t want to repeat the same test data in different parts of a scenario. Now we can use that in the assertion check.

We also need to add a GetMessagesHeard method to our Person class. Let’s just return null for now.

…​and watch SpecFlow run the tests again.

This is great! Whenever we do BDD, getting to our first failing test is a milestone. Seeing the test fail proves that it is capable of detecting errors in our code!

Never trust an automated test that you haven’t seen fail!

Now all we have to do is write the code to make it do what it’s supposed to.

So we have our failing scenario:

Lucy is expected to hear Sean’s message, but she hasn’t heard anything: we got null back from the GetMessagesHeard method.

In this case, we’re going to cheat. We have a one-line fix that will make this scenario pass, but it’s not a particularly future-proof implementation. Can you guess what it is?

I told you it wasn’t very future proof! But let’s see what SpecFlow says to that.

Fantastic! Our scenario is passing for the first time. As long as this is the only message anyone ever shouts, we’re good to ship this thing! But I’m afraid this is not going to be the case so let’s work a bit more on it.

Now, the fact that such a poor implementation can pass all our tests shows us that we need to work on our tests. A more comprehensive set of tests would guide us towards a better implementation.

It’s a good habit to look for the most simple solution though. We can trust that, as our tests evolve, so will our solution.

Instead of writing a note on our TODO list, let’s write another test that shouts a different message. Usually we’d expect a developer to do this using a unit test, but to keep things simple in this lesson, we’re going to write another scenario.

We’ve worked hard. It’s time for a coffee, so let’s come up with an example that has Sean offering free coffee.

It fails, reminding us we need to find a solution that doesn’t rely on hard-coding the message. Now when we come back to this code, we can just run the tests and SpecFlow will tell us what we need to do next. We’re done for today!

Of course, if you’re in the mood, you can always try to implement a solution yourself that makes both scenarios pass. Have fun!

In the previous chapter we explored the fundamental components of a SpecFlow test suite, and how we use SpecFlow to drive out a solution, test-first.

First we specified the behaviour we wanted, using a Gherkin scenario in a feature file. Then we wrote step definitions to translate the plain english from our scenario into concrete actions in code. Finally, we used the step definitions to guide us in building out our very basic domain model for the Shouty application.

We tend to think of the code that actually pokes around with the system as distinct from the step definitions, so we’ve drawn an extra box labelled "automation code" for this.

Automation code can do almost anything to your application: it can drive a web browser around your site, make HTTP requests to a REST API, or — as you’ve already seen — drive a domain model directly.

Automation code is a big topic that we’ll come back to. First, we want to concentrate on step definitions. Good step definitions are important because they enable the readability of your scenarios. The better you are at matching plain language phrases from Gherkin, the more expressive you can be when writing scenarios. Teams who do this well refer to their features as living documentation - a specification document that never goes out of date.

When SpecFlow first started, we used to use regular expressions to match plain language phrases from Gherkin steps.

Regular expressions have quite an intimidating reputation.

Although regular expressions are still the default matching option, we started to adopt a simpler option that has been introduced in Cucumber. It is called Cucumber expressions.

In the SpecFlow version that we use in this course, Cucumber Expressions are not supported by default. You can enable this feature by adding the CucumberExpressions.SpecFlow.3-4 NuGet package to the project, as we described in Chapter 2.

SpecFlow is backwards compatible so you can still use the power of regular expressions if that’s your thing.

This chapter is all about Cucumber Expressions.

Let’s look at the Shouty scenario from the last chapter.

As SpecFlow starts to execute this feature, it will come to the first step of the scenario Given Lucy is located 15 metres from Sean and say to itself "now - do I have any step definitions that match the phrase Lucy is located 15 metres from Sean?"

The most simple cucumber expression that would match that step is this one:

That’s pretty simple isn’t it? Cucumber expressions are just string patterns, and the most simple pattern you can use is a perfect match.

In C#, we can use this pattern to make a step definition like this:

We use a normal C# string to pass the cucumber expression to SpecFlow.

Sometimes, we want to write step definitions that allow us to use different values in our Gherkin scenarios. For example, we might want to have other scenarios that place Lucy a different distance away from Sean.

To capture interesting values from our step definitions, we can use a feature of Cucumber Expressions called parameters.

For example, to capture the number of metres, we can use the {int} parameter: which is passed as an argument to our step definition:

Now we’re capturing that value as an argument. The value 100 will be passed to our code automatically by SpecFlow.

Because we’ve used Cucumber Expressions' built-in {int} parameter type, the value has been cast to a int data type for us automatically, so we can do maths with it if we want.

Cucumber Expressions have a bunch of built-in parameter types: {int}, {float}, {word} and {string}. You can also define your own, as we’ll see later.

Although it’s important to try to use consistent terminology in our Gherkin scenarios to help develop the ubiquitous language of your domain, we also want scenarios to read naturally, which sometimes means allowing a bit of flexibility.

Ideally, the language used in scenarios should never be constrained by your step definitions. Otherwise they’ll end up sounding like they were written by robots. Or worse, they read like code.

One common example is the problem of plurals. Suppose we want to place Lucy and Sean just 1 metre apart:

Because we’ve used the singular metre instead of the plural metres we don’t get a match as you can see from the different step color:

What a pain!

Fear not. We can just surround the s in parentheses to make it optional, like this:

We build the project and now our step matches:

This is one way to smooth off some of the rough edges in your cucumber expressions, and allow your scenarios to be as expressive as possible.

Another is to allow alternates - different ways of saying the same thing. For example, to accept this step:

…​we can use this Cucumber Expression:

Now we can use either 'standing' or 'located' in our scenarios, and both will match just fine as you can see…​

Although you can get a long way with Cucumber Expressions' built-in parameter types, you get real power when you define your own custom parameter types. This allows you to transform the text captured from the Gherkin into any object you like before it’s passed into your step definition.

For example, let’s define our own {person} custom parameter type that will convert the string Lucy into an instance of Person automatically.

We can start with the step definition, which would look something like this:

If we build or run the tests at this point we’ll see an error, because we haven’t defined the {Person} parameter type yet.

Here’s how we define one.

Let’s create a new class called ParameterTypes in the Support folder:

We’re going to create a method, which takes the name of a person as a string, and returns an instance of our Person class with that name.

SpecFlow will use the name of the return type - Person - as the parameter name we use inside the curly brackets in our step definition expressions, as soon as we’ve wired it up.

To do that, we add the StepArgumentTransformation attribute , that is from the SpecFlow namespace…​ . We also need to add a Binding attribute to the class , otherwise SpecFlow will not find our conversion method.

By default, any name is recognized as a person, but we could restrict it to a specific names using - gasp! - a regular expression.

This is not what we want in this project now, so let’s leave it as it was. You can find more examples about using StepArgumentTransformation in the SpecFlow documentation.

All of this means that when we run our step, we’ll be passed an instance of Person into our step definition automatically.

Custom parameters allow you to bring your domain model - the names of the classes and objects in your solution - and your domain language - the words you use in your scenarios and step definitions - closer together.

In the previous chapter, we talked about the importance of having readable scenarios, and you learned some new skills with Cucumber Expressions to help you achieve that goal. Those skills will give you the confidence to write scenarios exactly the way you want, knowing you’ll be able to match the Gherkin steps easily from your step definition code.

We emphasise readability because from our experience, writing Gherkin scenarios is a software design activity. Cucumber was created to bridge the communication gap between business domain experts and development teams. When you collaborate with domain experts to describe behaviour in Gherkin, you’re expressing the group’s shared understanding of the problem you need to solve. The words you use in your scenarios can have a deep impact on the way the software is designed, as we’ll see in later chapters.

The more fluent you become in writing Gherkin, the more useful a tool it becomes to help you facilitate this communication. Keeping your scenarios readable means you can get feedback at any time about whether you’re building the right thing. Over time, your features become living documentation about your system. We can’t emphasize enough how important it is to see your scenarios as more than just tests.

Maintaining a living document works both ways: the scenarios will guide your solution design, but you may also have to update your Gherkin to reflect the things you learn as you build the solution. This dance back and forth between features and solution code is an important part of BDD.

In this chapter, we’ll learn about feature descriptions, the Background keyword, and about keeping scenarios and code up-to-date with your current understanding of the project.

First, let’s catch up with what’s been happening on the Shouty project.

While we were away, the developers of Shouty have been busy working on the code. Let’s have a look at what they’ve been up to.

We’ll start out by running our scenarios.

Great! It looks like both these scenarios are working now - both the different messages that Sean shouts are being heard by Lucy.

Let’s dig into the code and see how these steps have been automated.

In the step definition layer, we can see that a new class has been defined, the Network. We’re creating an instance of the network here. Then we pass that network instance to each of the Person instances we create here. So both instances of Person depend on the same instance of network. The Network is what allows people to send messages to one another.

There are also a couple of new unit test classes in the Shouty solution, one for the Network class, and another one for the Person class.

Unit tests are fine-grained tests that define the precise behaviour of each of those classes. We’ll talk more about this in a future lesson, but feel free to have a poke around in there in the meantime.

The Run All Tests command will run those unit tests as well as the SpecFlow scenarios.

The first thing I notice coming back to the code is that the feature file is still talking about the distance between Lucy and Sean, but we haven’t actually implemented any behaviour around that yet.

This happens to us all the time - we have an idea for a new feature, but then we find the problem is too complex to solve all at once, so we break it down into simpler steps. If we’re not careful, little bits of that original idea can be left around like clutter, in the scenarios and in the code. That clutter can get in the way, especially if plans change.

We’re definitely going to develop this behaviour, but we’ve decided to defer it to our next iteration. Our current solution is just focussed on broadcasting messages between the people on the network.

Let’s clean up the feature to reflect that current understanding.

After the feature keyword, we have space in a Gherkin document to write any arbitrary text that we like. We call this the feature’s description. This is a great place to write up any notes or other details that can’t easily be expressed in examples. You might have links to wiki pages or issue trackers, or to wireframes. You can put anything you like in here, as long as you don’t start a line with a Gherkin keyword, like “Rule:” or “Scenario:”.

In this case, we can add a high level description of the Shouty application. Because Shouty doesn’t yet filter by proximity, we can also write a todo list here so it’s clear that we do intend to get to that soon.

Changing the description doesn’t change anything about how SpecFlow will run this feature. It just helps the human beings reading this document to understand more about the system you’re building.

Our two scenarios are examples of how Shouty can broadcast a shout to other users. This is one of the main business rules, which we can document using the Rule keyword. We’ll learn more about this in a later chapter.

The step “Given Lucy is 15 metres from Sean” is misleading, since the distance between the two people is not yet relevant in our current model.

The step definition calls the MoveTo method on Person,

but the MoveTo method doesn’t actually do anything.

Let’s simplify this code to do just what it needs to do right now, and no more. We can start from the scenario by changing this single step to express what’s actually going on. We’ll work on one scenario at a time, and update the other one once we’re happy with this one.

Now the scenario names make sense, and we have two steps, each creating a person. Notice we’re starting to reveal some more of our domain language here: we’ve introduced the words Person and name. Person is already a part of our domain language, so it’s nice to have that revealed in the language of the scenario. Name may well become a property of our person soon, so it’s also useful to have that surfaced so we can get feedback about it from the team.

One thing we’ve lost by doing this is the idea that, eventually, the two people will need to be close to each other for the message to be transmitted. We definitely wouldn’t remove detail like that without discussing it with the other people who were involved in writing and reviewing this scenario.

In this case, as well as adding it to the TODO list above, we’ve decided to document the range rule, and write a couple of new empty scenarios to remind us to implement that behaviour later.

Let’s press on. We can invoke the Define steps…​ command to generate new step definition snippets for the new steps. Copy them to clipboard and paste them into our steps file.

In the next lesson we’ll look at a couple of ways that we can implement these new step definitions.

We now have two step definitions to implement, and that presents us with a bit of a problem. We need the same instance of Network available in both. We could just assume that the Lucy step will always run first and create it there, but that seems fragile. If someone wrote a new scenario that didn’t create people in the right order, they’d end up with no Network instance, and weird bugs. We want our steps to be as independent as possible, so they can be easily composed into new scenarios.

There are a couple of different ways to create this network instance in C#. The most straightforward is to use a network field and initialize it in the declaration of the StepDefinitions class. Every time SpecFlow runs a scenario it creates a new instance of this class, so we’ll get a fresh instance of the Network for each scenario.

As an alternative, that can be useful if you have more complex setup to do, you can use a hook.

We need an instance of Network in every scenario, so we can declare a BeforeScenario Hook that creates one before each scenario starts, like this:

Now we can use that Network instance as we create Lucy and Sean in these two new steps.

It should be working again now. Let’s run the tests to check.

Good. Let’s do the same with the other scenario.

Now we can remove this old step definition.

We know we’ll need something like this in the future when we implement the proximity rule, but we don’t want to second-guess what that code will look like, so let’s clean it out for now.

Now we have one last bit of dead code left, the MoveTo method on Person. Let’s clean that up too.

And we’re still green!

OK, so we’ve cleaned things up a bit, to bring the scenarios, the code and our current understanding of the problem all into sync. What’s nice to see is how well those new steps that create Lucy and Sean match the code inside the step definition.

When step definitions have to make a big leap to translate between our plain-language description of the domain in the Gherkin scenario, and the code, that’s usually a sign that something is wrong. We like to see step definitions that are only one or two lines long, because that usually indicates our scenarios are doing a good job of reflecting the domain model in the code, and vice-versa.

One problem that we still have with these scenarios is that we’re very fixed to only being able to use these two characters, Lucy and Sean. If we want to introduce anyone else into the scenario, we’re going to be creating quite a lot of duplicate code. In fact, the two step definitions for creating Lucy and Sean are almost identical, apart from those instance fields.

On a real project we wouldn’t bother about such a tiny amount of duplication at this early stage, but this isn’t a real project! Let’s play with the skills we learned in the last chapter to make a single step definition that can create Lucy or Sean.

The first problem we’ll need to tackle is these hard-coded instance field names.

We can use a Dictionary to store all the people involved in the scenario.

Let’s try replacing Lucy first.

We’ll start by creating a new Dictionary in the before hook, like this.

Now we can store Lucy in a key in that Dictionary. We’ll use her name as the key, hard-coding it for now.

Finally, where we check Lucy’s messages heard here in the assertion, we need to fetch her out of the Dictionary.

With that little refactoring done, we can now try and make this first step generic for any name.

Using your new found Cucumber expression skills from the last chapter, you’ll know that if we replace the word Lucy here with a parameter expression, we’ll have the name passed into our step definition as an argument, here. {word} is a special parameter type matches to a…​ word. What else. Now we can use that as the key in the Dictionary.

If we try and run the tests now, we get an error from SpecFlow about an ambiguous match.

Our generic step definition is now matching the step “a person named Sean”, but so is the original one. In bigger projects, this can be a real issue, so this warning is important.

Let’s remove the old step definition, and fetch Sean from the Dictionary here where he shouts his message.

Great, we’re green again.

Let’s switch back to the feature file to show you one more technique for improving the readability of your scenarios.

When we have common context steps - the Givens - in all the scenarios in our feature, it can sometimes be useful to get those out of the way.

We can literally move them into the background, using a Background keyword, like this:

As far as SpecFlow is concerned, these scenarios haven’t changed. It will still create both Lucy and Sean as the first things it does when running each of these scenarios.

But from a readability point of view, we can now see more clearly what’s important and interesting about these two scenarios - in this case, the message being shouted.

Notice we just went straight into When steps in our scenarios. That’s absolutely fine. We still have a context for the scenario, but we’ve chosen to push it off into the background.

Again, it’s debatable whether we’d bother to use a Background to do this on a real project, but this at least illustrates the technique. We rarely use Backgrounds in our projects, because although they can improve readability by removing the duplication of repeated contexts, they also harm readability by requiring people to read the Background in conjunction with each Scenario.

To maintain trust in the BDD process, it’s important to keep your features fresh. Even when you drive the development from BDD scenarios, you’ll still learn lessons from the implementation that might need to be fed back into your Gherkin documentation.

In this case, we discovered that we could find a smaller slice of this story, and defer the business rule about proximity until our next iteration. Splitting stories like this is a powerful agile technique, and one that BDD can help you to master. Now we have a clean codebase and suite of scenarios that reflects the current state of the system’s development.

We’re ready to start the next iteration.

Welcome back to Cucumber School.

Last time we worked on cleaning up the Shouty features to keep them in sync with the current status of the project. We stripped the scenarios back to only specify the behaviour of passing messages between people. We made it clear that the proximity rule had not yet been implemented.

You’ll already remember from the Cucumber expressions chapter how important it is to be expressive in your scenarios, and keep them readable. In this chapter we’re going to learn some new tricks with Gherkin that will give you even more flexibility about how you write scenarios.

Once again the Shouty developers — have been hard at work implementing that proximity rule. Let’s have a look at how they got on.

Right, so those two scenarios we just left as placeholders: the one where the listener is within range, and the one where the listener is out of range are passing. Fantastic! If we look at our step definitions, we can see how they have been implemented.

Let’s review the changes to the feature file in more detail.

We now have four scenarios: our original two from the last time we looked at the code, and the two placeholders we wrote as reminders.

We used the second scenario - Listener hears a different message - to triangulate and force us to replace the hard-coded message output with a proper implementation. Now we have a domain model that uses a variable for the message, there’s an insignificant chance of this behaviour regressing, so we can safely remove the second scenario.

Keeping excess scenarios is wasteful: they clutter up your feature files, distracting your readers. When you run your features as tests, excess scenarios make them take longer to run than necessary. The one where a "listener hears a "message" is a perfectly good way of checking that the message has been sent correctly.

The first scenario has changed since we last looked at it - it now specifies the range of a shout and the location of Sean and Lucy. This scenario exists to illustrate that a listener hears the message exactly as the shouter shouted it. All the additional details are incidental and make the scenario harder to read.

Let’s ensure that this scenario includes only essential information for the reader and remove all references to location and range.

We’ll need to make changes to the step definitions to make sure that a Network class is always created - which we can do using an instance field.

We’ve defaulted the range to 100.

If a scenario needs to document specific range, that can still be done by explicitly including a "Given the range is …​" step.

We’ll also need to add a step definition that can create a person without the scenario needing to specify where they are located. The step definition gives each person created this way a default location of 0.

Let’s run the scenarios to check we haven’t broken anything…​ and we’re good!

Looking at the two new scenarios - Listener is within range & Listener is out of range - we can see that they also contain incidental details. Since their purpose is to illustrate the "Shouts should only be heard if listener is within range" rule, there’s no need to actually document the content of the shout.

Let’s remove the details that aren’t relevant to the range rule.

Next we add a step definition that allows Sean to shout, without needing us to specify the exact message.

One that allows us to check that Lucy has heard exactly one shout - because she’s in range of the shouter.

And one that allows us to check that Larry hasn’t heard any messages at all - because he’s out-of-range.

And finally run all tests - and we’re still green.

That’s better. We’ve removed inessential details, so that each scenario contains only the information needed to illustrate its business rule.

The scenarios would still run green if we removed the steps that set the range of a shout , because the range already has a default value. We’re not going to, because since those scenarios are illustrating the rule that deals with the range of a shout, it’s an essential part of context for anyone reading them.

A happy side-effect is that, in order to set the range from our scenario, we’ve had to make it a configurable property of the system . So if our business stakeholders ever change their minds about the range, we won’t have to go hunting around in the code for where it’s been hard-coded.

Let’s look at the two scenarios that illustrate the rule about range again . Notice how the steps that create the Sean, Lucy, and Larry are very similar.

When we see steps like this, Gherkin’s Given When Then syntax starts to feel a bit clunky. Imagine if we could just write out a table, like this:

Well, we’re in luck. We can!

Gherkin has a special syntax called Data Tables, that allows you to specify tabular data for a step, using pipe characters to mark the boundary between cells.

As you can see, the step definition implicitly takes a single argument of type Table, which is a representation of a Data Table in SpecFlow. As it represents a table of Person objects, we can rename it to personsTable

This object has a rich API for using the tabular data. The Rows property of the table class can be used to access the data rows of the table — all rows, except the header row. A particular cell value can be retrieved from a row by using the header name. So, Lucy’s location can be accessed by getting the "name" cell of the row at index 1.

Now we can easily iterate through the rows and turn them into instances of Person:

With that done, we can update the other scenario …​

Now we can check that everything is still green.

and delete our old step definition, which is now unused.

SpecFlow strips all the white space surrounding each cell , so we can have a nice neat table in the Gherkin but still get clean values in the step definition underneath.

Notice we’ve still had to convert the location from a string to an integer , because SpecFlow can’t know that’s the type of value in our table.

To improve the readability and maintainability of your step definition you can have SpecFlow automatically convert the table into a list of any class you want. If our Person object had a name field we could automatically create instances of Person from this table. But things aren’t always that simple.

Instead, we’ll define a simple Whereabouts class to represent the data in the table.

We’ve made it a nested type to the step definition class, as it doesn’t form part of our core domain.

Then we can create a conversion method similarly to the ones we created in Chapter 3 that converts a Table instance to an array of Whereabouts. We have to add the [StepArgumentTransformation] attribute to it so that SpecFlow recognizes this conversion.

Now, if you declare your table parameter as a Whereabouts array , SpecFlow will automatically call our conversion method.

Let’s run the scenarios to check that we’re still green. And we are!

That looks much nicer - people positioned using a table in the feature file and really clean code that creates and positions people according to the data.

We separated the Data Table conversion from the step definition. The step definition is nice and clean now, but our conversion method is still fairly complex as it needs to handle the table headers and the cell data conversion.

But even though it is complex, we can notice a pattern in it. We take the "name" cell and update the Name property of the Whereabouts object. We take the "location" cell and update the Location property…​ This is not a big surprise for us, because we let the domain terms in the scenarios drive our domain model.

When this happens, this sort of consistency allows us to simplify the code further. SpecFlow defines an extension method on the Table class, called CreateSet. CreateSet can do exactly what we need here: create instances of a particular class and update its properties based on the table cells.

The CreateSet extension method is defined in the TechTalk.SpecFlow.Assist namespace, so we have to add a using statement for that.

With that change our conversion method is also clean and our tests still pass.

Data tables are very useful for setting up data in Given steps, but you can also use them for specifying outcomes.

One rule that we’ve been implying, but have never actually explored with an example, is that people can hear more than one shout. So far we’ve only specified a single message, so let’s try writing a scenario where Sean shouts more than once:

See how natural it is to use a Data Table here?

So how do we implement this step definition? First, let’s paste the generated snippet into the StepDefinitions class and rename the parameter. What we want to do in this step definition is compare the messages that are actually heard with the messages we expected to hear. Getting the actual messages is easy, we just need to call the GetMessagesHeard method. For the expected messages, we need to take the "message" cell from each table row like this. And finally we can make sure they are the same using an assert statement.

Oops! It looks like there is a problem. The two lists are different. By checking the error message we can see that there is a typo in our scenario. But let’s not fix this yet. Instead, let’s look at another typical pattern for a data table related assertion.

The GetMessagesHeard method returned a list of strings, but in many cases the list that we want to compare the Data Table with is a list of objects. Let’s imagine that later we will extend the GetMessagesHeard method to not only return the message but also the person who shouted it. To simulate that let’s add an alternative version of GetMessagesHeard, called GetMessagesHeardEx. The new method returns a list of HeardMessage instances. The HeardMessage class has only one property, the message, but later we might extend it.

Let’s first change the part of the step definition that calculates the actual messages by calling our extended method.

To compare this with the expected messages in the table, we need to check if the "message" cell matches the Message property for each row. If there was a "shouter name" cell we would have to match that to a ShouterName property and so on. This is the same consistency that we had when we used CreateSet. For such assertions, we can use another helper method, called CompareToSet.

It works similarly to CreateSet. The CompareToSet is an extension method on the Table class from the Assist namespace, and you need to pass the list of objects that you would like to compare the table with.

Both CreateSet and CompareToSet can be further customized, but this is just enough for us. Let’s run the tests and look at the error message.

Oh! We’ve found the typo: we should have included exclamation marks on the expected messages. Well, at least this gives you a chance to see the nice diff output from CompareToSet when the table and the list are different. We see the expected values prefixed with a minus, and the actual values prefixed with a plus.

Let’s fix just one of these so you can see how the diff output changes.

The matching bagels! line no longer has a minus, and for the mismatched row, the actual value still has a minus, and the expected value has a plus.

Let’s fix this last typo , and we should be green again.

Great.

When writing scenarios, occasionally we want to use a really long piece of data.

For example, let’s introduce a new rule about the maximum length of a message

…​and add a scenario to illustrate it , making the string just over the boundary of the rule:

That’s pretty ugly isn’t it!

Still, we’ll press on and get it to green, then we’ll show you how to clean it up.

Our existing step definition handles that ugly step with the long message just fine, but the last outcome step is undefined . We could either add a new step definition, or parametrize "Larry should not hear a shout". Let’s modify the existing step definition

OK, so we have a failing acceptance test. Let’s dive down into our solution and implement this new rule. It seems like the Network should be responsible for implementing this rule, so let’s go to its unit tests. As we explain in more detail in the next lesson, we start by adding a new unit test to specify this extra responsibility.

We’ll create a 181-character message like this and then assert that the message should not be heard when it’s broadcast.

Let’s run that test. Good, it fails. Laura’s still getting the message at the moment. Now how are we going to implement this?

It looks like we’re already implementing the proximity rule here in the broadcast method. Let’s add another if statement here about the message length.

Run the unit test again… and it’s passing. Great.

The code here has got a little bit messy and hard to read. One very basic move we could make to improve it would be to just extract a couple of temporary variables, one for the range rule and one for the length rule.

That’s better. This code could be improved even further of course, but at least we haven’t made it any worse.

Let’s just run the tests to check. Great - everything’s still green.

As now we have everything passing again, we can tidy up the Gherkin to use a new piece of syntax we’ve been wanting to tell you about: a DocString.

DocStrings allow you to specify a text argument for a step that spans over multiple lines. We could change our step to look like this instead:

Now the scenario is much more readable.

We have to add a new step definition too . It doesn’t need a parameter in the Cucumber Expression  — the DocString gets passed as a string argument to the step definition automatically.

Now we can fill out the rest of our new step definition.

Let’s check that we’re still green  — and we are!

These two step definitions do the same thing and they even have the same parameters. So we can keep just one of them and add the When attribute of the other to it.

Having multiple Given, When or Then attributes on the same method is another way to handle alternates.

We don’t use DocStrings very often - having such a lot of data in a test can often make it quite brittle. But when you do need it, it’s useful to know about.

You might have noticed that we’ve followed a pattern when we added behaviour to the system during this episode.

First we expressed the behaviour we wanted in a Gherkin scenario, wired up the step definitions, then ran SpecFlow to watch it fail.

Then, we found the first class in our domain model that needed to change in order to support that new behaviour. In this case, the Network class. We used a unit test to describe how we wanted instances of that class to behave. Then we ran the unit test and watched it fail.

We focused in and made changes to the class until its unit tests were passing. When the unit tests were passing, we then made some minor changes to clean up the code and make it more readable. This is the basic test-driven-development cycle: red, green, clean.

The technical name for this last clean-up step is refactoring. Refactoring is an ugly name for an extremely valuable activity: improving the design of existing code without changing its behaviour. You can think about it like cleaning up and washing the dishes after you’ve prepared a meal: basic housekeeping. But imagine the state of your kitchen if you never made time to do the dishes.

Go on, imagine it for a second.

Yuck!

Well, that’s how many, many codebase end up. The good thing about taking this course is that we’re teaching you how to write solid automated tests, and the good thing about having solid automated tests is that you can refactor with confidence, knowing that if you accidentally change the system’s behaviour, your tests will tell you.

Once we’re done refactoring, what do we do next? Run SpecFlow, of course! In this case, our scenario was passing with a single trip round the inner TDD loop, but sometimes you can spend several hours working through all the unit tests you need to get a single scenario to green.

Once the acceptance test is passing, we figure out the next most valuable scenario on our todo list, and start the whole thing all over again!

Together, these two loops make the BDD cycle. The outer loop, which starts with an acceptance test, keeps us focussed on what the business needs us to do next. The inner loop, where we continuously test, implement then refactor small units of code, is where we decide how we’ll implement that behaviour.

Both of these levels of feedback are important. It’s sometimes said that your acceptance tests ensure you’re building the right thing, and your unit tests ensure you’re building the thing right.

That’s all for this chapter. See you next time!

Hello, and welcome back to Cucumber School.

Last time we learned about two very different kinds of loops. First, we used DataTables to loop over data in your scenarios.

Then we learned about BDD cycles. We saw how the outer loop of BDD helps you to build the right thing while the inner loop helps you build the thing right.

In this lesson, we’re going to teach you all about how to run different SpecFlow scenarios.

When we start working on a new scenario we often take a dive down to the inner TDD loop where we use a unit testing tool to drive out new classes or modify the behaviour of existing ones. When our unit tests are green and the new code is implemented we return to the SpecFlow scenarios to verify whether we have made overall progress or not.

If we have lots of SpecFlow scenarios, it can be distracting to run all of them each time we do this. We often want to focus on a single scenario - or perhaps just a couple - to get feedback on what we’re currently working on.

There are several ways to do this. SpecFlow converts the scenarios to tests that can be executed by the test execution framework we configured for the project, which is xUnit in our case. Because of that the SpecFlow scenarios appear as regular coded tests in the test runner tools you use, for example in the Visual Studio Test Explorer window. These tools usually provide several filtering options.

Probably the easiest way to filter is to run only the scenario with a specified name.

Simply typing the name of the scenario to the Test Explorer window search box filters the list to that particular scenario. You can run or debug the selected scenario from the context menu but the easiest is to hit Run All that runs all scenarios in the filtered view. It is worth spending a few minutes learning the keyboard shortcut of that, Ctrl R,V by default, as it can speed up your loops drastically.

You can filter for the scenario name as you have seen but you can use the search box to filter for keywords as well. Let’s use it to run all scenarios with the text "range" in their name.

The Test Explorer window can also be used to filter for the outcome, for example if you want to re-run all failing tests, or you can also run tests based on their hierarchy. Since the feature files appear as a separate node in the hierarchy, you can use this to run all scenarios from a particular feature file.

The search box can contain complex search expressions. You can discover these options by clicking on the Add search filter links or by checking out the documentation. We’ll use this to show you how to filter using tags.

Let’s say we want to work on this scenario for the next couple of hours. First, we’ll put a focus tag right here, above this scenario. Tags start with an at-sign and are case sensitive.

As we said, SpecFlow converts the scenarios to tests. This conversion happens at compile-time, so I need to build my solution to be able to have the changes applied.

SpecFlow converts the tags to test categories or traits as the Test Explorer window calls these. So to be able to filter for a particular tag, we have to enter a trait expression.

Trait expressions start with the word Trait followed by a colon and the name of the tag without the at sign.

Now we can run only the scenarios tagged with focus…​

It’s entirely up to you what you name your tags. When we’re working on a particular area of the application it is common to use a temporary tag like this - we’ll remove it before we check our code into source control.

Tags can be used for other purposes as well. If you have lots of scenarios it can be time-consuming to run them all every time. For example, you can tag a few of them with @smoke and run only those before you check in code to source control. Running just the smoke tests will give you a certain level of confidence that nothing is broken without having to run them all.

We filtered the Test Explorer window but as you can see no scenarios are shown. This is because the list haven’t refreshed yet. In fact we haven’t even saved the file yet! We could save the file, build the project and run the tests, but the Run All command does all of these steps.

Here it is! Now you probably understand better why learning the keyboard shortcut for this command helps a lot. Ctrl R,V!

Running the smoke tests gives you a quick feedback. If you’re running SpecFlow on a Continuous Integration Server as well, you could run all the scenarios there, detecting any regressions you might have missed by only running the smoke tests.

Tags give you a way to organize your scenarios that cut across feature files. You can think of them like sticky labels you might put into a book to mark interesting pages that you want to refer back to.

Some teams also use tags to reference external documents, for example, tickets in an issue tracker or planning tool. Let’s pretend we are using an issue tracker while working on Shouty and all the behaviour we built so far is related to the issue number 15. We could tag the whole feature file with this single line at the top. All the scenarios within that file now inherit that tag, so if we filter for this tag, Visual Studio will run all the scenarios in the feature file.

You can use more complex tag expressions to select the scenarios you want to run. For example, you could use a trait expression to exclude all the scenarios tagged as @slow. Let’s mark a few of them as slow…​ and this time I build the project to show the filtering results. Then rewrite the trait expression in the search box to filter for the slow scenarios. Now if I add a dash ("-") to the front you can see all that are not slow. Now when we run the tests, the "@slow" scenarios won’t be run.

You can read about how to build more complicated filter expressions in Visual Studio in the Visual Studio documentation

There’s one more thing to learn about tags. They can be combined with hooks, so that you can be selective about which hooks to run when. We’ll cover that in a future chapter.

In the previous lesson, we ran the SpecFlow scenarios in Visual Studio. That gives you the quick feedback that you need during development.

But we also would like to regularly ensure that the changes we made didn’t cause any regressions. For that we can kick off the full test execution on a Continuous Integration Server or from a console on your local machine so we can keep working on other things while it’s finished.

It does not matter whether you need to configure your Continuous Integration build or you want to run the test from a console locally, the .NET command line tools have to be used for this. But fortunately it is pretty easy.

Let’s open a CMD console or a PowerShell window and change the directory to the folder of your project.

Now use the dotnet test command.

The dotnet test command is part of the cross-platform .NET framework, so you can do the same on Linux or on macOS as well. And also inside a Docker container of course.

The output is not very verbose but it clearly shows that all of our test were passing, so our changes did not cause any unwanted side-effects. Good to know.

We wanted to perform a full regression test by running the tests from the command line, therefore we did not apply any filters. But sometimes we need to. Maybe we just want to run the smoke tests.

The dotnet test command provides a --filter option where you can specify a filter expression. In the previous lesson we discussed that SpecFlow converts tags to test categories or traits. In Visual Studio we used a trait expression, but unfortunately the same expression won’t work here. The exact expression syntax that has to be used depends on the test execution framework you use. Since we’re use xUnit, we have to filter using the Category=smoke expression. In MsTest and NUnit we would have to say TestCategory instead of Category, but the expression is otherwise the same. Let’s fix this for xUnit and run the tests.

As you can see, only two tests ran, so the filtering worked.

You can also use the filter expression to exclude tagged scenarios from the execution. For that you have to use the not equal operator. Let’s exclude the slow tests now.

Again, with MsTest or NUnit you would need to use TestCategory here. You can compose even more complex filter expressions for dotnet test. It is worth checking out the documentation for details.

Teams often use such filters on the Continuous Integration Server if they write - or we can say formulate - the scenarios some time before the actual implementation work starts. Having these scenarios in source control would cause the build fail although there is nothing wrong with the implementation. We just have tests that are not supposed to pass yet. If the team agree to tag these scenarios, for example with @formulated than they can easily exclude them on the build server with this expression. .

Seeing all these scenarios pass means that we are progressing well. Just don’t forget to remove the @formulated tag once you are done!

In this lesson you’ve learnt how to run scenarios from the command line and how to filter the set of scenarios to run using tags. On the Continuous Integration Server we probably also want to preserve and publish the test results. We’ll cover that in lesson 4.

SpecFlow is first and foremost a tool that facilitates a common understanding between people on a project. To be able to express the domain requirements of projects implemented specifically for a non-English speaking local market, translating everything to English might be a barrier. In this case it is better to describe the requirements in the spoken language the stakeholders speak.

SpecFlow supports over 70 different languages, thanks to contributions from people from all over the world.

To see the translation of the Gherkin keywords for a particular language, check out the Gherkin syntax documentation. For example, this is how the keywords translate to Hungarian.

Let’s see how someone would create a feature file written in a different spoken language. I will do this for the Hungarian language, but you can choose another language you speak.

For that we can add a new feature file first…​ and make it empty.

The first line tells SpecFlow which language the feature file is written in.

Now we can start writing our feature file. Remember, it should start with the feature keyword. The auto-complete feature of the Visual Studio integration is for your help. It shows which keyword I should use.

We can keep editing the file by adding a scenario. In the selected language, of course…​

This is Cucumber School and not a Hungarian lesson, but you might have guessed that my scenario tells the same story that we described in the "Listener hears a message" scenario earlier. But if we look at this new scenario we can see that Visual Studio reports the steps as undefined. This is because we have used the English step texts in our Cucumber Expressions. In order to make this pass, we would need to add step definitions with Hungarian texts in the expressions or add an additional Given, When or Then attribute to our step definition methods with the Hungarian text, like this.

If we build the project, the first two steps appear to be defined now.

We can fix the other two step definitions as well…​ and we have our first Hungarian shouty scenario pass!

Usually the feature file languages are not mixed within a project. If you don’t want to add the same language header line to all of the feature files, you can also change the default feature file language in SpecFlow.

For that we need to add a SpecFlow configuration file to our project. The SpecFlow configuration file is a simple JSON file, named specflow.json. . This file needs to be copied to the output directory.

As there is no auto-complete support for this file, it’s easiest to check out the SpecFlow documentation. The first example here does exactly what we want, so we can just copy and paste it into our project, and change the language to the one we have chosen.

When we build the project, we got lots of errors from our English feature file. This is because we changed the default and this file does not have a language setting at the top. But the new file works without the language setting now…​

In Chapter 3 we talked about step definition parameters and that SpecFlow helps you to convert them to a .NET data type. SpecFlow performs these conversions using the feature file language. So for example if you include a fractional number that needs to be converted to double, SpecFlow will expect the decimal separator character used in the language of your feature file. In the Hungarian language for example, a comma is used as the decimal separator instead of a dot. This is also important for date conversions.

This default behavior can be changed by explicitly declaring what culture setting SpecFlow should use for conversions. For example we can force the conversions to be made using US English culture even in Hungarian feature files.

There are a few other things you can configure that you can find in the documentation. It’s worth highlighting the one that can instruct SpecFlow to load step definitions from other external projects as well. This is useful in bigger applications with multiple SpecFlow projects.

For the rest of this chapter, let’s remove the non-English feature file and reset the configuration by removing the specflow.json file.

That’s quite a lot to digest, but to make SpecFlow really useful to your team, it’s good to spend some time learning the details of how to configure it. In this lesson, we showcased the SpecFlow configuration options and you learned how to write your scenarios in different spoken languages.

In the previous lessons we’ve learnt how to configure SpecFlow and how to run scenarios from Visual Studio and from the command line. The test output we have seen there contains a brief summary of the execution and the failures. If the test were executed on a Continuous Integration Server we probably also want to preserve and publish the test results. This is what we will look at in this lesson.

The dotnet test tool outputs the execution results through configurable loggers. The logger can be specified using the --logger option. There are multiple logger providers you can choose from. You can even write your own. But most of the teams use the TRX logger, that saves the execution results into a Visual Studio Test Results or TRX file. Let’s use this.

The execution finished and reported that the results have been saved to a file that has the current timestamp inside the TestResults folder of the project. Let’s look at this file.

Visual Studio has been associated to this file extension, so if I open this file from file explorer , it opens it in Visual Studio. We can see all the individual test executions and even the details of each of them where we see the step execution results.

The TRX files are XML files so it is easy to make any tools that processes the results and converts them to another format. It is, for example, a common expectation to present the execution results as an HTML file that can be checked by all stakeholders even if they don’t have Visual Studio.

There are many small open source tools that can do such a conversion. For example the fork of the Trxer project by @stevencohn on GitHub is a nice one that provides some dedicated styling for SpecFlow results.

This one is a .NET assembly that I downloaded earlier, so now I can invoke it with the dotnet command we have seen already.

The result is a single HTML file that I could publish to my team workspace or just open locally. It shows the execution summary as well as the individual scenario results grouped by features.

The TRX file format is also understood by most of the Continuous Integration platforms as well, so they can show the details included in the build results or even highlight whether the same tests were failing for the first time.

Processing the TRX file on the build server is not that easy with non-deterministic file names. In this case it is more useful to specify the output file name for the TRX logger. This can be done by specifying the logfilename parameter of the TRX logger like this.

The SpecFlow+ LivingDoc Generator from the Tricentis SpecFlow team uses a slightly different approach. This free tool does not process the TRX file, but produces its own result file and from that it is able to generate comprehensive documentation of your requirements based on the feature files and the scenarios you defined. Although the generated documentation also contains the test results, its primary focus is to produce the so-called Living Documentation — documentation that shows whether the application currently fulfills the requirements.

In order to setup SpecFlow+ LivingDoc Generator, we need to install the SpecFlow.Plus.LivingDocPlugin NuGet package.

Now let’s run the tests. We don’t need the TRX file now, so running dotnet test is enough. The SpecFlow+ LivingDoc plugin generates a result file in the output folder, named TestExecution.json by default.

Here it is!

Now we need to run the actual generator. The generator has to be installed as a .NET tool for the project.

Since this is the first .NET tool that we have install for this project, we need to first initialize the .NET tool configuration. For that we need to go up to the solution folder in our console window and invoke dotnet new tool-manifest . This creates a file named dotnet-tools.json in the .config folder of the solution. You should add this file to your source control.

Now we can install the SpecFlow+ LivingDoc Generator. This can be done, using the dotnet tool install command that pulls the generator from NuGet.org and installs it for this project.

As the message says, the tool has been installed successfully and can be used with the dotnet livingdoc command.

First, let’s step back to our SpecFlow project folder and start with dotnet livingdoc. Now we need to specify a couple of settings for the generator. If you get lost, you can always call dotnet livingdoc help or check the documentation.

We need to say test-assembly, because we want to generate the result from an assembly, and we have to specify the compiled assembly of our Shouty project.

Finally we need to provide the path of the generated TestExecution.json file using the --test-execution-json option. And run.

We’ve got the generated HTML file. As you can see, the result focuses on the feature structure, but the details are also displayed.

Since the result is a single HTML file, it is easy to share it with all stakeholders. If you use Azure DevOps to track your work items, you can use the integrated version of SpecFlow+ LivingDoc as well.

There are more and more tools that can help you to implement living documentation for SpecFlow, like Pickles Doc, Augurk or SpecSync for Azure DevOps. The SmartBear Cucumber team is working on a platform called Cucumber Reports that can be used to publish and share living documentation. At the time of the recording this is not yet available for SpecFlow, but hopefully it will be soon.

In this lesson we have enumerated a couple of options to share your results and your living documentation with the entire team. This is essential in order to facilitate a common understanding between people on the project — which is our ultimate goal.

With that we finished this chapter. See you next time!

In terms of the three practices we introduced in Chapter 1 - Discovery, Formulation and Automation - what went wrong with the Premium Accounts feature?

Thinking about it, we can see that the development team jumped straight into Automation - writing the implementation of the feature. They did the bare minimum of Formulation - just enough to automate a test for the feature, but really we did a lot of the Formulation later on as we cleaned it up. Finally, much of the Discovery happened at the end once Tamsin had a chance to test the feature by hand.

So in essence, they did everything backwards.

In software projects, it’s often the unknown unknowns that can make the biggest difference between success and failure. In BDD, we always try to assume we’re ignorant of some important information and try to deliberately discover those unknown unknowns as early as possible, so they don’t surprise us later on.

A team that invests just a little bit extra in Discovery, before they write any code, saves themselves a huge amount of wasted time further down the line.

In lesson 1, we showed you an example of the Three Amigos - Tester, Developer and Product owner - having a conversation about a new user story.

Nobody likes long meetings, so we’ve developed a simple framework for this conversation that keeps it quick and effective. We call this Example Mapping.

An Example Mapping session takes a single User Story as input, and aims to produce four outputs:

We capture these, as we talk, using index cards, or another virtual equivalent.

Working with these simple artefacts rather than trying to go straight to formulating Gherkin, allows us to keep the conversation at the right level - seeing the whole picture of the user story without getting lost in details.

We first developed example mapping in face-to-face meeting using a simple a multi-colour pack of index cards and some pens. For teams that are working remotely, there are many virtual equivalents of that nowadays.

We use the four different coloured cards to represent the four main kinds of information that arise in the conversation.

We can start with the easy stuff: Take a yellow card and write down the name of the story.

Now, do we already know any rules or acceptance criteria about this story?

Write each rule down on a blue card:

They look pretty straightforward, but let’s explore them a bit by coming up with some examples.

Darren the developer comes up with a simple scenario to check he understands the basics of the “buy” rule: "I start with 10 credits, I shout buy my muffins and then I want to buy some socks, then I have zero credits. Correct?"

"Yes", says Paula.

Darren writes this example up on a green card, and places it underneath the rule that it illustrates.

Tammy the tester chimes in: "How about the one where you shout a word that contains buy, like buyer for example? If you were to shout I need a buyer for my house. Would that lose credits too?"

Paula thinks about it for a minute, and decides that no, only the whole word buy counts. They’ve discovered a new rule! They write that up on the rule card, and place the example card underneath it.

Darren asks: "How do the users get these credits? Are we building that functionality as part of this story too?"

Paula tells him that’s part of another story, and they can assume the user can already purchase credits. They write that down as a rule too.

This isn’t a behaviour rule - it’s a rule about the scope of the story. It’s still useful to write it down since we’ve agreed on it. But it won’t need any examples. We could also have chosen to use a red card her to write down our assumption.

Still focussed on the “buy” rule, Tammy asks: "What if they run out of credit? Say you start with 10 credits and shout buy three times. What’s the outcome?"

Paula looks puzzled. "I don’t know". She says. I’ll need to give that some thought.

Darren takes a red card and writes this up as a question.

They apply the same technique to the other rule about long messages, and pretty soon the table is covered in cards, reflecting the rules, examples and questions that have come up in their conversation. Now they have a picture in front of them that reflects back what they know, and still don’t know, about this story.

As you’ve just seen, an example mapping session should go right across the breadth of the story, trying to get a complete picture of the behaviour. Inviting all three amigos - product owner, tester and developer - is important because each perspective adds something to the conversation.

Although the apparent purpose of an example mapping session is to take a user story, and try to produce rules and examples that illustrate the behaviour, the underlying goal is to achieve a shared understanding and agreement about the precise scope of a user story. Some people tell us that example mapping has helped to build empathy within their team!

With this goal in mind, make sure the session isn’t just a rubber-stamping exercise, where one person does all the talking. Notice how in our example, everyone in the group was asking questions and writing new cards.

In the conversation, we often end up refining, or even slicing out new user stories to make the current one smaller. Deciding what a story is not - and maximising the amount of work not done - is one of the most useful things you can do in a three amigos session. Small stories are the secret of a successful agile team.

Each time you come up with an example, try to understand what the underlying rule or rules are. If you discover an example that doesn’t fit your rules, you’ll need to reconsider your rules. In this way, the scope of the story is refined by the group.

Although there’s no doubt of the power of examples for exploring and talking through requirements, it’s the rules that will go into the code. If you understand the rules, you’ll be able to build an elegant solution.

As Dr David West says in his excellent book "Object Thinking", If you problem the solution well enough, the solution will take care of itself.

Sometimes, you’ll come across questions that nobody can answer. Instead of getting stuck trying to come up with an answer, just write down the question.

Congratulations! You’ve just turned an unknown unknown into a known unknown. That’s progress.

Many people think they need to produce formal Gherkin scenarios from their three amigos conversations, but in our experience that’s only occasionally necessary. In fact, it can often slow the discussion down.

The point of an example mapping session is to do the discovery work. You can do formulation as a separate activity, next.

One last tip is to run your example mapping sessions in a timebox. When you’re practiced at it, you should be able to analyse a story within 25 minutes. If you can’t, it’s either too big, or you don’t understand it well enough yet. Either way, it’s not ready to play.

At the end of the 25 minutes, you can check whether everyone thinks the story is ready to start work on. If lots of questions remain, it would be risky to start work, but people might be comfortable taking on a story with only a few minor questions to clear up. Check this with a quick thumb-vote.