Skip to content

Jimmy Bogard
Syndicate content
Strong opinions, weakly held
Updated: 7 hours 25 min ago

Clean Tests: Isolation with Fakes

Tue, 03/24/2015 - 17:58

Other posts in this series:

So far in this series, I’ve walked through different modes of isolation – from internal state using child containers and external state with database resets and Respawn. In my tests, I try to avoid fakes/mocks as much as possible. If I can control the state, isolating it, then I’ll leave the real implementations in my tests.

There are some edge cases in which there are dependencies that I can’t control – web services, message queues and so on. For these difficult to isolate dependencies, fakes are acceptable. We’re using AutoFixture to supply our mocks, and child containers to isolate any modifications. It should be fairly straightforward then to forward mocks in our container.

As far as mocking frameworks go, I try to pick the mocking framework with the simplest interface and the least amount of features. More features is more headache, as mocking frameworks go. For me, that would be FakeItEasy.

First, let’s look at a simple scenario of creating a mock and modifying our container.

Manual injection

We’ve got our libraries added, now we just need to add a way to create a fake and inject it into our child container. Since we’ve built an explicit fixture object, this is the perfect place to put our code:

public T Fake<T>()
{
    var fake = A.Fake<T>();

    Container.EjectAllInstancesOf<T>();
    Container.Inject(typeof(T), fake);

    return fake;
}

We create the fake using FakeItEasy, then inject the instance into our child container. Because we might have some existing instances configured, I use “EjectAllInstancesOf” to purge any configured instances. Once we’ve injected our fake, we can now both configure the fake and use our container to build out an instance of a root component. The code we’re trying to test is:

public class InvoiceApprover : IInvoiceApprover
{
    private readonly IApprovalService _approvalService;

    public InvoiceApprover(IApprovalService approvalService)
    {
        _approvalService = approvalService;
    }

    public void Approve(Invoice invoice)
    {
        var canBeApproved = _approvalService.CheckApproval(invoice.Id);

        if (canBeApproved)
        {
            invoice.Approve();
        }
    }
}

In our situation, the approval service is some web service that we can’t control and we’d like to stub that out. Our test now becomes:

public class InvoiceApprovalTests
{
    private readonly Invoice _invoice;

    public InvoiceApprovalTests(Invoice invoice,
        SlowTestFixture fixture)
    {
        _invoice = invoice;

        var mockService = fixture.Fake<IApprovalService>();
        A.CallTo(() => mockService.CheckApproval(invoice.Id)).Returns(true);

        var invoiceApprover = fixture.Container.GetInstance<IInvoiceApprover>();

        invoiceApprover.Approve(invoice);
        fixture.Save(invoice);
    }

    public void ShouldMarkInvoiceApproved()
    {
        _invoice.IsApproved.ShouldBe(true);
    }

    public void ShouldMarkInvoiceLocked()
    {
        _invoice.IsLocked.ShouldBe(true);
    }
}

Instead of using FakeItEasy directly, we go through our fixture instead. Once our fixture creates the fake, we can use the fixture’s child container directly to build out our root component. We configured the child container to use our fake instead of the real web service – but this is encapsulated in our test. We just grab a fake and start going.

The manual injection works fine, but we can also instruct AutoFixture to handle this a little more intelligently.

Automatic injection

We’re trying to get out of creating the fake and root component ourselves – that’s what AutoFixture is supposed to take care of, creating our fixtures. We can instead create an attribute that AutoFixture can key into:

[AttributeUsage(AttributeTargets.Parameter)]
public sealed class FakeAttribute : Attribute { }

Instead of building out the fixture items ourselves, we go back to AutoFixture supplying them, but now with our new Fake attribute:

public InvoiceApprovalTests(Invoice invoice, 
    [Fake] IApprovalService mockService,
    IInvoiceApprover invoiceApprover,
    SlowTestFixture fixture)
{
    _invoice = invoice;

    A.CallTo(() => mockService.CheckApproval(invoice.Id)).Returns(true);

    invoiceApprover.Approve(invoice);
    fixture.Save(invoice);
}

In order to build out our fake instances, we need to create a specimen builder for AutoFixture:

public class FakeBuilder : ISpecimenBuilder
{
    private readonly IContainer _container;

    public FakeBuilder(IContainer container)
    {
        _container = container;
    }

    public object Create(object request, ISpecimenContext context)
    {
        var paramInfo = request as ParameterInfo;

        if (paramInfo == null)
            return new NoSpecimen(request);

        var attr = paramInfo.GetCustomAttribute<FakeAttribute>();

        if (attr == null)
            return new NoSpecimen(request);

        var method = typeof(A)
            .GetMethod("Fake", Type.EmptyTypes)
            .MakeGenericMethod(paramInfo.ParameterType);

        var fake = method.Invoke(null, null);

        _container.Configure(cfg => cfg.For(paramInfo.ParameterType).Use(fake));

        return fake;
    }
}

It’s the same code as inside our context object’s “Fake” method, made a tiny bit more verbose since we’re dealing with type metadata. Finally, we need to register our specimen builder with AutoFixture:

public class SlowTestsCustomization : ICustomization
{
    public void Customize(IFixture fixture)
    {
        var contextFixture = new SlowTestFixture();

        fixture.Register(() => contextFixture);

        fixture.Customizations.Add(new FakeBuilder(contextFixture.Container));
        fixture.Customizations.Add(new ContainerBuilder(contextFixture.Container));
    }
}

We now have two options when building out fakes – manually through our context object, or automatically through AutoFixture. Either way, our fakes are completely isolated from other tests but we still build out our root components we’re testing through our container. Building out through the container forces our test to match what we’d do in production as much as possible. This cuts down on false positives/negatives.

That’s it for this series on clean tests – we looked at isolating internal and external state, using Fixie to build out how we want to structure tests, and AutoFixture to supply our inputs. At one point, I wasn’t too interested in structuring and refactoring test code. But having been on projects with lots of tests, I’ve found that tests retain their value when we put thought into their design, favor composition over inheritance, and try to keep them as tightly focused as possible (just like production code).

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Clean Tests: Isolating the Database

Mon, 03/02/2015 - 19:35

Other posts in this series:

Isolating the database can be pretty difficult to do, but I’ve settled on a general approach that allows me to ensure my tests are built from a consistent starting point. I prefer a consistent starting point over something like rolled back transactions, since a rolled back transaction assumes that the database is in a consistent state to begin with.

I’m going to use my tool Respawn to build a reliable starting point in my tests, and integrate it into my tests. In my last post, I walked through creating a common fixture in which my tests use to build internal state. I’m going to extend that fixture to also include my Respawn project:

public class SlowTestFixture
{
    private static IContainer Root = IoC.BuildCompositionRoot();
    private static Checkpoint Checkpoint = new Checkpoint
    {
        TablesToIgnore = new[]
        {
            "sysdiagrams",
            "tblUser",
            "tblObjectType",
        },
        SchemasToExclude = new[]
        {
            "RoundhousE"
        }
    };

    public SlowTestFixture()
    {
        Container = Root.CreateChildContainer();
        Checkpoint.Reset("MyConnectionStringName");
    }

Since my SlowTestFixture is used in both styles of organization (fixture per test class/test method), my database will either get reset before my test class is constructed, or before each test method. My tests start with a clean slate, and I never have to worry about my tests failing because of inconsistent state again. The one downside I have is that my tests can’t be run in parallel at this point, but that’s a small price to pay.

That’s pretty much all there is – because I’ve created a common fixture class, it’s easy to add more behavior as necessary. In the next post, I’ll bring all these concepts together with a couple of complete examples.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Reliable database tests with Respawn

Thu, 02/19/2015 - 19:21

Creating reliable tests that exercise the database can be a tricky beast to tame. There are many different sub-par strategies for doing so, and most of the documented methods talk about resetting the database at teardown, either using rolled back transactions or table truncation.

I’m not a fan of either of these methods – for truly reliable tests, the fixture must have a known starting point at the start of the test, not be relying on something to clean up after itself. When a test fails, I want to be able to examine the data during or after the test run.

That’s why I created Respawn, a small tool to reset the database back to its clean beginning. Instead of using transaction rollbacks, database restores or table truncations, Respawn intelligently navigates the schema metadata to build out a static, correct order in which to clear out data from your test database, at fixture setup instead of teardown.

Respawn is available on NuGet, and can work with SQL Server or Postgres (or any ANSI-compatible database that supports INFORMATION_SCHEMA views correctly).

You create a checkpoint:

private static Checkpoint checkpoint = new Checkpoint
{
    TablesToIgnore = new[]
    {
        "sysdiagrams",
        "tblUser",
        "tblObjectType",
    },
    SchemasToExclude = new []
    {
        "RoundhousE"
    }
};

You can supply tables to ignore and schemas to exclude for tables you don’t want cleared out. In your test fixture setup, reset your checkpoint:

checkpoint.Reset("MyConnectionStringName");

Or if you’re using a database besides SQL Server, you can pass in an open DbConnection:

using (var conn = new NpgsqlConnection("ConnectionStringName"))
{
    conn.Open();

    var checkpoint = new Checkpoint {
        SchemasToInclude = new[]
        {
            "public"
        },
        DbAdapter = DbAdapter.Postgres
    };

    checkpoint.Reset(conn);
}

Because Respawn stores the correct SQL in the right order to clear your tables, you don’t need to maintain a list of tables to delete or recalculate on every checkpoint reset. And since table truncation won’t work with tables that include foreign key constraints, DELETE will be faster than table truncation for test databases.

We’ve used this method at Headspring for the last six years or so, battle tested on a dozen projects we’ve put into production.

Stop worrying about unreliable database tests – respawn at the starting point instead!

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Clean Tests: Isolating Internal State

Tue, 02/17/2015 - 19:46

Other posts in this series:

One of the more difficult problems with slow tests that touch shared resources is building a clean starting point. In order for tests to be reliable, the environment in which the test executes needs to be in a reliable, consistent starting state. In slow tests, in which I’m accessing out-of-process dependencies, I’m worried about two things:

  • External state is known and consistent
  • Internal state is known and consistent

In order to keep my sanity, I want to put the responsibility of building that known starting point into a Standard Fixture. This fixture is responsible for creating that starting point, and it’s this starting point that ensures the long-term maintainability of my system.

Consistent internal state

Since I’m using AutoFixture for the creation and configuration of my fixture, it will be AutoFixture I use to build out my Standard Fixture. My standard fixture will be a single class in which my tests will interact with, and because the name “Fixture” is a bit overused in many libraries, I have to name my class somewhat specifically, and it will start with building out an isolated sandbox for my internal state:

public class SlowTestFixture
{
    private static IContainer Root = IoC.BuildCompositionRoot();

    public SlowTestFixture()
    {
        Container = Root.CreateChildContainer();
    }

    public IContainer Container { get; }
}

I use a DI container as my composition root in my systems, and this combined with child containers allows me to ensure that I have a unique, isolated sandbox for running my tests. The root container is my blueprint for an execution context, and represents what I do in production. The child container’s configuration, whatever I might do to it, lives only for the context of this one test.

Throughout the rest of my tests, I can access that container to build components as need be. The next piece I’ll need is to tell AutoFixture about this fixture, and to use it both when someone needs access to the context as well as when someone needs an instance of something.

In AutoFixture, this is done via fixture customizations:

public class SlowTestsCustomization : ICustomization
{
    public void Customize(IFixture fixture)
    {
        var contextFixture = new SlowTestFixture();

        fixture.Register(() => contextFixture);

        fixture.Customizations.Add(new ContainerBuilder(contextFixture.Container));
    }
}

Customizations alter behaviors of the AutoFixture’s fixture object, allowing me to add effectively new links in a chain of responsbility pattern. I want two behaviors added:

  • Access to the fixture
  • Building container-supplied instances

The first is simple, I can register individual instances with AutoFixture using the “Register” method. The second, since it depends on the type supplied, needs its own isolated customization:

public class ContainerBuilder : ISpecimenBuilder
{
    private readonly IContainer _container;

    public ContainerBuilder(IContainer container)
    {
        _container = container;
    }

    public object Create(object request, ISpecimenContext context)
    {
        var type = request as Type;

        if (type == null || type.IsPrimitive)
        {
            return new NoSpecimen(request);
        }

        var service = _container.TryGetInstance(type);

        return service ?? new NoSpecimen(request);
    }
}

AutoFixture calls each specimen builder, one at a time, and each specimen builder either builds out an instance or returns a null object, the “NoSpecimen” object.

Ultimately, the goal is to be able to have my tests to use a pre-built component, or to use the fixture as necessary:

public InvoiceApprovalTests(Invoice invoice,
    SlowTestFixture fixture,
    IInvoiceApprover invoiceApprover)
{
    _invoice = invoice;

    invoiceApprover.Approve(invoice);
    fixture.Save(invoice);
}

The last part I need to fill in is to modify Fixie to use my customizations when building up test instances. This is in my Fixie convention where I had previously configured Fixie to use AutoFixture to instantiate my test classes:

private object CreateFromFixture(Type type)
{
    var fixture = new Fixture();

    new SlowTestsCustomization().Customize(fixture);

    return new SpecimenContext(fixture).Resolve(type);
}

My tests now have an isolated sandbox for internal state, as each child container instance is isolated per fixture. If I need to inject stubs/fakes, I don’t affect any other tests because of how I’ve built the boundaries of my test in Fixie.

In the next post, I’ll look at isolating external state (the database).

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Cross-Platform AutoMapper (again)

Fri, 02/13/2015 - 17:04

Building cross-platform support for AutoMapper has taken some…interesting twists and turns. First, I supported AutoMapper in Silverlight 3.0 five (!) years ago. I did this with compiler directives.

Next, I got tired of compiler directives, tired of Silverlight, and went back to only supporting .NET 4.

Then in AutoMapper 3.0, I supported multiple platforms via portable class libraries. When that first came out, I started get reports of exceptions that I didn’t think should ever show up, but there was a problem. MSBuild doesn’t want to copy referenced assemblies that aren’t actually being used, so I’d get issues where you’d reference platform-specific assemblies in a “Core” library, but your “UI” project that referenced “Core” didn’t pull in the platform-specific assembly.

So began a journey to force the platform-specific assembly to get copied over, no matter what. But even that was an issue – I went through several different iterations of this before it finally, reliably worked.

Unless you’re on Xamarin, which doesn’t support using this method (of PowerShell scripts) to run install scripts on Mac.

Then I had a GitHub issue from Microsoft folks asking for CoreCLR support. And with vNext projects, the project itself describes the platforms to support, including all files in the directory. Meaning I wouldn’t be picking and choosing which files should be in the assembly or not. So, we’re back to square one.

A new path

With CoreCLR and the vNext project style that is folder-based rather than scattershot, pick-and-choose file based, I could only get CoreCLR support working by using conditional compiler directives. This was already in AutoMapper a few places, but mainly in files between the platform specific assemblies. I’ve always had to do a little bit of this:

image

Not absolutely horrible, but now with CoreCLR, I need to do this everywhere. To keep my sanity, I needed to include every file in every project. Ideally, I could just have the one portable library, but that won’t work until CoreCLR is fully released. With CoreCLR, I wanted to just have one single project that built multiple platforms. vNext class libraries can do this out-of-the-box:

image

However, I couldn’t move all platforms/frameworks since they’re not all supported in vNext class projects (yet). I still had to have individual projects.

Back when I supported Silverlight 3 for the first time, I abandoned support because it was a huge pain managing multiple projects and identical files. With vNext project files, which just includes all files in a folder without doing any explicit adding, I could have a great experience. I needed that with my other projects. The final project structure looked like this:

image

In the root PCL project, I’ll do all of the work. Refactoring, coding, anything. All of the platform-specific projects will just include all the source files to compile. To get them to do this, however, meant I needed to modify the project files to include files via wildcard:

image

My projects automatically include *all* files within folders (I needed to explicitly specify individual folders for whatever reason). With this configuration, my projects now include all files automatically:

image

I just have to be very careful that when I’m adding files, I only do this in the core PCL project, where files ARE added explicitly. There seems to be strange behavior that if I added a file manually to a project with wildcard includes, all of the files will be explicitly added. Not what I’d like.

Ultimately, this greatly simplified the deployment story as well. Each dependency only includes the one, single assembly:

image

At the end of the day, this deployment strategy is best for the users. I don’t have to worry about platform-specific extension libraries, GitHub issues about builds breaking in certain environments or the application crashing in cloud platforms.

If I had to do it over again, I’m not unhappy with the middle step I took of platform-specific extension assemblies. It forced me to modularize whereas with pure compiler directives I could have accepted a spaghetti mess of code.

Eventually, I’d like to collapse all into one project, but until it’s supported, this seems to work for everyone involved.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Clean Tests: Building Test Types

Thu, 02/05/2015 - 23:38

Posts in this series:

In the primer, I described two types of tests I generally run into in my systems:

  • Arrange/act/assert fully encapsulated in a single method
  • Arrange/act in one place, assertions in each method

Effectively, I build tests in a procedural mode or in a context/specification mode. In xUnit Test Patterns language, I’m building execution plans around:

There’s another pattern listed there, “Testcase Class per Feature”, but I’ve found it to be a version of one of these two – AAA in a single method, or split out.

Most test frameworks have some extension point for you to be able to accomplish both of these patterns. Unfortunately, none of them are very flexible. In my tests, I want to have complete control over lifecycle, as my tests become more complicated to set up. My ideal would be to author tests as I do everything else:

  • Method arguments for variation in a single isolated test
  • Constructor arguments for delivering fixtures for multiple tests

Since I’m using Fixie, I can teach Fixie how to recognize these two types of tests and build individual test plans for both kinds. We could be silly and cheat with things like attributes, but I think we can be smarter, right? Looking at our two test types, we have two kinds of test classes:

  • No-arg constructor, methods have arguments for context/fixtures
  • Constructor with arguments, methods have no arguments (shared fixture)

With Fixie, I can easily distinguish between the two kinds of tests. I could do other things, like key off of namespaces (put all fast tests in one folder, slow tests in another), or separate by assemblies, it’s all up to me.

But what should supply my fixtures? With most other test frameworks, the fixtures need to be plain – class with a no-arg constructor or similar. I don’t want that. I want to use a library in which I can control and build out my fixtures in a deterministic, flexible manner.

Enter AutoFixture!

I’ll teach Fixie how to run my tests, and I’ll teach AutoFixture how to build out those constructor arguments. AutoFixture is my “Arrange”, my code is the Act, and for assertions, I’ll use Shoudly (this one I don’t care as much about, just should-based is enough).

First, let’s look at the simple kinds of tests – ones where the test is completely encapsulated in a single method.

Testcase Class per Class

For TestClass Per Class, my Fixie convention is:

public class TestcaseClassPerClassConvention : Convention
{
    public TestcaseClassPerClassConvention()
    {
        Classes
            .NameEndsWith("Tests")
            .Where(t => 
                t.GetConstructors()
                .All(ci => ci.GetParameters().Length == 0)
            );

        Methods.Where(mi => mi.IsPublic && mi.IsVoid());

        Parameters.Add(FillFromFixture);
    }

    private IEnumerable<object[]> FillFromFixture(MethodInfo method)
    {
        var fixture = new Fixture();

        yield return GetParameterData(method.GetParameters(), fixture);
    }

    private object[] GetParameterData(ParameterInfo[] parameters, Fixture fixture)
    {
        return parameters
            .Select(p => new SpecimenContext(fixture).Resolve(p.ParameterType))
            .ToArray();
    }
}

First, I need to tell Fixie what to look for in terms of test classes. I could have gone a lot of routes here like existing test frameworks “Things with a class attribute” or “Things with methods that have an attribute” or a base class or a namespace. To keep things simple, I look for classes named “Tests”. Next, because I want to target a workflow where AAA is in a single method, I make sure that this class has only no-arg constructors.

For test methods, that’s a bit easy – I just want public void methods. No attributes.

Finally, I want to fill parameters from AutoFixture. I tell Fixie to add parameters from AutoFixture, resolving each parameter value one at a time from AutoFixture.

For now, I’ll leave the AutoFixture configuration alone, but we’ll soon be layering on more behaviors as we go.

With this in place, my test becomes:

public class CalculatorTests
{
    public void ShouldAdd(Calculator calculator)
    {
        calculator.Add(2, 3).ShouldBe(5);
    }

    public void ShouldSubtract(Calculator calculator)
    {
        calculator.Subtract(5, 3).ShouldBe(2);
    }
}

So far so good! Now let’s look at our Testcase Class per Fixture example.

Testcase Class per Fixture

When we want to have a single arrange/act, but multiple assertions, our test lifecycle changes. We now want to not have to re-run the Arrange/Act every single time, we only want it run once and then each Assert work off of the results of the Act. This means that with our test class, we want it only run/instantiated once, and then the asserts happen. This is different than parameterized test methods, where we want the fixture recreated with every test.

Our Fixie configuration changes slightly:

public class TestcaseClassPerFixtureConvention : Convention
{
    public TestcaseClassPerFixtureConvention()
    {
        Classes
            .NameEndsWith("Tests")
            .Where(t => 
                t.GetConstructors().Count() == 1
                && t.GetConstructors().Count(ci => ci.GetParameters().Length > 0) == 1
            );

        Methods.Where(mi => mi.IsPublic && mi.IsVoid());

        ClassExecution
            .CreateInstancePerClass()
            .UsingFactory(CreateFromFixture);
    }

    private object CreateFromFixture(Type type)
    {
        var fixture = new Fixture();

        return new SpecimenContext(fixture).Resolve(type);
    }
}

With Fixie, I can create as many configurations as I like for different kinds of tests. Fixie layers them on each other, and I can customize styles appropriately. If I’m migrating from an existing testing platform, I could even configure Fixie to run the existing attribute-based tests!

In the configuration above, I’m looking for test classes ending with “Tests”, but also having a single constructor that has arguments. I don’t know what to do with classes with multiple constructors, so I’ll just ignore those for now.

The test methods I’m looking for are the same – except now I’ll not configure any method parameters. It would be weird to combine constructor arguments with method parameters for this style of test, so I’m ignoring that for now.

Finally, I configure test execution to create a single instance per class, using AutoFixture as my test case factory. This is the piece that starts to separate Fixie from other frameworks – you can completely customize how you want your tests to run and execute. Opinionated frameworks are great – but if I disagree, I’m left to migrate tests. Not a fun proposition.

A test that uses this convention becomes:

public class InvoiceApprovalTests
{
    private readonly Invoice _invoice;

    public InvoiceApprovalTests(Invoice invoice)
    {
        _invoice = invoice;

        _invoice.Approve();
    }

    public void ShouldMarkInvoiceApproved()
    {
        _invoice.IsApproved.ShouldBe(true);
    }

    public void ShouldMarkInvoiceLocked()
    {
        _invoice.IsLocked.ShouldBe(true);
    }
}

The constructor is invoked by AutoFixture, filling in the parameters as needed. The Act, inside the constructor, is executed once. Finally, I make individual assertions on the result of the Act.

With this style, I can build up a context and incrementally add behavior via assertions. This is a fantastic approach for lightweight BDD, since I’m focusing on behaviors and adding them one at a time.

Next up, we’ll look at going one step further and integrating the database into our tests and using Fixie to wrap interesting behaviors around them.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs