Skip to content

Jimmy Bogard
Syndicate content
Strong opinions, weakly held
Updated: 2 hours 56 min ago

My OSS CI/CD Pipeline

Tue, 05/24/2016 - 23:39

As far back as I’ve been doing open source, I’ve borrowed other project’s build scripts. Because build scripts are almost always committed with source control, you get to see not only other projects’ code, but how they build, test and package their code as well.

And with any long-lived project, I’ve changed the build process for my projects more times than I care to count. AutoMapper, that I can remember, started off on NAnt (yes, it’s that old).

These days, I try to make the pipeline as simple as possible, and AppVeyor has been a big help with that goal.

The CI Pipeline

For my OSS projects, all work, including my own, goes through a branch and pull request. Some source control hosts allow you to enforce this behavior, including GitHub. I tend to leave this off on OSS, since it’s usually only me that has commit rights to the main project.

All of my OSS projects are now on the soon-to-be-defunct project.json, and all are either strictly project.json or a mix of main project being project.json and others being regular .csproj. Taking MediatR as the example, it’s entirely project.json, while AutoMapper has a mix for testing purposes.

Regardless, I still rely on a build script to execute a build that happens both on the local dev machine and the server. For MediatR, I opted for just a plain PowerShell script that I borrowed from projects online.  The build script really represents my build pipeline in its entirety, and it’s important for me that this build script actually live as part of my source code and not tied up in a build server. Its steps are:

  • Clean
  • Initialize
  • Build
  • Test
  • Package

Not very exciting, and similar to many other pipelines I’ve seen (in fact, I borrow a lot of ideas from Maven, which has a predefined pipeline).

The script for me then looks pretty straightfoward:

# Clean
if(Test-Path .\artifacts) { Remove-Item .\artifacts -Force -Recurse }

# Initialize
EnsurePsbuildInstalled

exec { & dotnet restore }

# Build
Invoke-MSBuild

# Test
exec { & dotnet test .\test\MediatR.Tests -c Release }

# Package
$revision = @{ $true = $env:APPVEYOR_BUILD_NUMBER; $false = 1 }[$env:APPVEYOR_BUILD_NUMBER -ne $NULL];
$revision = "{0:D4}" -f [convert]::ToInt32($revision, 10)

exec { & dotnet pack .\src\MediatR -c Release -o .\artifacts --version-suffix=$revision }

Cleaning is just removing an artifacts folder, where I put completed packages. Initialization is installing required PowerShell modules and running a “dotnet restore” on the root solution.

Building is just msbuild on the solution, executed through a PS module, which MSbuild defers to the dotnet CLI as needed. Testing is the “dotnet test” command against xUnit. Finally, packaging is “dotnet pack”, passing in a special version number I get from AppVeyor.

As part of my builds, I include the incremental build number in my packages. Because of how SemVer works, I need to make sure the build number is alphabetical, so I prefix a build number with leading zeroes, and “57” becomes “0057”.

In my project.json file, I’ve set the version up so that the version from the build gets substituted at package time. But my project.json file determines the major/minor/revision:

{
  "version": "4.0.0-beta-*",
  "authors": [ "Jeremy D. Miller", "Joshua Flanagan", "Josh Arnold" ],
  "packOptions": {
    "owners": [ "Jeremy D. Miller", "Jimmy Bogard" ],
    "licenseUrl": "https://github.com/HtmlTags/htmltags/raw/master/license.txt",
    "projectUrl": "https://github.com/HtmlTags/htmltags",
    "iconUrl": "https://raw.githubusercontent.com/HtmlTags/htmltags/master/logo/FubuHtml_256.png",
    "tags": [ "html", "ASP.NET MVC" ]
  },
}

This build script is used not just locally, but on the server as well. That way I can ensure I’m running the exact same build process reproducibly in both places.

The CD Pipeline

The next part is interesting, as I’m using AppVeyor to be my CI/CD pipeline, depending on how it detects changes. My goal with my CI/CD pipeline are:

  • Pull requests each build, and I can see their status inside GitHub
  • Pull requests do not push a package (but can create a package)
  • Merges to master push packages to MyGet
  • Tags to master push packages to NuGet

I used to push *all* packages to NuGet, but what wound up happening is I would have to move slower with changes because things would just “show up” on people before I had a chance to fully think through the changes to the public.

I still have pre-release packages, but these are a bit more thought-out than I have had in the past.

Finally, because I’m using AppVeyor, my entire build configuration lives in an “appveyor.yml” file that lives with my source control. Here’s MediatR’s:

version: '{build}'
pull_requests:
  do_not_increment_build_number: true
branches:
  only:
  - master
nuget:
  disable_publish_on_pr: true
build_script:
- ps: .\Build.ps1
test: off
artifacts:
- path: .\artifacts\**\*.nupkg
  name: NuGet
deploy:
- provider: NuGet
  server: https://www.myget.org/F/mediatr-ci/api/v2/package
  api_key:
    secure: zKeQZmIv9PaozHQRJmrlRHN+jMCI64Uvzmb/vwePdXFR5CUNEHalnZdOCg0vrh8t
  skip_symbols: true
  on:
    branch: master
- provider: NuGet
  name: production
  api_key:
    secure: t3blEIQiDIYjjWhOSTTtrcAnzJkmSi+0zYPxC1v4RDzm6oI/gIpD6ZtrOGsYu2jE
  on:
    branch: master
    appveyor_repo_tag: true

First, the build version I set to be just the build number. Because my project.json file drives the package/assembly version, I don’t need anything more complicated here. I also don’t want any other branches built, just master and pull requests. This makes sure that I can still create branches/PRs inside the same repository without having to be forced to use a second repository.

The build/test/artifacts should be self-explanatory, I want everything flowing through the build script so I don’t want AppVeyor discovering and trying to figure things out itself. Explicit is better.

Finally, the deployments. I want every package to go to MyGet, but only tagged commits to go to NuGet. The first deploy configuration is the MyGet configuration, that deploys only on master to my MyGet configuration (with user-specific encrypted API key). The second is the NuGet configuration, to only deploy if AppVeyor sees a tag.

For public releases, I:

  • Update the project.json as necessary, potentially removing the asterisk for the version
  • Commit, tag the commit, and push both the commit and the tag.

With this setup, the MyGet feed contains *all* packages, not just the CI ones. The NuGet feed is then just a “curated” feed of packages of more official packages.

The last part of a release, publicizing it, is a little bit more work. I still like GitHub releases but haven’t found yet a great way of automating a “tag the commit and create a release” process. Instead, I use the tool GitHubReleaseNotes to create the markdown of a release based on the tags I apply to my issues for a release. Finally, I’ll make sure that I update any documentation in the wiki for a release.

I like where I’ve ended up so far, and there’s always room for improvement, but it’s a far cry from when I used to have to manually package and push to NuGet.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

AutoMapper 5.0 Beta released

Wed, 05/18/2016 - 20:21

This week marks a huge milestone in AutoMapper-land, the beta release of the 5.0 work we’ve been doing over the last many, many months.

In the previous release, 4.2.1, I obsoleted much of the dynamic configuration API in favor of an explicit configuration step. That means you only get to use “Mapper.Initialize” or “new MapperConfiguration”. You can still use a static Mapper.Map call, or create a new Mapper object “new Mapper(configuration)”. The last 4.x release really paved the way to have a static and instance API that were lockstep with each other.

In previous versions of AutoMapper, you could call “Mapper.CreateMap” anywhere in your code. This made two things difficult: performance optimization and dependent configuration. You could get all sorts of weird bugs if you called the configuration in the “wrong” order.

But that’s gone. In AutoMapper 5.0, the configuration is broken into two steps:

  1. Gather configuration
  2. Apply configuration in the correct order

By applying the configuration in the correct order, we can ensure that there’s no order dependency in your configuration, we handle all of that for you. It seems silly in hindsight, but at this point the API inside of AutoMapper is strictly segregated between “DSL API”, “Configuration” and “Execution”. By separating all of these into individual steps, we were able to do something rather interesting.

With AutoMapper 5.0, we are able to build execution plans based on type map configuration to explicitly map based on exactly your configuration. In previous versions, we would have to re-assess decisions every single time we mapped, resulting in huge performance hits. Things like “do you have a condition configured” and so on.

A strict separation meant we could overhaul the entire execution engine, so that each map is a precisely built expression tree only containing the mapping logic you’ve configured. The end result is a 10X performance boost in speed, but without sacrificing all of the runtime exception logic that makes AutoMapper so useful.

One problem with raw expression trees is that if there’s an exception, you’re left with no stack trace. When we built up our execution plan in an expression tree, we made sure to keep those good parts of capturing context when there’s a problem so that you know exactly which property in exactly which point in the mapping had a problem.

Along with the performance issues, we tightened up quite a bit of the API, making configuration consistent throughout. Additionally, a couple of added benefits moving to expressions:

  • ITypeConverter and IValueResolver are both generic, making it very straightforward to build custom resolvers
  • Supporting open generic type converters with one or two parameters

Overall, it’s been a release full of things I’ve wanted to tackle for years but never quite got the design right. Special thanks to TylerCarlson1 and lbargaoanu, both of whom passed the 100 commit mark to AutoMapper.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Entities aren’t resources, resources aren’t representations

Thu, 05/12/2016 - 18:29

One of the easy mistakes in building a REST API is trying to take your rows out of the database and expose them directly as JSON. Such technology exists, where you can directly expose stored procedures as SOAP web services, or protocols like OData.  You can also just expose your entities directly as JSON through a web service, but you’re missing out on some big distinctions of resources and representations.

So first, what is a resource? That’s actually quite easy – a resource is anything with a URL. The converse is not true, a URL is not a resource, mainly because the URL is the Uniform Resource Locator. If you can locate a resource, then it exists. And Fielding was pleased.

The representation is a bit different – it describes the current state of the resource (when requested).

So what does this have to do with entities? Well, nothing. There is no relationship between entities and resources. And that’s a good thing, it’s exactly how the web works.

When you navigate to a web page, an online retailer, and look at a list of products, what is the resource? From the client’s perspective, we don’t know. We have no idea of the implementation details of the resource, other than a way to locate it and request a representation. Again, Fielding was pleased, as this decoupled us from the implementation details of the resource.

So where does this leave us when building APIs?

A decoupled API

In APIs that serve the client according to their needs, the resource often includes details from many different data sources, combined together to build a representation to the end user. A REST API builds all this together:

image

In hypermedia APIs I build, the resource state combines multiple entities together from one or more data sources into the state of the resource. This plus links, forms a queries makes a “resource”, though it’s still a bit more abstract than that. Finally I build out the representation, perhaps JSON API, HAL, Siren etc.

If I made my entities equivalent to resources, I’m likely forcing clients to make many round trips to all the entities they need, but even more, I’m directly exposing the implementation details of my service to the world. Perhaps I’m removing some fields here and there, but in reality this is only one step above handing clients a connection string to my database. Sure, it’s easy and convenient to expose entities directly as resources and representations, but there’s some serious coupling that comes with that choice that you must accept.

In my experience, if I approach the API design strictly from the client perspective, on what they’re trying to achieve and why, it’s actually quite rare that I arrive at an API design that is simply database CRUD exposed as HTTP. And Fielding was pleased, as this design most closely matches the everyday use of the web. Makes sense. That’s what REST is about – a set of architectural constraints describing basically, the web.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs

Validation inside or outside entities?

Fri, 04/29/2016 - 21:45

A common question I get asked, especially around a vertical slice architecture, is where does validation happen? If you’re doing DDD, you might want to put validation inside your entities. But personally, I’ve found that validation as part of an entity’s responsibility is just not a great fit.

Typically, an entity validating itself will do so with validation/data annotations on itself. Suppose we have a Customer and its First/Last names are “required”:

public class Customer
{
    [Required]
    public string FirstName { get; set; }
    [Required]
    public string LastName { get; set; }
}

The issue with this approach is twofold:

  • You’re mutating state before validation, so your entity is allowed to be in an invalid state.
  • There is no context of what the user was trying to do

So while you can surface these validation errors (typically from an ORM) to the end user, it’s not easy to line up the original intent with the implementation details of state. Generally I avoid this approach.

But if you’re all up in DDD, you might want to introduce some methods to wrap around mutating state:

public class Customer
{
  public string FirstName { get; private set; }
  public string LastName { get; private set; }
    
  public void ChangeName(string firstName, string lastName) {
    if (firstName == null)
      throw new ArgumentNullException(nameof(firstName));
    if (lastName == null)
      throw new ArgumentNullException(nameof(lastName));
      
    FirstName = firstName;
    LastName = lastName;
  }
}

Slightly better, but only slightly, because the only way I can surface “validation errors” are through exceptions. So you don’t do exceptions, you use some sort of command result:

public class Customer
{
  public string FirstName { get; private set; }
  public string LastName { get; private set; }
    
  public CommandResult ChangeName(ChangeNameCommand command) {
    if (command.FirstName == null)
      return CommandResult.Fail("First name cannot be empty.");
    if (lastName == null)
      return CommandResult.Fail("Last name cannot be empty.");
      
    FirstName = command.FirstName;
    LastName = command.LastName;
    
    return CommandResult.Success;
  }
}

Again, this is annoying to surface to the end user because I have one validation error at a time being returned. I can batch them up, but how do I correlate back to the field name on the screen? I really can’t. Ultimately, entities are lousy at command validation. Validation frameworks, however, are great.

Command validation

Instead of relying on an entity/aggregate to perform command validation, I entrust it solely with invariants. Invariants are all about making sure I can transition from one state to the next wholly and completely, not partially. It’s not actually about validating a request, but performing a state transition.

With this in mind, my validation centers around commands and actions, not entities. I could do something like this instead:

public class ChangeNameCommand {
  [Required]
  public string FirstName { get; set; }
  [Required]
  public string LastName { get; set; }
}

public class Customer
{
  public string FirstName { get; private set; }
  public string LastName { get; private set; }
    
  public void ChangeName(ChangeNameCommand command) {
    FirstName = command.FirstName;
    LastName = command.LastName;
  }
}

My validation attributes are on the command itself, and only when the command is valid do I pass it to my entities for state transition. Inside my entity, I’m responsible for successfully accepting a ChangeNameCommand and performing the state transition, ensuring my invariants are satisfied. In many projects, I wind up using FluentValidation instead:

public class ChangeNameCommand {
  public string FirstName { get; set; }
  public string LastName { get; set; }
}

public class ChangeNameValidator : AbstractValidator<ChangeNameCommand> {
  public ChangeNameValidator() {
    RuleFor(m => m.FirstName).NotNull().Length(3, 50);
    RuleFor(m => m.LastName).NotNull().Length(3, 50);
  }
}

public class Customer
{
  public string FirstName { get; private set; }
  public string LastName { get; private set; }
    
  public void ChangeName(ChangeNameCommand command) {
    FirstName = command.FirstName;
    LastName = command.LastName;
  }
}

The key difference here is that I’m validating a command, not an entity. And since entities themselves are not validation libraries, it’s much, much cleaner to validate at the command level. Because the command is the form I’m presenting to the user, any validation errors are easily correlated to the UI since the command was used to build the form in the first place.

Validate commands, not entities, and perform the validation at the edges.

Post Footer automatically generated by Add Post Footer Plugin for wordpress.

Categories: Blogs