Scaling Agile appears to be a common topic these days. Of course there are good advices and bad advices on how to do that. But how do you know which is which? A few weeks back I came up with the idea of an anti-maturity model. If you have dealt with a few maturity models in the past, these usually run from level 1 to level 5, where level 5 means more mature. My anti-maturity model runs differently, with level 0 indicating that you are probably on the right track, and level MAX_INT that you are probably not doing to well. Why does this scale run differently? As part of my work, I realized there is always someone out there who can come up with an even worse way of doing things than that other guy that I thought was worst.Level 5 – The Brooks
You realize that the constraint of 3-9 team members is artificial. Since you are a large organization, you need to have adaptations, and that includes team sizes of 100 team members. They already work this way for decades. Changing a working team is going interrupt their productivity. That’s not something you want to put at risk just by going to another methodology.Level 4 – The Blue-printers
Scaling Agile is easy at this level. All you need to do is to use Scrum since almost everyone uses it. Then you run your Sprints starting with Sprint 1 – Requirements, Sprint 2 is on Architecture. Sprint 3 is on Design, Sprint 4-5 is on Programming, and Sprint 6 is to find the one to blame. Since you waste a lot of your time doing that, you won’t ship any software at all.
Of course, you can run the sprints in parallel, since you will stick with your requirements department, enterprise architects, and designers.Level 3 – The Conways
Some of your IT teams are using agile. The larger organization though uses proven methods like project plans to adhere to that. Your teams are not cross-functional. Instead, you still have a department in place for all the testing, and you are going to stick with that, since you can’t release the crap the teams build, anyways.
Your architecture in the software maps directly to the organizational structure. You make sure to stick to those structure by introducing additional roles, like a technical counterpart for the business-facing ProductOwner that gives out architecture stories to your teams.Level 2 – The C3POs
You realize that you have a lot of people involved. The concept of Scrum of Scrums is too limited for you to work. Since you have 42 management levels involved, you should go for a Scrum of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums of Scrums, so that all your important people can serve as a ProductOwner of a ProductOwner of a ProductOwner of a ProductOwner of a ProductOwner, and so on. The technical term for that model is Chief ProductOwner, and your most senior one is afterwards called the CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCPO. You have the same structure in your ScrumMaster organization. The top-most person there is called the Master of Desaster.Level 1 – The Downstreamers
You have tiny little teams which work together – your database team creates the database first, so that in the second sprint your business domain team can code the business domain, and your UI guys will be dealing with the user interface in sprint 3. Your ProductOwners are self-organizing, and driven by their own project managers. Your product is driven by three customers, and your ProductOwners deal with the necessary re-prioritzation for that – each on their own.Level 0 – Emergent Scaling
You realize that scaling agile is a complex problem. That means that you need emergent practices, and can’t work from a blue-print or project plan. You realize that you should probe the underlying organization for the changes necessary, and see what happens as you incorporate little experiments while maybe scaling the organization to use Agile all over the place – including accounting, marketing, and sales.
You don’t focus on one agile methodologies or framework, but make local adaptations as necessary to provide the biggest customer benefits all the way through. Your teams deliver business value from end to end, and each of the teams works as their own start-up in the larger organization. Since you have decoupled the whole organization, you don’t need many people while still being able to ship products every second of the day.Conclusion
If you have read thus far, congratulations. If you find there is still something you want to add, please feel free to do so.
In case you want to outline this model to your management, I also came up with a name, an abbreviation for it: ScAAMM.
As all these years, I went to the Agile Testing Days in Potsdam, Germany. On Tuesday evening I received an award for being the “Most Influential Agile Testing Professional Person” 2013. Since this is an award based upon votes on the internet, I want to say thank you for voting for me.
I had prepared a speech, but didn’t deliver it fully. Here is the full thing that I prepared.
Back in 2009 I read a blog entry. A guy named Zeger van Hese wrote it on his blog. He was writing about a shy guy speaking at a conference. That guy had a quite compelling story to tell about how his team of testers discovered agile development practices, and implemented them. The guy seemed pretty nervous when he talked, Zeger wrote. That guy was me, having my first conference talk at Agile Testing Days 2009. I was anxious with all these people that influenced me in the room: Lisa Crispin, Elisabeth Hendrickson, and the evening before that I demoed my talk to Gojko Adzic and Mike Scott.
Skip forward a few years, and I am happy to receive this award. The Most Influential Agile Testing Person. Thank you.
I remember when I was confronted with being influential for the first time in my life. I was still young, a 16 year old teenager. I just gave up swimming, and stepped into a coaching position at my local swimming club. I also stepped up to organize our youth groups.
At that time I had all the terrible ideas in mind that teenagers come up with. I was smoking, drinking beer, partying all night, etc. At some point one of our more senior people took me aside and told me that it’s not a good idea to stand in front of a group of kids with a cigarette in hand. That was when I realized that I indeed had influence with my behavior on others – and that I needed to be cautious with stupid actions.
Boy, was I scared. Receiving this award – Most Influential Agile Testing Person – reminds me about this fact – and it still scares me.
Back in late 2008 I have been involved with the uprising of the Software Craftsmanship movement. From the mailing list, I remember a lively discussion from those early days. The model we had in mind was that of the ancient crafts with their mentor and apprentice model to educate the next generation of software developers. In one discussion we pointed out to the people that influenced us in order to point to our self-picked mentors.
Today, this list would be endless for me. James Bach, Michael Bolton, Elisabeth Hendrickson, Gojko Adzic, Jerry Weinberg, Johanna Rothman, my colleagues at it-agile, past and present, Matt Barcomb, Brett Schuchert, Uncle Bob Martin, Micah Martin, Steve Freeman, Kent Beck, Michael Feathers, Alistair Cockburn, Andreas Leidig, Doug Bradbury, ….
I think this list will never be complete, but these folks influenced me largely – and most of them still do.
When I took a look into the program on Saturday evening to see who will be here, and who will be speaking, I was amazed by the amount of people attending Agile Testing Days this year that influenced me. Lisa Crispin, Janet Gregory, Dan North, my early mentor Matt Heusser, my first apprentice Meike Mertsch, Tony Bruce, Ajay Balamurugadas who introduced me to Weekend Testing, Huib Schoots, Jean-Paul Varwijk, Anna Royzman, Pete Walen, James Lyndsay, Bart Knaack, the TestLab gurus, Seb Rose, J.B. Rainsberger, … even this list seems infinite, and it’s hard for me to not run into a familiar face in the hallways at this conference.
What that look on the program on Saturday evening taught me is that this conference heavily influenced me. It seems to be a large family reunion. It always has felt like that in the years past. That said, keep on being influenced, maybe by me, and also by these great people that I learned so much from on my own.
“Where should our developers book their hours on when we move to Scrum?” I was asked recently at a client. I am helping them roll out their new development methodology which leverages a big deal of Scrum among 17 teams. One of the questions in the larger organization was, how to do time-tracking. I knew I needed to dive deeper into that.
“Why do you need time-tracking?” “Well, you see we have customer projects alongside with the development of our product platform. Customer projects run in two phases: before feature complete, and going for the real deadline one year later. During the latter phase customers file a lot of bugs when they start testing the functionalities we delivered earlier. Sometimes we want to fix a bug for all customers, then this becomes product development effort. Otherwise we charge the project.
“We have customers that are troublesome. In the end, we want to decide which customers don’t pay off well. That’s why we want to keep track of how much development work flows into each of the customer projects.”
A long story short, the 17 teams might find work from either customer project or the platform product in the backlog. A sprint might then be filled with either a mix of items from all of these sources, or it might be just product, or just bug-fixing. Depending on the demand from the customers. Oh, they decided to start with component teams, I think I should say that as well.
I don’t think we came up with a solution that no one else in the world thought of, yet, I found it clever. We came up with the decision that the Product Owners of each of the 17 teams can deliver percentage numbers – how much work did my team for customer X, customer Y, and how much for the product – after the Sprint, or at least on a monthly basis (as this appeared to be the reporting cadence). Let’s look at the benefits of this approach.Improved accounting
The development team costs the same amount of cash each sprint. In order to do the math for accounting, it’s sufficient to track the development on this granularity level and after the fact. In fact, it turned out to be more precise than anything in place.
Before our discussion developers also fixed bugs. Usually when it came to tracking their time the developers were struggling with where to book the particular 10 minute bugfix. Did the product manager approve to include that in his budget? Is this a bugfix just for customer X, even though customer Y needs this as well? In the end, developers ended up tracking their bugfixing efforts for the budget that was easily available to them – and that usually did not have to do anything with booking hours to the right cost center.
That said, with the old approach they were hardly able to answer their initial question: Which customer projects pay off, and which should we abandon in the future? The numbers were ambiguous because the structure behind it did not support the people who needed to use the system. With the new approach, the numbers were still vague, but a bit more reliable when it comes to answering the initial question – and therefore more helpful.Empowering of Product Owners
The Product Owners become more empowered since they now have real accountability in regards to how they juggle the different efforts with their ordering in their product backlogs. Since they should report the percentage split among the different cost centers, they now have to deal with the accounting side of the efforts. They are now really in charge to care for the money that their development team costs, and might even have to negotiate those numbers with the project managers for the different customer projects.
The development team costs the same amount of cash each sprint. After the Sprint it reviewed, the Product Owners have a clear picture about which bugs are fixed, which backlog items have been completed (and which maybe didn’t), and how much effort was put into each of the different cost centers. By negotiating the numbers on the larger portfolio level, they now even need to really think about where to get money from. I expect these conversations to change over time into “we need to charge customer X for this thing more than we thought” or something like that.Easier time tracking for developers
The initial thought before involving me was to create cost centers for all customer projects, and each component team. Together with separating between development efforts, bug fixing efforts, and meeting efforts this would have become a nightmare for every developer involved in the teams.
Now, the proposal included that each component team simply gets one dedicated cost center. Developers book all their hours one effort: “My Scrum Team”. We even dropped the separation between “development”, “bug fixing”, and “meetings”. That means that the overhead of the Scrum meetings, or any larger organizational meetings is now spread across the different customer projects in fair amounts. Thereby the particular pick of development methodology also flows into the contracts for future customer projects.Simplicity
I think we did not re-invent time-tracking. In fact I am quite sure that someone already did something like that. I like the approach as it answers a reasonable question from the business perspective while it favors simplicity. The development manager talked right away with accounting and the HR department whether the approach would be feasible, and they want to give a try.
At the Global Scrum Gathering in Paris, France, Henrik Kniberg kicked off the first day with his opening keynote. He was speaking about how culture overcomes process – whether you want to or not.
Kniberg defined culture are the stuff that people do without noticing it. When people carry their bodies to the Daily Stand-up just because the ScrumMaster tells them, that is process. When people get to the Daily Stand-up because that is how we do software development around, then that is culture.
Kniberg explained that an Agile culture leads to better products and happier employees. In the end, we end up in a better world. (I missed some unicorns at this point.)
Kniberg described how fragile Agile is. He showed the story from the Swedish police which he wrote about in Lean and Agile from the trachnes. They started a project to build a software for the officers on the streets. They introduced Agile & Lean, a gradual rollout, real users were involved, bottom-up decision making, value-driven and a suitable tech platform. The project ended in a media success with happy users and a happy team. Even they won an CIO award in the end.
Skip forward a few years, and the project was overtaken by the surrounding organization. They implemented waterfall with big-bang rollouts, and an inappropriate tech platform. Real users were no longer involved, and decisions were made in a top-down manner. Overall they changed from value- to cost-driven. It was a media disaster with furious users and furious team members. This is how you can burn one billion euros, Kniberg concluded.
Kniberg continued with the story of Spotify to show a positive example of a great culture. Even though Spotify doubles their employee numbers, they have industry’s top-notch employee happiness. Why?
Kniberg explained the Shu-Ha-Ri model. At Shu-level you follow the rules, at Ha level you start to adapt the rules as you see fit, at Ri-level you ignore the rules. Kniberg cited Scrumbutophobia, that is the fear of doing Scrum wrong. The pre-valent symptom for Scrumbutophobia is a Scrum implementation being stuck in the Shu-level focusing on the practices alone, rather than the principles. For Spotify, Kniberg explained that they focused on five principles from the Agile world, these are Continuous Improvement, Iterative development, trust, simplicity, and servant leadership. However, they found a sixth principle that had great impact: autonomous teams.
In Spotify teams are called squads. Autonomous squads follow a mission, are small, and co-located. Squads organize themselves and have full end-to-end resposibility for the stuff they build – from design to commit to deploy to maintenance. Each squad has a mission. Within the scope of its mission, a squad is empowered to decide what to build, how to build it, and how to work together while doing it. Spotify steers the autonomy of squads with their mission. The broader the mission, the more autonomy the squad has. Kniberg cited Pink from his book Drive that motivation comes from autonomy, master and purpose. That is why autonomy has such a large role at Spotify.
Squads follow the guideline to be autonomous, but don’t suboptimize. Kniberg showed some examples from their office how they get this in place, like hang-out areas for cross-squad collaboration, sprint demos and open discussions.
Kniberg continued by explaining the relationship between autonomy and alignment. He explained with a 2×2 matrix that high autonomy and high alignment lead to an innovative organization, and collaborative culture rather than conformance or people diverting into different directions. One way they applied that at Spotify was to measure dependencies between squads by asking the people.
All squads at Spotify own their own quality, they sit together, have a mission, and a ProductOwner in their team. They also follow an Agile approach. Most squads do retrospectives, have a physical taskboard, do demos. Most squads also either use Scrum, or Kanban, or both, and have an Agile Coach alongside with Daily Stand-Up meetings. Some squads rely on estimates, and measure their velocity. Some also have Scrum of Scrums meetings, and maintain burn-up or burn-down charts.
Kniberg showed an example from their happiness survey. The email contained the results from the latest happiness survey: 91% happy, 4% not happy. Instead of putting up how remarkable that is, the email stated that this is not sufficient for them, and if you happen to be among the 4% of the people being unhappy, please get in touch. This is how you nurture a happiness culture.
Kniberg continued with concepts that I had heard at other Spotify talks before. If you happen to be at a conference with some Spotify talk, I recommend going there. To me it did not have too much to do with culture, but more with how they cared about their people. Kniberg explained how the concept of tribes combines squads, and how chapters came into play at a certain point of scaling.
Kniberg showed the vicious cycle of most release processes in place. Releasing is hard, so you release seldom. Because you pile up so much stuff to release, releasing it becomes hard, of course, and this is where the vicious cycle closes. On the other hand, if you make releasing easy, you release more often automatically. One way Spotify achieves this is by decoupling as much as possible.
Kniberg continued with the importance or trust. He explained how fear can kill motivation. He cited a CEO from some other large company that said
The reward for doing a good job today is having a job tomorrow.
Of course, such a culture leads to fear, and blaming.
Instead Kniberg showed how important failure is for learning. Citing facebook’s “move fast, and break things” is something you need to build inside your culture. At Spotify they have retrospectives and post-mortems focused not on blaming but on learning. They also have blog entries on their internal blog starting with “how we shot ourselves in the foot” that expose that learning to the larger organization, and keep the blaming out the door. One thing that helps with this is to have a limited blast radius for all the components. If something goes wrong, not the whole platform goes down, but just the single piece that squad is responsible for. Of course the squad then knows they need to fix that immediately.
Kniberg continued on why value & impact is more important than velocity. He explained how Spotify embodied the Lean Startup cycle in their culture. They start with an idea or problem. They build a narrative and a prototype to solve the problem. Then they build a minimal viable product, which they deploy, and analyze data to learn quickly.
Kniberg showed a continuum between predictability and innovation. He put waterfall methodologies on the end with the predictability. Typically Scrum lies on the middle of that continuum. Kniberg stated that Scrum at Spotify lies more on the innovation side than typical Scrum implementations. They unleash innovation with hack days, and hack weeks which may take up to 10% of employee time.
Kniberg paraphrased empirical process improvement mind-set that experiments & data are mote important than discussions and opinions. In an experiment-friendly culture, for example, when deciding between Tool A or Tool B, you try both, and compare them after some time, and then make the decision. Kniberg described that at Spotify they got rid of meetings that were useless for them. They also don’t have PMO & PM roles, time reporting, hand-offs, acceptance test phase, task estimates, and all the other things you may find at other corporations. Instead they decised to keep retrospective, daily standups, google docs, git, and go to conferences. Kniberg also showed how the put improvement boards and the “Definition of Awesome” in place with multiple Kanban boards for improvements where you could zoom into a ticket to follow the whole improvement process, and the experiments they ran before.
Kniberg cited lessons they learned from big projects. The first mantra is to avoid big projects whenever possible. If you can’t avoid them, then sync daily to resolve squad dependencies, and demo your product weekly in order to evaluate the integrated product.
Kniberg described that they also ran a big experiment: a personal bonus system. They found out it doesn’t work since it was a large failure. Another bit experiment Spotify ran was a tech-wide hackweek. One whole week everyone in tech (300 people) could build whatever they wanted, with whomever they wanted, in however way they wanted. They also had a demo & party on Friday. It was a large success.
Kniberg continued how to spread and nurture the culture. They have dedicated roles for culture and improvement. There are Agile coaches for some squads alongside with the People Operations team that focuses on the culture. They together form the Agile coaching community. One element to grow their culture is that every new employee is sent on a boot camp for one week. After that week, they put software into production in their second week, and demo it. Another example in Spotify’s culture are social groups, like board games, movie visits, and so on.
Kniberg concluded with their pain point. They have unstable squads, which means new squads are formed regularly anew, or new people come into the squad restarting the whole team building, eventually. Also, scaling breaks stuff all the time so that yesterday’s “brilliant solution” is today’s impediment. Cross-timezone collaboration and technical debt are two other points Kniberg mentioned. But, Spotify found out that culture has a healing effect. Although all these points seem painful, they do not last for long at Spotify.
A lot from Kniberg’s talk reminded me about the Spotify talk I heard at Agile 2013. I also could see things at it-agile that we struggle with, and that work for us. I think I can take some of the factors like alignment and autonomy back to implement them in our company.