Imagine this hypothetical conversation I didn’t have with someone last week…
THEM: “Is there a DevOps framework?”
ME: “Noooooo, it doesn’t work like that”
ME: “Well DevOps is more like a philosophy, or a set of values and principles. The way you apply those principles and values varies from one organisation to the next, so a framework wouldn’t really work, especially if it was quite prescriptive, like Scrum”
THEM: “But I really want one”
ME: “Ok, I’ll tell you what, I’ll hack an existing framework to make it more devopsy, does that work for you?”
THEM: “Take my money”
So, as you can see, in a hypothetical world, there is real demand for a DevOps framework. The trouble with a DevOps framework, as is always the problem with anything to do with DevOps, nobody can actually agree what the hell DevOps means, so any framework is bound to upset a whole bunch of people who simply disagree with my assumption of what DevOps means.
So, with that massive elephant in the room, I’m just going to blindly ignore it and crash on with this experimental little framework I’m calling DevOpScrum.
In my view (which is obviously the correct view) DevOps is a lot more than just automation. It’s not about Infrastructure as Code and Containers and all that stuff. All that stuff is awesome and allows us to do things in better and faster ways than we ever could before, but it’s not the be-all-and-end-all of DevOps. DevOps for me is about the way teams work together to extract greater business value, and produce a better quality solution by collaborating, working as an empowered team, and not blaming others (and also playing with cool tools, obvs). And if DevOps is about “the way teams work together” then why the hell shouldn’t there be a framework?
The best DevOps framework is the one a team builds itself, tailored specifically for that organisation’s demands, and sympathetic to its constraints. Incidentally, that’s one reason why I like Kanban so much, it’s so adaptable that you have the freedom to turn it into whatever you want, whereas scrum is more prescriptive, and if you meddle with it you not only confuse people, you anger the Scrum gods. However, if you don’t have time to come up with your own DevOps framework, and your familiar with Scrum already, then why not just hack the Scrum framework and turn it into a more DevOps-friendly solution?
Which brings us nicely to DevOpScrum, a DevOps Framework with all the home comforts of Scrum, but with a different name so as not to offend Scrum purists.
The idea with DevOpScrum is to basically extend an existing framework and insert some good practices that encourage a more operational perspective, and encourage greater collaboration between Dev and Ops.
How does it work?
Start by taking your common-or-garden Scrum framework, and then add the following:
Operability features on the backlog
A definition of Done that includes “deployable, monitored, scalable” and so on (i.e doesn’t just focus on “has the product feature been coded?”)
Continuous Delivery as a mandatory practice!
And there you have it. A scrum-based DevOps Framework.
Let’s look into some of the details…
We’ll start with The Team…
A product owner (who appreciates operability – what we once called “Non-Functional Requirements in the olden days. That term is so not cool anymore. It’s less cool than bumbags).
Devs, Testers, BAs, DBAs and all the usual suspects.
Infrastructure/Ops people. Some call them DevOps people these days. These are people who know infrastructure, networking, the cloud, systems administration, deployments, scalability, monitoring and alerting – that sort of stuff. You know, the stuff Scrum forgot about.
Roles & Responsibilities
Pretty similar to scrum, to be fair. The Product Owner has ultimate responsibility for deciding priorities and is the person you need to lobby if you think your concerns need to be prioritised higher. For this reason, the Product Owner needs to understand the importance of Operability (i.e the ability to deploy, scale, monitor, maintain and so on), which is why I recommend Product Owners in a DevOps environment get some good DevOps training (by pure coincidence we run a course called “The DevOps Product Owner” which does exactly what I just described! Can you believe that?!).
There’s no scrum master in this framework, because it isn’t scrum. There’s a DevOpScrum coach instead, who basically does the scrum master coach and is responsible for evangelising and improving the application of the DevOps values and principles.
DevOps Engineers – One key difference in this framework is that the team must contain the relevant infrastructure and Ops skills to get stuff done without relying on an external team (such as the Ops team or Infrastructure team). This role will have the skills to provide Continuous Delivery solutions, including deployment automation, environment provisioning and cloud expertise.
Yep, there’s sprints. 2 weeks is the recommended length. Anything longer than that and it’s hardly a sprint, it’s a jog. Whenever I’ve worked in 3 week sprints in the past, I’ve usually seen people take it really easy in the first couple of weeks, because the end of the sprint seemed so far away, and then work their asses off in the final week to hit their commitments. It’s neither efficient nor sustainable.
Another big difference with scrum is that the Product Backlog MUST contain operability features. The backlog is no longer just about product functionality, it’s about every aspect of building, delivering, hosting, maintaining and monitoring your product. So the backlog will contain stories about the infrastructure that the application(s) run on, their availability rates, disaster recovery objectives, deployability and security requirements (to name just a few). These things are no longer assumed, or lie outside of the team – they are considered “first class citizens” so to speak.
I recommend twice-weekly backlog grooming sessions of about an hour, to make sure the backlog is up-to-date and that the stories are in good shape prior to Sprint Planning.
Because the backlog is different, sprint planning will be subtly different as well. Obviously we’ve got a broader scope of stories to cover now that we’ve got operational stories in the backlog, but it’s important that everyone understands these “features”, because without them, you won’t be able to deliver your product in the best way possible.
I encourage the whole team to be involved, as per scrum, and treat each story on merit. Ask questions and understand the story before sizing it.
I recommend INVEST as a guiding principle for stories. Don’t be tempted to put too much detail in a story if it’s not necessary. If you can get the information through conversation with people, and they’re always available, then don’t bother writing that stuff up in detail, it’s just wasting time and effort.
The difference between Scrum and DevOpScrum in respect to stories is that in DevOpScrum we expect to see a large number of stories not written from an end-user’s perspective. Instead, we expect to see stories written from an operation engineers perspective, or an auditor’s perspective, or a security and compliance perspective. This is why I often depart from the As a… I want… So that… template for non “user” stories, and go with a “What:… Why:…” approach, but it doesn’t matter all that much.
Same as Scrum but if I catch anyone doing that tired old “what I did yesterday, what I’m doing today, blockers…” nonsense I’ll personally come and find you and make a really, really annoying noise.
Please come up with something better, like “here’s what I commit to doing today and if I don’t achieve it I’ll eat this whole family pack of Jelly Babies” or something. Maybe something more sensible than that. Maybe.
At the end of your sprint, get together and work out what you’ve learned about the way you work, the technology and tools you’ve used, the product you’re working on and the general agile health of your team. Also take a look at how the overall delivery of your product is looking. Most importantly, ask yourself if you’re collaborating effectively, in a way that’s helping to produce a well-rounded product, that’s not only feature-rich but operationally polished as well.
Learn whatever you can and keep a record of what you’ve learnt. If any of these lessons can be turned into stories and put on the backlog as improvements, then go for it. Just make sure you don’t park all of your lessons somewhere and never visit them again!
Deliver Working Software
As with Scrum, in DevOpScrum we aim to deliver something every 2 weeks. But it doesn’t have to just be a shiny front-end to demo to your customers, you could instead deliver your roll-back, patching or Disaster Recovery process and demo that instead. Believe it or not, customers are concerned with that stuff too these days.
I personally believe this should be the guiding practice behind DevOpScrum. If you’re not familiar with Continuous Delivery (CD) then Dave Farley and Jez Humble’s book (entitled Continuous Delivery, for reasons that become very obvious when you read it) is still just about the best material on the subject (apart from my blog, of course).
As with Continuous Integration, CD is more than just a tool, it’s a set of practices and behaviours that encourage good working practices. For example, CD requires high degrees of automation around testing, deployment, and more recently around server provisioning and configuration.
So there it is in some of its glory, the DevOpScrum framework (ok, it’s just a blog about a framework, there’s enough material here to write an entire book if any reasonable level of detail was required). It’s nothing more than Scrum with a few adjustments to make it more DevOps aligned.
As with Scrum, this framework has the usual challenges – it doesn’t cater for interruptions (such as production incidents) unless you add in a triage function to manage them.
There’s also a whole bunch of stuff I’ve not covered, such as release planning, burn-ups, burn-downs and Minimum Viable Products. I’ve decided to leave these alone as they’re simply the same as you’d find in scrum.
Does this framework actually work? Yes. The truth is that I’ve actually been working in this way for several years, and I know other teams are also adapting their scrum framework in very similar ways, so there’s plenty of evidence to suggest it’s a winner. Is it perfect? No, and I’m hoping that by blogging about it, other people will give it a try, make some adjustments and help it evolve and improve.
The last thing I ever wanted to do was create a DevOps framework, but so many people are asking for a set of guidelines or a suggestion for how they should do DevOps, that I thought I’d actually write down how I’ve been using Scrum and DevOps for some time, in a way that has worked for me. However, I totally appreciate that this worked specifically for me and my teams. I don’t expect it to work perfectly for everyone.
As a DevOps consultant, I spend much of my time explaining how DevOps is a set of principles rather than a set of practices, and the way in which you apply those principles depends very much upon who you are, the ways in which you like to work, your culture and your technologies. A prescriptive framework simply cannot transcend all of these things and still be effective. This is why I always start any DevOps implementation with a blank canvas. However, if you need a kick-start, and want to try DevOpScrum then please go about it with an open mind and be prepared to make adjustments wherever necessary.
We are happy to announce OSS-Fuzz, a new Beta program developed over the past years with the Core Infrastructure Initiative community. This program will provide continuous fuzzing for select core open source software.
Open source software is the backbone of the many apps, sites, services, and networked things that make up "the internet." It is important that the open source foundation be stable, secure, and reliable, as cracks and weaknesses impact all who build on it.
Recent security storiesconfirm that errors likebuffer overflow anduse-after-free can have serious, widespread consequences when they occur in critical open source software. These errors are not only serious, but notoriously difficult to find via routine code audits, even for experienced developers. That's wherefuzz testing comes in. By generating random inputs to a given program, fuzzing triggers and helps uncover errors quickly and thoroughly.
In recent years, several efficient general purpose fuzzing engines have been implemented (e.g. AFL and libFuzzer), and we use them to fuzz various components of the Chrome browser. These fuzzers, when combined with Sanitizers, can help find security vulnerabilities (e.g. buffer overflows, use-after-free, bad casts, integer overflows, etc), stability bugs (e.g. null dereferences, memory leaks, out-of-memory, assertion failures, etc) and sometimeseven logical bugs.
OSS-Fuzz's goal is to make common software infrastructure more secure and stable by combining modern fuzzing techniques with scalable distributed execution. OSS-Fuzz combines various fuzzing engines (initially, libFuzzer) with Sanitizers (initially, AddressSanitizer) and provides a massive distributed execution environment powered by ClusterFuzz.
Early successesOur initial trials with OSS-Fuzz have had good results. An example is the FreeType library, which is used on over a billion devices to display text (and which might even be rendering the characters you are reading now). It is important for FreeType to be stable and secure in an age when fonts are loaded over the Internet. Werner Lemberg, one of the FreeType developers, wasan early adopter of OSS-Fuzz. Recently the FreeType fuzzer found a new heap buffer overflow only a few hours after the source change:
ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000ffa
READ of size 2 at 0x615000000ffa thread T0
SCARINESS: 24 (2-byte-read-heap-buffer-overflow-far-from-bounds)
#0 0x885e06 in tt_face_vary_cvtsrc/truetype/ttgxvar.c:1556:31
OSS-Fuzz automatically notifiedthe maintainer, whofixed the bug; then OSS-Fuzz automaticallyconfirmed the fix. All in one day! You can see the full list of fixed and disclosed bugs found by OSS-Fuzz so far.
Contributions and feedback are welcomeOSS-Fuzz has already found 150 bugs in several widely used open source projects (and churns ~4 trillion test cases a week). With your help, we can make fuzzing a standard part of open source development, and work with the broader community of developers and security testers to ensure that bugs in critical open source applications, libraries, and APIs are discovered and fixed. We believe that this approach to automated security testing will result in real improvements to the security and stability of open source software.
OSS-Fuzz is launching in Beta right now, and will be accepting suggestions for candidate open source projects. In order for a project to be accepted to OSS-Fuzz, it needs to have a large user base and/or be critical to Global IT infrastructure, a general heuristic that we are intentionally leaving open to interpretation at this early stage. See more details and instructions on how to apply here.
Once a project is signed up for OSS-Fuzz, it is automatically subject to the 90-day disclosure deadline for newly reported bugs in our tracker (see details here). This matches industry's best practices and improves end-user security and stability by getting patches to users faster.
Help us ensure this program is truly serving the open source community and the internet which relies on this critical software, contribute and leave your feedback on GitHub.
A few weeks ago I put out an appeal for resources for testers who are pulled into live support situations:
Looking for blogs, books, videos or other advice for testers pulled into real-time customer support, e.g. helping diagnose issues #testing— James Thomas (@qahiccupps) October 28, 2016 One suggestion I received was The Mom Test by Rob Fitzpatrick, a book intended to help entrepreneurs or sales folk to efficiently validate ideas by engagement with an appropriate target market segment. And perhaps that doesn't sound directly relevant to testers?
But it's front-loaded with advice for framing information-gathering questions in a way which attempts not to bias the the answers ("This book is specifically about how to properly talk to customers and learn from them"). And that might be, right?
The conceit of the name, I'm pleased to say, is not that mums are stupid and have to be talked down to. Rather, the insight is that "Your mom will lie to you the most (just ‘cuz she loves you)" but, in fact, if you frame your questions the wrong way, pretty much anyone will lie to you and the result of your conversation will be non-data, non-committal, and non-actionable. So, if you can find ways to ask your mum questions that she finds it easy to be truthful about, the same techniques should work with others.
The content is readable, and seems reasonable, and feels like real life informed it. The advice is - hurrah! - not in the form of some arbitrary number of magic steps to enlightenment, but examples, summarised as rules of thumb. Here's a few of the latter that I found relevant to customer support engagements, with a bit of commentary:
- Opinions are worthless ... go for data instead
- You're shooting blind until you understand their goals ... or their idea of what the problem is
- Watching someone do a task will show you where the problems and inefficiencies really are, not where the customer thinks they are ... again, understand the real problem, gather real data
- People want to help you. Give them an excuse to do so ... offer opportunities for the customer to talk; and then listen to them
- The more you’re talking, the worse you’re doing ... again, listen
These are useful, general, heuristics for talking to anyone about a problem and can be applied with internal stakeholders at your leisure as well as with customers when the clock is ticking. (But simply remembering Weinberg's definition of a problem and the Relative Rule has served me well, too.)
Given the nature of the book, you'll need to pick out the advice that's relevant to you - hiding your ideas so as not to seem like you're needily asking for validation is less often useful to a tester, in my experience - but as someone who hasn't been much involved in sales engagements I found the rest interesting background too.Image: Amazon
- Acting as a Leader
- Being a developer
- Having a systems focus
- Thinking like an entrepreneur
- Balancing strategic with tactical thinking
- Communicating well
Good software architects understand that their role as a leader is not necessarily telling developers what to do. Rather, good architects act like a guide, shepherding a team of developers towards the same technical vision drawing upon leadership skills such as story-telling, influencing, navigating conflict and building trust with individuals to turn their architectural vision into reality.
A good leader, and thus, a good architect, will listen carefully to the opinions of each contributor, fine-tuning their vision with feedback from the team. This leads well onto the next point.Being a developer
Making good architectural choices is a function of balancing an ideal target architectural state with the current state of a software system. As an example, there is no sense in adding a document database to a system if the problem domain is better suited for a relational database, even if that’s boring. An architect may feel tempted to impose technologies or architectural choices without considering the fit for the problem space – AKA behaviours of the “ivory tower architect.”
The best way an architect can mitigate this is by spending time with developers and time in the code. Understanding how the system has been built up, and the constraints of the system as it stands today will give the architect more information about the right choices for today’s environment.Having a systems focus
Seasoned developers know that code is only one aspect to working software. To make code run, a seasoned developer understands there are other important quality attributes necessary for code to run well in its production environment. They consider aspects like deployment processes, automated testing, performance, security, and supportability. Where developers may approach these quality attributes ad hoc, an architect will focus on understanding not just the code but also the quality attributes necessary to meet the many needs of different stakeholders such as support, security, and operations staff.
The good architect focuses on finding solutions that can satisfy as many of these different stakeholder needs instead of choosing a tool or approach optimised for the preferences or style of a single contributor.Thinking like an entrepreneur
All technology choices have costs and benefits, and a good architect will consider new technology choices from both perspectives. Successful entrepreneurs are willing to take risks, but seek ways to learn quickly and fail fast. Architects can approach technology choices in a similar way, seeking real-world information about short- and long-term costs and the likely benefits they will realise.
A good example is when the architect avoids committing to a new tool based on reading a new article, or having heard about it at a conference. Instead they seek to understand how relevant the tool is in their environment by running an architectural spike to gather more information. They don’t pick a tool based on how good the sales pitch is, but what value it offers, given what they need for their system. They also look for the hidden costs of tools such as how well is a tool supported (e.g. level of documentation, community adoption), how much lock-in the tool brings or the extra risks it introduces over the long-term.Balancing strategic with tactical thinking
A lot of teams build their software reactively with individual developers choosing tools and technologies that they are most comfortable with, or have the most experience with.
The good architect keeps an eye out for what newer technologies, tools or approaches might be useful but does not necessarily draw upon them immediately. Technology adoption requires a considered approach looking at a long-term horizon. Architects will seek for a good balance between agility (allowing the team to move fast) and alignment (keeping enough consistency) at both a team and organisational level.
An exercise like the Build your own Tech Radar is a useful tool to explore technologies with strategy in mind.Communicating well
Architects know that effective communication is a key skill for building trust and influencing people outside of the team. They know that different groups of people use different vocabulary and that using the technical terms and descriptions with business people makes communication more difficult. Instead of talking about patterns, tools and programming concepts, the architect uses words their audience will be familiar with. Communicating technical choices to business people with words like risk, return, costs, and benefits will serve an architect better than the words they use with their development team.
An architect also realises that communicating within the team is just as important as outside, and will use diagrams and group discussions to establish and refine the technical vision, and use a written log like an Architectural Decision Log or a wiki to provide a historical trail for future generations.Conclusion
Doing the job of a well-rounded architect is not easy. There are so many elements to focus us, each drawing upon many skills that a developer often doesn’t focus on practicing. What is most important is not necessarily the ability an architect has, but that they have enough expertise in each of these different areas to be effective. An architect who is skillful in only one of these six areas described above will not be as effective as an architect who has a good level of expertise in all of them.
Wikipedia’s article on equivalence class partitioning (ECP) is a great example of the poor thinking and teaching and writing that often passes for wisdom in the testing field. It’s narrow and misleading, serving to imply that testing is some little game we play with our software, rather than an open investigation of a complex phenomenon.
(No, I’m not going to edit that article. I don’t find it fun or rewarding to offer my expertise in return for arguments with anonymous amateurs. Wikipedia is important because it serves as a nearly universal reference point when criticizing popular knowledge, but just like popular knowledge itself, it is not fixable. The populus will always prevail, and the populus is not very thoughtful.)
In this article I will comment on the Wikipedia post. In a subsequent post I will describe ECP my way, and you can decide for yourself if that is better than Wikipedia.
“Equivalence partitioning or equivalence class partitioning (ECP) is a software testing technique that divides the input data of a software unit into partitions of equivalent data from which test cases can be derived.”
Not exactly. There’s no reason why ECP should be limited to “input data” as such. The ECP thought process may be applied to output, or even versions of products, test environments, or test cases themselves. ECP applies to anything you might be considering to do that involves any variations that may influence the outcome of a test.
Yes, ECP is a technique, but a better word for it is “heuristic.” A heuristic is a fallible method of solving a problem. ECP is extremely fallible, and yet useful.
“In principle, test cases are designed to cover each partition at least once. This technique tries to define test cases that uncover classes of errors, thereby reducing the total number of test cases that must be developed.”
This text is pretty good. Note the phrase “In principle” and the use of the word “tries.” These are softening words, which are important because ECP is a heuristic, not an algorithm.
Speaking in terms of “test cases that must be developed,” however, is a misleading way to discuss testing. Testing is not about creating test cases. It is for damn sure not about the number of test cases you create. Testing is about performing experiments. And the totality of experimentation goes far beyond such questions as “what test case should I develop next?” The text should instead say “reducing test effort.”
“An advantage of this approach is reduction in the time required for testing a software due to lesser number of test cases.”
Sorry, no. The advantage of ECP is not in reducing the number of test cases. Nor is it even about reducing test effort, as such (even though it is true that ECP is “trying” to reduce test effort). ECP is just a way to systematically guess where the bigger bugs probably are, which helps you focus your efforts. ECP is a prioritization technique. It also helps you explain and defend those choices. Better prioritization does not, by itself, allow you to test with less effort, but we do want to stumble into the big bugs sooner rather than later. And we want to stumble into them with more purpose and less stumbling. And if we do that well, we will feel comfortable spending less effort on the testing. Reducing effort is really a side effect of ECP.
“Equivalence partitioning is typically applied to the inputs of a tested component, but may be applied to the outputs in rare cases. The equivalence partitions are usually derived from the requirements specification for input attributes that influence the processing of the test object.”
Typically? Usually? Has this writer done any sort of research that would substantiate that? No.
ECP is a process that we all do informally, not only in testing but in our daily lives. When you push open a door, do you consciously decide to push on a specific square centimeter of the metal push plate? No, you don’t. You know that for most doors it doesn’t matter where you push. All pushable places are more or less equivalent. That is ECP! We apply ECP to anything that we interact with.
Yes, we apply it to output. And yes, we can think of equivalence classes based on specifications, but we also think of them based on all other learning we do about the software. We perform ECP based on all that we know. If what we know is wrong (for instance if there are unexpected bugs) then our equivalence classes will also be wrong. But that’s okay, if you understand that ECP is a heuristic and not a golden ticket to perfect testing.
“The fundamental concept of ECP comes from equivalence class which in turn comes from equivalence relation. A software system is in effect a computable function implemented as an algorithm in some implementation programming language. Given an input test vector some instructions of that algorithm get covered, ( see code coverage for details ) others do not…”
At this point the article becomes Computer Science propaganda. This is why we can’t have nice things in testing: as soon as the CS people get hold of it, they turn it into a little logic game for gifted kids, rather than a pursuit worthy of adults charged with discovering important problems in technology before it’s too late.
The fundamental concept of ECP has nothing to do with computer science or computability. It has to do with logic. Logic predates computers. An equivalence class is simply a set. It is a set of things that share some property. The property of interest in ECP is utility for exploring a particular product risk. In other words, an equivalence class in testing is an assertion that any member of that particular group of things would be more or less equally able to reveal a particular kind of bug if it were employed in a particular kind of test.
If I define a “test condition” as something about a product or its environment that could be examined in a test, then I can define equivalence classes like this: An equivalence class is a set of tests or test conditions that are equivalent with respect to a particular product risk, in a particular context.
This implies that two inputs which are not equivalent for the purposes of one kind of bug may be equivalent for finding another kind of bug. It also implies that if we model a product incorrectly, we will also be unable to know the true equivalence classes. Actually, considering that bugs come in all shapes and sizes, to have the perfectly correct set of equivalence classes would be the same as knowing, without having tested, where all the bugs in the product are. This is because ECP is based on guessing what kind of bugs are in the product.
If you read the technical stuff about Computer Science in the Wikipedia article, you will see that the author has decided that two inputs which cover the same code are therefore equivalent for bug finding purposes. But this is not remotely true! This is a fantasy propagated by people who I suspect have never tested anything that mattered. Off the top of my head, code-coverage-as-gold-standard ignores performance bugs, requirements bugs, usability bugs, data type bugs, security bugs, and integration bugs. Imagine two tests that cover the same code, and both involve input that is displayed on the screen, except that one includes an input which is so long that when it prints it goes off the edge of the screen. This is a bug that the short input didn’t find, even though both inputs are “valid” and “do the same thing” functionally.The Fundamental Problem With Most Testing Advice Is…
The problem with most testing advice is that it is either uncritical folklore that falls apart as soon as you examine it, or else it is misplaced formalism that doesn’t apply to realistic open-ended problems. Testing advice is better when it is grounded in a general systems perspective as well as a social science perspective. Both of these perspectives understand and use heuristics. ECP is a powerful, ubiquitous, and rather simple heuristic, whose utility comes from and is limited by your mental model of the product. In my next post, I will walk through an example of how I use it in real life.
I am very excited that we will be hosting CITCON in New York City on December 9 & 10, 2016.
Registrations are still open: http://citconf.com/newyork2016/
I am proud that my company, Intent Media https://intentmedia.com/, has signed on as the Venue Sponsor. As Chief Technology Officer, I am excited to showcase some of the great things we have been doing at Intent like
* mob programming
* serverless architectures
* employee growth based management
* continuous delivery
* polyglot programming
Should be tons of fun! Join us!
This month's Lean Coffee was hosted by Abcam. Here's some brief, aggregated comments and questions on topics covered by the group I was in.
Suggest techniques for identifying and managing risk on an integration project.
- Consider the risk in your product, risk in third-party products, risk in the integration
- Consider what kinds of risk your stakeholders care about; and to who (e.g. risk to the bottom line, customer data, sales, team morale ...)
- ... your risk-assessment and mitigation strategies may be different for each
- Consider mitigating risk in your own product, or in those you are integrating with
- Consider hazards and harms
- Hazards are things that pose some kind of risk (objects and behaviours, e.g. a delete button, and corruption of database)
- Harms are the effects those hazards might have (e.g. deleting unexpected content, and serving incomplete results)
- Consider probabilities and impacts of each harm, to provide a way to compare them
- Advocate for the resources that you think you need
- ... and explain what you won't (be able to) do without them
- Take a bigger view than a single tester alone can provide
- ... perhaps something like the Three Amigos (and other stakeholders)
- Consider what you can do in future to mitigate these kinds of risks earlier
- Categorise the issues you've found already; they are evidence for areas of the product that may be riskier
- ... or might show that your test strategy is biased
- Remember that the stuff you don't know you don't know is a potential risk too: should you ask for time to investigate that?
Didn't get time to discuss some of my own interests: How-abouts and What-ifs, and Not Sure About Uncertainty.
Can templates be used to generate tests?
- Some programming languages have templates for generating code
- ... can the same idea apply to tests?
- The aim is to code tests faster; there is a lot of boilerplate code (in the project being discussed)
- How would a template know what the inputs and expectations are?
- Automation is checking rather than testing
- Consider data-driven testing and QuickCheck
- Consider asking for testability in the product to make writing test code easier (if you are spending time reverse-engineering the product in order to test it)
- ... e.g. ask for consistent Ids of objects in and across web pages
- Could this (perceived) problem be alleviated by factoring out the boilerplate code?
How can the coverage of manual and automated testing be compared?
- Code coverage tools could, in principle, give some idea of coverage
- ... but they have known drawbacks
- ... and it might be hard to tie particular tester activity to particular paths through the code to understand where overlap exists
- Tagging test cases with e.g. story identifiers can help to track where coverage has been added (but not what the coverage is)
- What do we really mean by coverage?
- What's the purpose of the exercise? To retire manual tests?
- One participant is trying to switch to test automation for regression testing
- ... but finding it hard to have confidence in the automation
- ... because of the things that testers can naturally see around whatever they are looking at, that the automation does not give
What are the pros and cons of being the sole tester on a project?
- Chance to take responsibility, build experience ... but can be challenging if the tester is not ready for that
- Chance to make processes etc that works for you ... but perhaps there are efficiencies in sharing process too
- Chance to own your work ... but miss out on other perspectives
- Chance to express yourself ... but can feel lonely
- Could try all testers on all projects (e.g. to help when people are on holiday or sick)
- ... but this is potentially expensive and people complain about being thinly sliced
- Could try sharing testing across the project team (if an issue is that there's insufficient resource for the testing planned)
- Could set up sharing structures, e.g. team standup, peer reviews/debriefs, or pair testing across projects
What do (these) testers want from a test manager?
- Clear product strategy
- As much certainty as possible
- Allow and encourage learning
- Allow and encourage contact with testers from outside the organisation
- Recognition that testers are different and have different needs
- Be approachable
- Give advice based on experience
- Work with the tester
- ... e.g. coaching, debriefing, pointing out potential efficiency, productivity, testing improvements
- Show appreciation
- Must have been a tester
In The Dots I referenced How To Make Sense of Any Mess by Abby Covert. It's a book about information architecture for non-information architects, one lesson per page, each page easily digestible on its own, each page informed by the context on either side.
As a tester, I find that there's a lot here that intersects with the way I've come to view the world and how it works and how I work with and within it. I thought it would be interesting to take a slice through the book by noting down phrases and sentences that I found thought-provoking as I went.
So, what's below is information from the book, selected and arranged by one reader, and so it is also information about that reader.
Mess: a situation where the interactions between people and information are confusing or full of difficulties. (p. 169)
Messes are made of information and people. (p.11)
Information is whatever is conveyed or represented by a particular arrangement or sequence of things. (p. 19)
The difference between information, data, and content is tricky, but the important point is that the absence of content or data can be just as informing as the presence. (p. 21)
Intent is language. (p. 32)
Think about nouns and verbs. (p. 98)
Think about relationships between nouns and verbs. (p. 99)
I once spent three days defining the word "customer". (p. 88)
We create objects like maps, diagrams, prototypes, and lists to share what we understand and perceive. Objects allow us to compare our mental models with each other. (p. 57)
People use aesthetic cues to determine how legitimate, trustworthy, and useful information is. (p. 64)
Ambiguous instructions can weaken our structures and their trustworthiness. (p. 131)
Be careful not to fall in love with your plans or ideas. Instead, fall in love with the effects you can have when you communicate clearly. (p. 102)
Why, what and how are deeply interrelated. (p. 43)
We make places. (p. 86)
No matter what you're making, your users will find spaces between places. (p. 87)
We listen to our users and our guts. There is no one right way. There is only your way. (p. 101)
Murk: What alternative truths or opinions exist about what you're making or trying to achieve? (p. 113)
Uncertainty comes up in almost every project. But you can only learn from those moments if you don't give up. (p. 118)
One tiny decision leads to another, and another. (p. 85)
Perfection isn't possible, but progress is. (p. 148)
One of the questions that we asked ourselves at CEWT 3 was what we were going to do with the things we'd discovered during the workshop. How would, could, should we attempt to share any insights we'd had, and with who?
One of the answers I gave was that Karo and me would present our talks at Team Eating, the regular Linguamatics brown-bag lunch get-together. And this week we did that, to an audience of testers and non-testers from across the company. The talks were well-received and the questions and comments were interesting.
One of them came from Rog, our UX Specialist. I presented a slide which showed how testing, for me, is not linear or strictly hierarchical, and it doesn't necessarily proceed in a planned way from start to finish, and it can involve people and objects and information outside of the software itself. Testing can be gloriously messy, I probably said:
His comment was (considerably paraphrased) that that's how design feels to him. We spoke for a while afterwards and he showed me this, the squiggle of design:
I saw his squiggle and raised him a ring, showing images from a blog post I wrote earlier this year. In Put a Ring on It I described how I attempt to deal (in testing, and in management) with an analog of the left-hand end of that squiggle, by constraining enough uncertainty that I can treat what remains as atomic and proceed without needing to consider it further, at that time, so that I can shift right:
He reminded me that, perhaps a year earlier, we'd spoken about information architecture and that this was relevant to the discussion were were having right there and then. He lent me a book, How to Make Sense of Any Mess by Abby Covert.
The book discusses information-based approaches to understanding a problem, working out what kinds of changes might exist and be acceptable, choosing a route to achieving a change, monitoring progress towards it, and adapting to whatever happens along the way. I started reading it that evening and came immediately across something that resonated strongly with me:
Intent is Language: Intent is the effect we want to have on something ... The words we choose matter. They represent the ideas we want to bring into the world ... For example, if we say we want to make sustainable, eco-centered design solutions, we can't rely on thick, glossy paper catalogs to help us reach new customers. By choosing those words we completely changed our options.Covert goes on to suggest that for our designs we list two sets of adjectives: those that describe properties we want and those that describe properties we don't want. The second list should not be simple negative versions of the first and the aim should be that a neutral observer should not be able to tell which is the desired set. In this way, we can attempt to capture our intent in language in a way which can be shared with others and hopefully result in a shared vision of a shared goal.
Later in the book, she suggests some structures for managing the information that is intrinsic to any mess-resolution project. Here I saw a link to another book that I'm reading at the moment, one that I borrowed from Sime, another colleague at Linguamatics: Beautiful Evidence by Edward Tufte.
This book considers ways to improve the presentation of evidence, of information, by removing anti-patterns, by promoting clarity, by exploiting aspects of the human perceptual system. It does this in order to provide increased opportunity for greater data density, enhanced contextual information about the data, the provision of comparative data, and ultimately more useful interpretation of the data presented.
Covert's high-level information structures are useful tools for organisation of thoughts and, in one phrase - "keep it tidy" - with one brief page of prose to accompany it, she opens a door into Tufte's more detailed world.
I had begun to reflect on these things while speaking to another couple of my colleagues and noted that I continue to see value returned to me by reading around testing and related areas. The value is not necessarily immediate, but I perceive that, for example, it adds depth to my analyses, it allows me to make connections that I otherwise would not, it helps me to avoid dead ends by giving a direction that might otherwise not have been obvious.
I was a long way into my career (hindsight now shows me) before I realised that reading of this kind was something that I could be doing regularly rather than only when I had a particular problem to solve. I now read reasonably widely, and also listen to a variety of podcasts while I'm walking to work and doing chores.
And so it was interesting to me that yesterday, with all of the above fresh in my mind, while I was raking up the leaves in our back garden, a recently-downloaded episode of You Are Not So Smart with James Burke came on. In his intro, David McRaney says this, reflecting Burke's own words from a television series made in the 1970's, called Connections:
Innovation took place in the spaces between disciplines, when people outside of intellectual and professional silos, unrestrained by categorical and linear views, synthesized the work of people still trapped in those institutions ...Innovation, yes, and testing.
Images: Eil, ReVision Lab, Amazon
Edit: after reading this post, Sime pointed out Jon Bach's graphical representation of his exploratory testing, which bears a striking surface resemblance to the squiggle of design:
In a recent post, we broadly talked about What Test Engineers do at Google. In this post, I talk about one aspect of the work TEs may do: building and improving test infrastructure to make engineers more productive.
Refurbishing legacy systems makes new tools necessary
A few years ago, I joined an engineering team that was working on replacing a legacy system with a new implementation. Because building the replacement would take several years, we had to keep the legacy system operational and even add features, while building the replacement so there would be no impact on our external users.
The legacy system was so complex and brittle that the engineers spent most of their time triaging and fixing bugs and flaky tests, but had little time to implement new features. The goal for the rewrite was to learn from the legacy system and to build something that was easier to maintain and extend. As the team's TE, my job was to understand what caused the high maintenance cost and how to improve on it. I found two main causes:
- Tight coupling and insufficient abstraction made unit testing very hard, and as a consequence, a lot of end-to-end tests served as functional tests of that code.
- The infrastructure used for the end-to-end tests had no good way to create and inject fakes or mocks for these services. As a result, the tests had to run the large number of servers for all these external dependencies. This led to very large and brittle tests that our existing test execution infrastructure was not able to handle reliably.
At first, I explored if I could split the large tests into smaller ones that would test specific functionality and depend on fewer external services. This proved impossible, because of the poorly structured legacy code. Making this approach work would have required refactoring the entire system and its dependencies, not just the parts my team owned.
In my second approach, I also focussed on large tests and tried to mock services that were not required for the functionality under test. This also proved very difficult, because dependencies changed often and individual dependencies were hard to trace in a graph of over 200 services. Ultimately, this approach just shifted the required effort from maintaining test code to maintaining test dependencies and mocks.
My third and final approach, illustrated in the figure below, made small tests more powerful. In the typical end-to-end test we faced, the client made RPCcalls to several services, which in turn made RPC calls to other services. Together the client and the transitive closure over all backend services formed a large graph (not tree!) of dependencies, which all had to be up and running for the end-to-end test. The new model changes how we test client and service integration. Instead of running the client on inputs that will somehow trigger RPC calls, we write unit tests for the code making method calls to the RPC stub. The stub itself is mocked with a common mocking framework like Mockito in Java. For each such test, a second test verifies that the data used to drive that mock "makes sense" to the actual service. This is also done with a unit test, where a replay client uses the same data the RPC mock uses to call the RPC handler method of the service.
This pattern of integration testing applies to any RPC call, so the RPC calls made by a backend server to another backend can be tested just as well as front-end client calls. When we apply this approach consistently, we benefit from smaller tests that still test correct integration behavior, and make sure that the behavior we are testing is "real".
To arrive at this solution, I had to build, evaluate, and discard several prototypes. While it took a day to build a proof-of-concept for this approach, it took me and another engineer a year to implement a finished tool developers could use.
The engineers embraced the new solution very quickly when they saw that the new framework removes large amounts of boilerplate code from their tests. To further drive its adoption, I organized multi-day events with the engineering team where we focussed on migrating test cases. It took a few months to migrate all existing unit tests to the new framework, close gaps in coverage, and create the new tests that validate the mocks. Once we converted about 80% of the tests, we started comparing the efficacy of the new tests and the existing end-to-end tests.
The results are very good:
- The new tests are as effective in finding bugs as the end-to-end tests are.
- The new tests run in about 3 minutes instead of 30 minutes for the end-to-end tests.
- The client side tests are 0% flaky. The verification tests are usually less flaky than the end-to-end tests, and never more.
Building and improving test infrastructure to help engineers be more productive is one of the many things test engineers do at Google. Running this project from requirements gathering all the way to a finished product gave me the opportunity to design and implement several prototypes, drive the full implementation of one solution, lead engineering teams to adoption of the new framework, and integrate feedback from engineers and actual measurements into the continuous refinement of the tool.
What I particularly look for in meetups is information, inspiration, and the stimulation of ideas. And I wasn't disappointed in this one. Here's some assorted thoughts.
I wonder how much of my note-taking is me and how much is me in my context?
- ... and how much I would change were I to move somewhere else, or do a different job at Linguamatics
- ... given that I already know that I have evolved note-taking to suit particular tasks over time
- ... further, I already know that I use different note-taking approaches in different contexts. But why? Can I explore that more deeply?
Is this blog post notes?
- ... what is a note?
- ... perhaps this is an article? It doesn't feel like a formal report, although perhaps it could turn into one
- ... but it's more than simple aides memoire
- ... but it's not exactly full sentences
- ... but it started as notes. Then I iterated on them and they become a draft, of sorts
- ... but how? Why? According to who?
- ... and when do notes turn into something else?
- ... and when should notes turn into something else?
By writing up my notes for this post I have remembered other things that aren't in my notes
- ... and thought things that I didn't think at the time
- ... and, a week later, after discussing the evening with Karo, I've had more thoughts (and taken notes of them)
I showed my notes from CEWT 3 to one of the other participants at the event
- ... and I realised that my written notes are very wordy compared to others'
- ... and that I layer on top of them with emphasis, connections, sub-thoughts, new ideas etc
What axes of comparison make sense when considering alternative note-taking techniques?
- ... what do they give over pen and paper? (which scores on ubiquity and familiarity and flexibility)
- ... what do they give over a simple use of words? (perhaps transcription of "everything" is a baseline?)
- ... what about shorthand? (is simple compression a form of note taking?)
- ... is voice a media for notes? Some people use voice recorders
- ... sketchnoting is richer in some respects, but more time-consuming
What advantages might there be of constraining note-taking?
- ... Rapid Reporter appears to be a line-by-line tool, with no editing of earlier material
- ... the tooling around SBTM enforces a very strict syntax
- ... the concentration on structure over text of mind maps
How might contextual factors affect note-taking?
- ... writing on graph paper vs lined paper vs plain paper; coloured vs white
- ... one pen vs many different pens; different colour pens
- ... a blank page vs a divided page (e.g. Cornell)
- ... a blank page vs a page populated with e.g. Venn diagram, hierarchical structure, shapes, pie charts
- ... scrap paper vs a Moleskine
- ... pencil vs fountain pen pen vs crayon vs biro
Time allocation during note-taking
- ... what kinds of techniques/advice are there for deciding how to apportion time to note-taking vs listening/observing?
- ... are different kinds of notes appropriate when listening to a talk vs watching an event vs interacting with something (I do those differently)
What makes sense to put into notes?
- ... verbatim quotes?
- ... feelings?
- ... questions?
- ... suggestions?
- ... connections?
- ... emotions?
- ... notes about the notes?
- ... what doesn't make sense, if anything? Could it ever make sense?
I am especially inspired to see whether I can distil any conventions from my own note-taking. I have particular contexts in which I make notes on paper - meetups are one - and those where I make notes straight onto the computer - 1-1 with my team, for instance, but also when testing. I make notes differently on the computer in those two scenarios.
I have written before about how I favour plain text for note-taking on the computer and I have established conventions that suit me for that. I wonder are any conventions present in multiple of the approaches that I use?
Good thought, I'll just note that down.
Our goal in organizing GTAC each year is to make it a first-class conference, dedicated to presenting leading edge industry practices. The quality of submissions we've received for GTAC 2016 so far has been overwhelming. In order to include the best talks possible, we are extending the deadline for speaker and attendee submissions by 15 days. The new timelines are as follows:
June 1, 2016 June 15, 2016 - Last day for speaker, attendee and diversity scholarship submissions.
June 15, 2016 July 15, 2016 - Attendees and scholarship awardees will be notified of selection/rejection/waitlist status. Those on the waitlist will be notified as space becomes available.
August 15, 2016 August 29, 2016 - Selected speakers will be notified.
To register, please fill out this form.
To apply for diversity scholarship, please fill out this form.
The GTAC website has a list of frequently asked questions. Please do not hesitate to contact email@example.com if you still have any questions.
I’ve said before that I’m a big believer that there’s no “one size fits all” solution for DevOps, and nothing in my experience as a DevOps Consultant has led me to change my mind on that one. Each organisation is subtly different enough to warrant their own approach to adopting, and then succeeding with DevOps.
However, I do think there are some good patterns for successful DevOps adoption. “The right ingredients” you might say. But as with cookery and chemistry experiments, it’s the quantity of, and order in which you introduce these ingredients that makes all the difference (I discovered this first-hand as a chemistry undergraduate J ).
Below is a list of 5 steps for starting out on a successful DevOps journey (“DevOps journey” = 100 cliché points btw). It’s not a solution for scaling DevOps – that’s step 6! But if you’re looking for somewhere to start, these 5 steps are essentially the blueprint I like to follow.
- Agree what your goals are, what problems you’re trying to solve, and what DevOps means to you (is it just automation or is it a mindset?). You all need to be on the same page before you start, otherwise you’ll misunderstand each other, and without knowing your goals, you won’t know why you’re doing what you’re doing.
- Build the platform. DevOps relies heavily on fast feedback loops, so you need to enable them before you go any further. This means putting in place the foundations of a highly automated Continuous Delivery platform – from requirements management though to branching strategy, CI, test automation and environment automation. Don’t try to create an enterprise-scale solution, just start small and do what you need to do to support 1 team, or this thing will never get off the ground. You’ll probably need to pull together a bunch of DevOps engineers to set this platform up – this is often how “DevOps teams” come about, but try to remember that this team should be a transitional phase, or at least vastly scaled down later on.
- Assemble the team. We’re talking about a cross-functional delivery team here. This team will include all the skills to design, build, test, deliver and support the product, so we’re looking at a Product Owner, Business Analyst, Developers, Testers, and Infrastructure Engineers among others (it largely depends on your product – it may need to be extended to include UX designers, Security and so on).
- Be agile, not waterfall. Waterfall’s just not going to work here I’m afraid. We’re going to need a framework that supports much faster feedback and encourages far greater collaboration at all times. So with that in mind, adopt a suitable agile framework like scrum or Kanban, but tailor it appropriately so that the “Ops” perspective isn’t left out. For example – your “definition of done” should stretch to include operability features. “Done” can no longer simply mean “passed UAT”, it now needs to mean “Deployable, monitorable and working in Pre-Live” at the very minimum. Another example: Your product backlog doesn’t just contain product functionality, it needs to include operability features too, such as scalability, maintainability, monitoring and alerting.
- Work together to achieve great things. Let the delivery team form a strong identity, and empower them to take full ownership of the product. The team needs autonomy, mastery and purpose to fully unlock its potential.
Once you’ve achieved step 5, you’re well on your way to DevOps, but it doesn’t end there. You need to embrace a culture of continuous improvement and innovation, or things will begin to stagnate.
As I mentioned earlier, you still need to scale this out once you’ve got it working in one team, and that’s something that a lot of people struggle with. For some reason, there’s a huge temptation to try and get every team on-board at the same time, and make sure that they all evolve at the same rate. There’s no reason to do this, and it’s not the right approach.
If you have 20 teams all going through a brand new experience at the same time, there’s going to be a great deal of turmoil, and they’re probably going to make some of the same mistakes – which is totally unnecessary. Also, teams evolve and change at different rates, and what works for one team might not work for another, so there’s no use in treating them the same!
A much better solution is to start with one or two teams, learn from your experience, and move on to a couple more teams. The lessons learnt won’t always be transferrable from one team to the next, but the likelihood is that you’ll learn enough to give yourself a huge advantage when you start the next teams on their journey.
Sure, this approach takes time, but it’s more pragmatic and in my experience, successful.
One final comment on the steps above concerns step 2 – building the Continuous Delivery platform. It’s easy to get carried away with this step, but try to focus on building out a Minimum Viable Product here. There’s no getting away from the need for a high degree of automation, especially around testing. The types of testing you might need to focus on will depend on your product, its maturity, complexity and the amount of technical debt you’re carrying.
Other aspects you’ll need to cover in your Continuous Delivery MVP are deployment and environment automation (of course). Thankfully there are external resources available to give you a kick-start here if you don’t have sufficient skills in-house (there are plenty of contractors who specialise in DevOps engineering, not to mention dedicated DevOps consultancies such as DevOpsGuys J). Don’t spend months and months assessing different cloud providers or automation tools. Speak to someone with experience and get some advice, and crack on with it. Picking the wrong tool can be painful, but no more painful than deferring the decision indefinitely. Anyway, it’s relatively easy to move from Chef to Ansible, or from AWS to Azure (just examples) these days.
Many years ago I worked for a company that spent over a year assessing TFS, while continuing to use VS etc in the meantime. I worked with another company more recently who spent a year assessing various cloud providers, all the while struggling along with creaking infrastructure that ended up consuming everyone’s time. My point is simply that it’s better to make a start and then switch than it is to spend forever assessing your options. It’s even better to take some expert advice first.
At CEWT 3 I offered a definition of testing up for discussion. This is it:
Testing is the pursuit of actual or potential incongruityAs I said there, I was trying to capture something of the openness, the expansiveness of what testing is for me: there is no specific technique; it is not limited to the software; it doesn't have to be linear; there don't need to be requirements or expectations; the same actions can contribute to multiple paths of investigation at the same time; it can apply at many levels and those levels can be distinct or overlapping in space and time.
And these are a selection of the comments and questions that it prompted before, during and after the event, loosely grouped:
- it is sufficiently open that people could buy into it, and read into it, particularly non-testers.
- it's accurate and to the point.
- it has the feel of Weinberg's definition of a problem.
- it sounds profound but I'm not sure whether there is any depth.
- it seems very close to the regular notion of targeting information/unknowns.
- can not testing be part of this idea of testing?
- how does the notion of tacit testing (from CEWT 3 discussion) fit in?
- Kaner talks about balancing freedom and responsibility in testing. Is that covered here?
- the definition doesn't talk about risk.
- it couldn't be used to help someone new to testing decide what to do when testing.
- I could imagine putting this onto a sticky and trying to align my actions with it.
- what do you mean by pursuit?
- incongruity is too complex a word.
- what other words could replace testing in the definition and it still hold?
- when I see or I wonder about whether it's exclusive (in the Boolean sense).
In this post I'm going to talk about just the words. I spent a deal of time choosing my words - and that in itself is a warning sign. If I have to graft to find words whose senses are subtly tuned to achieve just the interpretation that I want, then I worry that others will easily have a different interpretation.
And, despite this being a definition of testing for me, it's interesting to observe how often I appeal to my feelings and desires in the description below. Could the degree of personal investment compromise the possibility of it having general appeal or utility, I wonder.
"pursuit"Other definitions use words like search, explore, evaluate, investigate, find out, ... I was particularly keen to find a verb that captured two aspects of testing for me: finding out what is there, and digging into what has been found.
What I like about pursuit is that it permits (at least to me) both, and additionally conveys a sense of chasing something which might be elusive, itinerant, latent or otherwise hard to find. Oxford Dictionaries has these definitions, amongst others of pursue:
- follow or chase (someone or something)
- continue to investigate or explore (an idea or argument)
These map onto my two needs in ways that other verbs don't:
- search: feels more about the former and less about the latter.
- investigate: feels more natual when there's a thing to investigate.
- explore: could possibly do duty for me (and it's popular in testing definitions) but exploratory testing can be perceived as cutting out other kinds of testing and I don't want that interpretation.
- evaluate: needs data; pursuit can gather data.
- find out: feels like it has finality in it. To reflect the fact that testing is unlikely to be complete I'd want to say something like "Testing is the attempt to find out about actual or potential incongruity"
"incongruity"As one of the criticisms of my definition points out, this word is not part of most people's standard lexicon. Oxford Dictionaries says that it means this:
Not in harmony or keeping with the surroundings or other aspects of something.I like it because it permits nuance in the degree to which something needs to be out of place: it could be completely wrong, or just feel a bit odd in its context. But the price I pay for the nuance is the lack of common currency. On balance I accepted this in order to keep the definition short.
"actual or potential"I felt unhappy with a definition that didn't include this, such as:
Testing is the pursuit of incongruitybecause I wanted testing's possibilities to include suggesting that there might be a problem. If the definition of incongruity I am using permitted possible disharmony then I'd have been happier with this shorter variant.
I have subsequently realised that I am, to some extent, reflecting a testing/checking distinction here too: a check with fixed expectations can find actual incongruity while testing could in addition find potential incongruity.
However, the entire definition is, for me, in the context of the relative rule - so any incongruities of any kind are tied to a context, person, time - and also the need to govern the actions in the pursuit by some notions of what is important to the people who are important to whatever is being tested.
But, even given that, I still find it hard to accept the definition without potential. Perhaps because it flags the lack of certainty inherent in much testing.
Edit: Olekssii Burdin wrote his own definition of testing after reading this.