Skip to content

Blogs

Cambridge Lean Coffee

Hiccupps - James Thomas - Wed, 04/26/2017 - 22:11

This month's Lean Coffee was hosted by DisplayLink. Here's some brief, aggregated comments and questions  on topics covered by the group I was in.

How to spread knowledge between testers in different teams, and how often should people rotate between teams?
  • How to know what is the right length of time for someone to spend in a team?
  • When is someone ready to move on?
  • How do you trade off e.g. good team spirit against overspecialisation?
  • When should you push someone out of their comfort zone, show them how much they don't know?
  • Fortnightly test team meetings playing videos of conference talks.
  • Secondment to other teams.
  • Lean Coffee for the test team.
  • Daily team standup, pairing, weekly presentations, ad hoc sharing sessions after standup.
  • Is there a desire to share?
  • Yes. Well, they all want to know more about what the others do.
  • People don't want to be doing the same thing all the time.
  • Could you rotate the work in the team rather than rotate people out of the team?
  • It might be harder to do in scenarios where each team is very different, e.g. in terms of technologies being tested.
  • There are side-effects on the team too.
  • There can't be a particular standard period of time after which a switch is made - the team, person, project etc must be taken into account too.
  • Can you rotate junior testers around teams to gain breadth of experience?

What piece of testing wisdom would you give to a new tester?
  • Be aware of communities of practice. Lots of people have been doing this for years.
  • ... for over 50 years, in fact, and a lot of what the early testers were doing is still relevant today.
  • There is value in not knowing - because you can ask questions no-one else is asking.
  • Always trust your instinct and gut when you're trying to explore a new feature or an area.
  • Learn to deal with complexity, uncertainty and ambiguity. You need to be able to operate in spite of them.
  • Learn about people. You will be working with them.
  • ... and don't forget that you are a person too.
  • Use the knowledge of the experienced testers around you. Ask questions. Ask again.
  • Make a list of what could be tested, and how much each item matters to relevant stakeholders.
  • Pick skills and practice them.

Where you look from changes what you see.
  • I was testing a server (using an unfamiliar technology) from a client machine and got a result I wasn't sure was reasonable.
  • ... after a while I switched to another client and got a different result.
  • Would a deeper technical understanding have helped?
  • Probably. In analogous cases where I have expertise I can more easily think about what factors are likely to be important and what kinds of scenarios I might consider.
  • Try to question everything that you see: am I sure? How could I disprove this?
  • Ask what assumptions are being made.
  • What you look at changes what you see: we had an issue which wasn't repeatable with what looked like a relevant export from the database, only with the whole database.
  • Part of the skill of testing is finding comparison points.
  • Can you take an expert's perspective, e.g. by co-opting an expert.

Using mindmaps well for large groups of test cases.
  • With such a large mindmap I can't see the whole thing at once.
  • Do you want to see the whole thing at once?
  • I want to organised mindmaps so that I can expand sub-trees independently because they aren't overly related.
  • Is wanting to see everything a smell? Perhaps that the structure isn't right?
  • Perhaps it's revealing an unwarranted degree of complexity in the product.
  • Or in your thinking.
  • A mindmap is your mindmap. It should exist to support you.
  • What are you trying to visualise?
  • Could you make it bigger?
  • Who is the audience?
  • I don't like to use a mindmap to keep track of project progress (e.g. with status).
  • I like a mindmap to get thoughts down
  • I use a mindmap to keep track of software dependencies.
Image: https://flic.kr/p/iv7P
Categories: Blogs

New Monthly Subscription Pricing for e-Learning Courses

News ReleaseOklahoma City, OK  73170April 25, 2017
We are excited to announce a new way for software testers and business analysts to get the training you need to grow your career, with an affordable monthly membership that gives you maximum flexibility and choice in the topics you can explore.
Plus, you will become part of an active online community of other people who are also learning and growing in their careers. 

Now, we offer three ways to gain access to excellent e-Learning courses designed to help you expand your career in new areas:
Monthly membership (individual) - $49.99 USD/month gives you access to over 19 e-Learning courses that cover topics such as Basic Training in Software Testing, SOA Testing, Cloud Testing, User Acceptance Testing, Test Automation and many others, with new courses scheduled to be added to the catalog monthly. You can cancel your membership at any time.
Certification track membership (individual) - $69.99 USD/month gives you access to all the regular monthly membership courses, PLUS our popular ISTQB Foundation Level Course in Software Testing, ASTQB Mobile Tester Certification, and IQBBA Foundation Certification for Business Analysts. (The ISTQB Agile Foundation will be included in the near future.) Exams are not included and a 3-month minimum membership is required. Advanced certification courses are not included.
Fixed-price courses (individual) – This is our current pricing plan for all courses and includes lifetime access to the materials. You pay one price for one course. All of our e-learning courses will still be available as fixed-price courses. The only way we are offering ISTQB Advanced Certification courses is with fixed pricing. In the fixed price plan, exams are included for most courses with the exception of the ISTQB Advanced Security Tester certification.  This plan may be the most workable for people who are getting reimbursed by their employer for the training.
Here is the explainer video: https://youtu.be/X6pvEIkN7JY

All pricing plans can be quoted for teams and enterprises upon request.
As a member at any of the three levels, you have e-mail access to Randy to ask questions about the content of the course(s) you are taking. In addition, you will get member-only access to special webinars and early access to new course material and other content that Randy is creating.
These e-Learning courses are not just a collection of videos. Rather, they are complete course experiences with the same course notes you would see in a live classroom setting (the notes are shown in a PDF viewer that allows them to be printed), along with quizzes and exercises. The exercises are optional, but do reinforce the concepts in the course.
All of the courses are presented by Randall (Randy) Rice, a well-known author, consultant and trainer in the field of software testing, quality assurance and business analysis. Randy holds all ISTQB Foundation and Advanced Core Certifications, as well as the ISTQB Advanced Security Tester Certification. He has over 39 years of software development and testing experience, with the past 29 years specializing in software testing consulting and training. Randy is on the board of the American Software Testing Qualifications Board (ASTQB).
Upon the completion of a course and the submission of the course evaluation, you will receive a certificate in PDF format.
To learn more and to enroll, just visit https://www.mysoftwaretesting.com
For questions and custom price quotes, please contact Randy Rice at 405-691-8075 or from the "Contact Us" link at http://www.riceconsulting.com.

Categories: Blogs

How to create Windows 10 VM Template in SCVMM

Testing tools Blog - Mayank Srivastava - Mon, 04/24/2017 - 12:35
A VM template is a master copy which we use to deploy multiple VM with our defined  base setup. Usually, … More
Categories: Blogs

How Agile is creating a new Horizon for HR

A collaboration by Amy Jackson and Mario MoreiraGone are the days of certainty.  In order to stay competitive companies must constantly innovate, adapt to changing market conditions, and deliver value to their customers faster than ever before.  As a result, many organizations are embracing Agile principles and practices, which are highly collaborative, iterative and focused on delivering maximum value to customers. As Agile adapts organizations, so must Human Resources (HR) adapt.  HR is poised to become leaders in the Agile transformation.  From an organizational change perspective, HR can facilitate and improve organizational agility by crafting programs that improve collaboration, ownership, adaptability, speed, and customer focus.  This can include:
  • Continuous Learning –determine the appropriate Agile learning path for your teams.  For those just starting out, introduce the Agile Values and Principles and make parallels to the culture and behaviors your organization values. 
  • Adapting Leadership - rethink the role of the manager.  Consider moving from a command and control approach to servant leader/ coach.  Leaders should focus on coaching and removing impediments.
  • Empowering Teams – teams that are given clear direction and outcomes should be empowered to determine how they will work to achieve their outcome.  This autonomy will drive higher levels of creativity and engagement, and if done right, deliver maximum value to customers.
  • Adapting Performance Feedback – consider moving away from “traditional” annual reviews to more frequent feedback and faster feedback loops.  Individuals and teams can adapt more quickly and apply learnings to improve work.  Provide tools and techniques that empower employees to take ownership of their development.
  • Rewarding Agile Behaviors – evaluate programs to ensure they reward the behaviors and mindset you value.  In an agile environment, teams work collaboratively, consider rewards that promote teamwork and collaboration, or recognition for continuous learning, and rewards for delivering value to customers.  A one size fits all approach may not be appropriate.
  • Reshaping Talent Acquisition– hire for culture fit and mindset and make this a priority.   Working in an agile environment is not for everyone. 
In addition to focusing on programs that drive agility, HR as an organization should embrace new ways of working that reinforce the Agile Values and Principles.  First, educate yourself and don’t be afraid to experiment and try new ways of working within your HR team.  For example, if you’re considering the idea of Self-Organizing teams, consider experimenting within your team.  You will become more knowledgeable and better equipped with first-hand experience to help guide, coach and facilitate the organization in their journey to become agile. 
As you think about adapting your programs, consider using Open Space Technology.  Open Space is a great way to gather feedback, ideas and insights from your employees that can inform how you design programs for your teams.  This approach promotes collaboration.If you plan to change or modify one of your existing programs, consider breaking this work into small increments to avoid delivering a “big bang” fully baked program which may not meet the needs of your customer.  If you plan to move away from “traditional” performance management in favor of real-time continuous feedback consider starting with one team, educate them on the value of real-time feedback and then train them on how to give and receive feedback.  Gather their feedback and iterate as needed and then begin to scale the program.In addition, start connecting to customer value.  Consider creating a compelling purpose that is focused on customer value.  Strive to keep the (external) customer front and center by linking your programs to the value they will bring to the customer. Empower your employees to make decisions that are customer centric – this shift may mean that you change how you compensate or incentivize your employees by moving away from performance metrics that are internally focused in favor of rewarding behavior and actions that delight the customer. Strategic HR organizations have expertise in helping companies achieve objectives through focus on organizational culture and high-performing teams.  Given this capability there is a natural role for HR to play in an Agile culture.  HR has an opportunity to become Agile coaches and change agents.  Embrace and ready yourself for change. This may be the new horizon for HR.-----------------Learn more about Amy Jackson at: https://www.linkedin.com/in/amjackson/Mario Moreira writes more about Agile and HR in his book "The Agile Enterprise" in Chapter 21 "Reinventing HR for Agile"
Categories: Blogs

Bad Meaning Good

Hiccupps - James Thomas - Sun, 04/23/2017 - 12:07

Good Products Bad Products by James L. Adams seeks, according to its cover, to describe "essential elements to achieving superior quality." Sounds good! As I said in my first (and failed) attempt to blog about this book, I'm interested in quality. But in the introduction (p. 2) Adams is cautious about what he means by it:
Quality is a slippery, complex, and sometimes abstract concept ... Philosophers have spent a great deal of time dealing with the concept of quality. This is not a book on semantics or philosophy, so for our purposes we will simply assume that quality means "good." But, of course, that leaves us with "good for whom?" "good for what?" "good when?" "good where?" and if you really like to pick nits, "what do you mean by good?" I won't go there, either.My bias is towards being interested in the semantics and so I'd have liked not to have seen a concept apparently so fundamental to the book being essentially dismissed in the space of a paragraph on the second page of the introduction. Which isn't to say that quality is not referred to frequently in the book itself, nor that Adams has nothing to say about quality. He does, for example when thinking about why improving the quality of a manufacturing process is often considered a more tractable problem than improving the quality of the product being manufactured (p. 25):
characteristics of good products, such as elegance, and the emotions involved with outstanding products, namely love, are not easily described by [words, maths, experiment and quantification] - you can't put a number on elegance or love.Further, he's clearly thought long and hard about the topic, and I'd be surprised if he hasn't wrestled at length with definitions of quality - having spent no little time exploring my own definition of testing, I have sympathy for anyone trying to define anything they know and care about - before deciding to pursue this line. What's reassuring to see is that Adams is clear that whatever quality or goodness of a product is, it's relative to people, task, time and place.

He references David Garvin's Competing on the Eight Dimensions of Quality, which I don't recall coming across before, and which includes two dimensions that I found particularly interesting: serviceability (the extent to which you can fix a product when it breaks, and the timeliness with which that takes place) and perceived quality (which is to do with branding, reputation, context and so on).

I was reading recently about how experiments in the experience of eating show that, amongst many other factors, heavier cutlery - which we might naturally perceive to be better quality - enhances the perception of the taste of the food:
... we hypothesized that cutlery of better quality could have an influence on the perceived quality of the food consumed with it. Understanding the factors that determine the influence of the cutlery could be of great interest to designers, chefs, and the general public alike.Adams also provides a set of human factors that he deems important in relation to quality: physical fit, sensory fit, cognitive fit, safety and health, and complexity. He correctly, in my opinion, notes that complexity is a factor that influences the others, and deems it worthy of separation.

A particularly novel aspect for me is that he talks of it in part as a consideration that has influence across products. For example, while any given car might be sufficiently uncomplex to operate, the differences in details between cars can make using an unfamiliar one a disconcerting experience (p.91): "I ... am tired of starting the windshield wipers instead of the turn signal."  He admits a tension between desiring standardisation in products and wanting designers to be free to be creative. (And this is the nub of Don Norman's book, The Design of Everyday Things, that I wrote about recently.)

It's not a surprise to me that factors external to the product itself - such as familiarity and branding - govern its perceived quality, but it's interesting to see those extrinsic factors considered as a dimension of intrinsic quality. I wondered whether Weinberg's classic definition of quality has something to say about this. According to Weinberg (see for example Agile and the Definition of Quality):

  Quality is value to some person.

And value is a measure of the amount that the person would pay for the product. Consider I'm eating a meal at a restaurant: if my enjoyment of the food is enhanced by heavier cutlery, but the cost to me remains the same as with lighter cutlery, then in some real sense the value of the food to me is higher and so I can consider the food to be of higher quality. The context can affect the product.

Alternatively, perhaps in that experiment, what I'm buying is the whole dining experience, and not the plate of food. In which case, the experiential factors are not contextual at all but fundamental parts of the product. (And, in fact, note that I can consider quality of aspects of that whole differently.)

Weinberg's definition exists in a space where, as he puts it,
the definition of "quality" is always political and emotional, because it always involves a series of decisions about whose opinions count, and how much they count relative to one another. Of course, much of the time these political/emotional decisions – like all important political/emotional decisions – are hidden from public view. Political, yes, and also personal. Adams writes (p. 43)
Thanks to computers and research it seems to me that we have gotten better at purely technical problem solving but not necessarily at how to make products that increase the quality of people's lives - a situation that has attracted more and more of my interest.And so there's another dimension to consider: even a low quality item (by some measure, such as how well it is built) can improve a person's quality of life. I buy some things from the pound shop, knowing that they won't last, knowing that there are better quality versions of those items, because the trade-off for me, for now, between cost and benefit is the right one.

Bad product: good product, I might say.
Image: Amazon
Categories: Blogs

Deeper Testing (2): Automating the Testing

DevelopSense Blog - Sat, 04/22/2017 - 16:55
Here’s an easy-to-remember little substitution that you can perform when someone suggests “automating the testing”: “Automate the evaluation and learning and exploration and experimentation and modeling and studying of the specs and observation of the product and inference-drawing and questioning and risk assessment and prioritization and coverage analysis and pattern recognition and decision making and […]
Categories: Blogs

Walking the Lines

Hiccupps - James Thomas - Sat, 04/22/2017 - 11:12


I recently became interested in turning bad ideas into good ones after listening to Reflection as a Service. At around that time I was flicking through the references in Weinberg on Writing - I forget what for - when I spotted a note about Conceptual Blockbusting by James L. Adams:
A classic work on problem solving that identifies some of the major blocks - intellectual, emotional, social, and cultural - that interfere with ideation and design.I went looking for that book and found Adams' web site and a blog post where he was talking about another of his books, Good Products Bad Products:
For many (60?) years I have been interested in what makes some products of industry "good", and others "bad". I have been involved in designing them, making them, selling them, buying them, and using them.  I guess I wanted to say some things about product quality that I think do not receive as much attention as they should by people who make them and buy themI hadn't long finished The Design of Everyday Things by Don Norman but didn't recall much discussion of quality in it. I checked my notes, and the posts (1, 2) I wrote about the book, and found that none of them mention quality either.

I'm interested in quality, generally. And my company builds products. And Adams is saying that he has a perspective that is underappreciated. And he comes recommended by an author I respect. And so I ordered the book.

Shortly after I'd started reading it I was asked to review a book by Rich Rogers. Some of the material in Good Products Bad Products was relevant to it: some overlapping concepts, some agreement, and some differences. I don't think it played a major part in the *ahem* quality of my review, but I can say that I was able to offer different, I hope useful, feedback because of what I'd read elsewhere, but only been exposed to by a series of coincidences and choices.

I continue to be fascinated by chains of connections like these. But I'm also fascinated by the idea that there are many more connections that I could have made but never did, and also that by chasing the connections that I chose to, I never got some information that would allow me to make yet further connections. As I write this sentence, other ideas are spilling out. In fact, I stopped writing that sentence in order to note them down at the bottom of the document I'm working in.

In Weinberg on Writing there's a lot of talk about the collection and curation of fieldstones, Weinberg's term for the ideas that seed pieces of writing. Sometimes, for me, that process is like crawling blind through a swamp - the paucity of solid rock and the difficulty of finding it and holding on to it seems insurmountable. But sometimes it's more like a brick factory running at full tilt on little more than air. A wisp of raw materials is fed in and pallets full of blocks pour out of the other end.

Here's a couple of the thoughts I noted down a minute ago, expanded:

Making connections repeatedly reinforces those connections. And there's a risk of thinking becoming insular because of that. How can I give myself a sporting chance of making new connections to unfamiliar material? Deliberately, consciously seeking out and choosing unfamiliar material is one way. This week I went to a talk, Why Easter is good news for scientists, at the invitation of one of my colleagues. I am an atheist, but I enjoy listening to people who know their stuff and who have a passion for it, having my views challenged and being exposed to an alternative perspective.

It's also a chance to practice my critical thinking. To give one example: the speaker made an argument that involved background knowledge that I don't have and can't contest: that there are Roman records of a man called Jesus, alive at the right kind of time, and crucified by Pontius Pilate. But, interestingly, I can form a view of the strength of his case by the fact that he didn't cite Roman records of the resurrection itself. Michael Shermer makes a similar point in How Might a Scientist Think about the Resurrection?

Without this talk, at this time, I would not have had these thoughts, not have searched online and come across Shermer (who I was unfamiliar with but now interested in), and not have thought about the idea that absence of cited evidence can be evidence of absence of evidence to cite (to complicate a common refrain).

I am interested in the opportunity cost of pursuing one line of interest vs another. In the hour that I spent at the talk (my dinner hour, as it happens) I could have been doing something else (I'd usually be walking round the Science Park listening to a podcast) and would perhaps have found other interesting connections from that.

Another concept often associated with cost is benefit. any connections I make now might have immediate benefit, later benefit or no benefit. Similarly, any information I consume now might facilitate immediate connections, later connections or no connections ever.

Which connects all of this back to the beauty and the pain of our line of work. In a quest to provide evidence about the "goodness" or "badness" of some product (whatever that means, and with apologies to James Adams it'll have to be another blog post now) we follow certain lines of enquiry and so naturally don't follow others.

It's my instinct and experience that exposing myself to challenge, reading widely, and not standing still helps me when choosing lines of enquiry and when choosing to quit lines of enquiry. But I may just not have sufficient evidence to the contrary. Yet ...
Image: Wikipedia

Edit: I wrote a review of Good Products Bad Products later.
Categories: Blogs

New Vim Course Online - Vim Hates You

ISerializable - Roy Osherove's Blog - Thu, 04/20/2017 - 08:08

I’ve added a new course to http://courses.osherove.com - This time it’s all about Vim, which I’ve been using for a few good years now. 

 

You can check out my online Vim course at http://VimHatesYou.com 

Or you can go directly to the course page at http://courses.osherove.com/p/vim-hates-you

Categories: Blogs

Podcasts!

Agile Testing with Lisa Crispin - Wed, 04/19/2017 - 21:12
Listen to some podcastsListen to some podcasts!

I’ve been honored recently to participate in two different podcasts.

I talked with Ryan Ripley and  on Ryan’s Agile for Humans podcast. We talked about pairing for various things, including presenting and writing, and discussed common challenges we see today related to testing. I’m a big fan of the Agile for Humans podcast, I learn a lot from each episode, so please give some of them a listen.

I also joined Amitai along with Johanna Rothman and Tim Ottinger on John LeDrew’s AgilePath podcast. John’s podcasts set a whole new standard for podcasts in the software world. His first topic is safety on teams. I hope you will join us in exploring that important area.

The post Podcasts! appeared first on Agile Testing with Lisa Crispin.

Categories: Blogs

Domain Command Patterns - Validation

Jimmy Bogard - Tue, 04/18/2017 - 21:36

I don't normally like to debate domain modeling patterns (your project won't succeed or fail because of what you pick), I do still like to have a catalog of available patterns to me. And one thing that comes up often are "how should I model commands?":

https://twitter.com/Mystagogue/status/853045473115987969

In general, apps I build follow CQRS, where I split my application architecture into distinct commands and queries. However, no two applications are identical in terms of how they've applied CQRS. There always seem to be some variations here and there.

My applications also tend to have explicit objects for external "requests", which are the types bound to the HTTP request variables. This might be a form POST, or it might be a JSON POST, but in either case, there's a request object.

The real question is - how does that request object finally affect my domain model?

Request to Domain

Before I get into different patterns, I like to make sure I understand the problem I'm trying to solve. In the above picture, from the external request perspective, I need a few questions answered:

  • Was my request accepted or rejected?
  • If rejected, why?
  • If accepted, what happened?

In real life, there aren't fire-and-forget requests, you want some sort of acknowledgement. I'll keep this in mind when looking at my options.

Validation Types

First up is to consider validation. I tend to look at validation with at least a couple different levels:

  • Request validation
  • Domain validation

Think of request validation as "have I filled out the form correctly". These are easily translatable to client-side validation rules. If it were 100 years ago, this would be a desk clerk just making sure you've filled in all the boxes appropriately. This sort of validation you can immediately return to the client and does not require any sort of domain-specific knowledge.

A next-level validation is domain validation, or as I've often seen referred, "business rule validation". This is more of a system state validation, "can I affect the change to my system based on the current state of my system". I might be checking the state of a single entity, a group of entities, an entire collection of entities, or the entire system. The key here is I'm not checking the request against itself, but against the system state.

While you can mix request validation and domain validation together, it's not always pretty. Validation frameworks don't mix the two together well, and these days I generally recommend against using validation frameworks for domain validation. I've done it a lot in the past and the results...just aren't great.

As a side note, I avoid as much as possible any kind of validation that changes the state of the system and THEN validates. My validation should take place before I attempt to change state, not after. This means no validation attributes on entities, for example.

Validation concerns

Next, I need to concern myself with how validation errors bubble up. For request validation, that's rather simple. I can immediately return 400 Bad Request and a descriptive body of what exactly is off with the request. Typically, request validation happens in the UI layer of my application - built in with the MVC framework I'm using. Request validation doesn't really affect the design of my domain validation.

Domain Validation

Now that we've split our validation concerns into request validation and domain validation, I need to decide how I want to validate the domain side, and how that information will bubble up. Remember - it's important to know not only that my request has failed, but why it failed.

In the domain side, understanding the design of the why is important. Can I have one reason, or multiple reasons for failure? Does the reason need to include contextual data? Do I need to connect a failure reason to a particular input, or is the contextual data in the reason enough?

Next, how are the failures surfaced? When I pass the request (or command) to the domain, how does it tell me this command is not valid? Does it just return back a failure, or does it use some indirect means, like an exception?

public void DoSomething(SomethingRequest request) {  
    if (stateInvalid) {
        throw new DomainValidationException(reason);
    }
    entity.DoSomething();
}

or

public bool DoSomething(SomethingRequest request) {  
    if (stateInvalid) {
        return false;
    }
    entity.DoSomething();
    return true;
}

In either case, I have some method that is responsible for affecting change. Where this method lives we can look at in the next post, but it's somewhere. I've gotten past the request-level validation and now need domain-level validation - can I affect this change based on the current state of the system? Two ways I can surface this back out - directly via a return value, or indirectly via an exception.

Exceptional domain validation

At first glance, it might seem that using exceptions are a bad choice for surfacing validation. Exceptions should be exceptional, and not part of a normal operation. But exceptions would let me adhere to the CQS principle, where methods either perform an action, or return data, but not both.

Personally, I'm not that hung up on CQS for these outer portions of my application, which is more of OOP concern. Maybe if I'm trying to follow OOP to the letter it would be important. But I'm far more concerned with clean code than OOP.

If I expect the exceptional case to be frequent, that is, the user frequently tries to do something that my domain validation disallows, then this wouldn't be a good choice. I shouldn't use exceptions just to get around the CQS guideline.

However, I do try to design my UX so that the user cannot get themselves in an invalid state. Even validations - my UX should guide the user so that they don't put in invalid data. The HTML5 placeholder attribute or explanatory text helps there.

But what about domain state? This is a bit more complex - but ideally, if a user isn't allowed to perform a state change, for whatever reason, then they are not presented with an option to do so! This can be communicated either with a disabled link/button, or simply removing the button/link altogether. In the case of REST, we just wouldn't return links and forms that were not valid state transitions.

If we're up-front designing our UX to not allow the user to try to get themselves in a bad state, then exceptions would truly be exceptional, and then it's OK to use them I believe.

Returning success/failure

If we don't want to use exceptions, but directly return the success/failure of our operation, then at this point we need to decide:

  • Can I have one or multiple reasons for failure?
  • Do I need contextual information in my message?
  • Do I need to correlate my message to input fields?

I don't really have a go to answer for any of these, it's really depended on the nature of the application. But if I just needed a single reason, then I can have a very simple CommandResult:

public class CommandResult  
{
   private CommandResult() { }

   private CommandResult(string failureReason)
   {
       FailureReason = failureReason;
   }

   public string FailureReason { get; }
   public bool IsSuccess => string.IsNullOrEmpty(FailureReason);

   public static CommandResult Success { get; } = new CommandResult();

   public static CommandResult Fail(string reason)
   {
       return new CommandResult(reason);
   }

   public static implicit operator bool(CommandResult result)
   {
       return result.IsSuccess;
   }
}

In the above example, we just allow a single failure reason. And for simplicity's sake, an implicit operator to bool so that we can do things like:

public IActionResult DoSomething(SomethingRequest request) {  
    CommandResult result = service.DoSomething(request);
    return result ? Ok() : BadRequest(result.FailureReason);
}

We can of course make our CommandResult as complex as we need be to represent the result of our command, but I like to start simple.

Between these two options, which should you use? I've gone back and forth between the two and they both have benefits and drawbacks. At some point it becomes what your team is more comfortable with and what best fits their preferences.

With request and command validation, let's next turn to handling the command itself inside our domain.

Categories: Blogs

Web Performance and Load Test - the Web Test Recorder plugin loads but controls are grayed out in Windows 10 with IE 11

I had an issue with IE 11 where the Web Test Recorder was loading but all controls were grayed out.

I verified the “Web Test Recorder 14.0” and “Microsoft Web Test Recorder 14.0 Helper” add-ons were enabled in IE.
Then reset and restarted IE under (Tools> Internet Options> Advanced>Reset).  After that all IE add-ons were then disabled.  I re-enabled the Web Test Recorder add-ons when prompted. 
That resolved the issue with the controls being grayed out, but then I was getting a missing DLL exception clicking Pause or Stop during the recording: "System.DllNotFoundException: Unable to load DLL 'Microsoft.VisualStudio.QualityTools.RecorderBarBHO100.x64.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)"
This site said to copy the DLL to the IE folder to resolve the missing DLL exception.Copy Microsoft.VisualStudio.QualityTools.RecorderBarBHO100.dll (For older versions RecorderBarBHO90.dll  etc)  located under C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\PrivateAssemblies

To – C:\Program Files\Internet Explorer and for 64bit machines also Copy To – C:\Program Files (x86)\Internet Explorer

I copied the DLL to the IE folders and the issue is now resolved.
Categories: Blogs

Where do our flaky tests come from?

Google Testing Blog - Mon, 04/17/2017 - 23:45
author: Jeff Listfield

When tests fail on code that was previously tested, this is a strong signal that something is newly wrong with the code. Before, the tests passed and the code was correct; now the tests fail and the code is not working right. The goal of a good test suite is to make this signal as clear and directed as possible.

Flaky (nondeterministic) tests, however, are different. Flaky tests are tests that exhibit both a passing and a failing result with the same code. Given this, a test failure may or may not mean that there's a new problem. And trying to recreate the failure, by rerunning the test with the same version of code, may or may not result in a passing test. We start viewing these tests as unreliable and eventually they lose their value. If the root cause is nondeterminism in the production code, ignoring the test means ignoring a production bug.
Flaky Tests at Google

Google has around 4.2 million tests that run on our continuous integration system. Of these, around 63 thousand have a flaky run over the course of a week. While this represents less than 2% of our tests, it still causes significant drag on our engineers.
If we want to fix our flaky tests (and avoid writing new ones) we need to understand them. At Google, we collect lots of data on our tests: execution times, test types, run flags, and consumed resources. I've studied how some of this data correlates with flaky tests and believe this research can lead us to better, more stable testing practices. Overwhelmingly, the larger the test (as measured by binary size, RAM use, or number of libraries built), the more likely it is to be flaky. The rest of this post will discuss some of my findings.
For a previous discussion of our flaky tests, see John Micco's postfrom May 2016.
Test size - Large tests are more likely to be flaky

We categorize our tests into three general sizes: small, medium and large. Every test has a size, but the choice of label is subjective. The engineer chooses the size when they initially write the test, and the size is not always updated as the test changes. For some tests it doesn't reflect the nature of the test anymore. Nonetheless, it has some predictive value. Over the course of a week, 0.5% of our small tests were flaky, 1.6% of our medium tests were flaky, and 14% of our large tests were flaky [1]. There's a clear increase in flakiness from small to medium and from medium to large. But this still leaves open a lot of questions. There's only so much we can learn looking at three sizes.
The larger the test, the more likely it will be flaky

There are some objective measures of size we collect: test binary size and RAM used when running the test [2]. For these two metrics, I grouped tests into equal-sized buckets [3] and calculated the percentage of tests in each bucket that were flaky. The numbers below are the r2 values of the linear best fit [4].

Correlation between metric and likelihood of test being flaky Metric r2 Binary size 0.82 RAM used 0.76

The tests that I'm looking at are (for the most part) hermetic tests that provide a pass/fail signal. Binary size and RAM use correlated quite well when looking across our tests and there's not much difference between them. So it's not just that large tests are likely to be flaky, it's that the larger the tests get, the more likely they are to be flaky.

I have charted the full set of tests below for those two metrics. Flakiness increases with increases in binary size [5], but we also see increasing linear fit residuals [6] at larger sizes.


The RAM use chart below has a clearer progression and only starts showing large residuals between the first and second vertical lines.



While the bucket sizes are constant, the number of tests in each bucket is different. The points on the right with larger residuals include much fewer tests than those on the left. If I take the smallest 96% of our tests (which ends just past the first vertical line) and then shrink the bucket size, I get a much stronger correlation (r2 is 0.94). It perhaps indicates that RAM and binary size are much better predictors than the overall charts show.



Certain tools correlate with a higher rate of flaky tests
Some tools get blamed for being the cause of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky [7]. For a few of our common testing tools, I determined the percentage of all the tests written with that tool that were flaky. Of note, all of these tools tend to be used with our larger tests. This is not an exhaustive list of all our testing tools, and represents around a third of our overall tests. The remainder of the tests use less common tools or have no readily identifiable tool.
Flakiness of tests using some of our common testing tools Category % of tests that are flaky % of all flaky tests All tests 1.65% 100% Java WebDriver 10.45% 20.3% Python WebDriver 18.72% 4.0% An internal integration tool 14.94% 10.6% Android emulator 25.46% 11.9%

All of these tools have higher than average flakiness. And given that 1 in 5 of our flaky tests are Java WebDriver tests, I can understand why people complain about them. But correlation is not causation, and given our results from the previous section, there might be something other than the tool causing the increased rate of flakiness.
Size is more predictive than tool

We can combine tool choice and test size to see which is more important. For each tool above, I isolated tests that use the tool and bucketed those based on memory usage (RAM) and binary size, similar to my previous approach. I calculated the line of best fit and how well it correlated with the data (r2). I then computed the predicted likelihood a test would be flaky at the smallest bucket [8] (which is already the 48th percentile of all our tests) as well as the 90th and 95th percentile of RAM used.
Predicted flaky likelihood by RAM and tool Category r2 Smallest bucket
(48th percentile) 90th percentile 95th percentile All tests 0.76 1.5% 5.3% 9.2% Java WebDriver 0.70 2.6% 6.8% 11% Python WebDriver 0.65 -2.0% 2.4% 6.8% An internal integration tool 0.80 -1.9% 3.1% 8.1% Android emulator 0.45 7.1% 12% 17%

This table shows the results of these calculations for RAM. The correlation is stronger for the tools other than Android emulator. If we ignore that tool, the difference in correlations between tools for similar RAM use are around 4-5%. The differences from the smallest test to the 95th percentile for the tests are 8-10%. This is one of the most useful outcomes from this research: tools have some impact, but RAM use accounts for larger deviations in flakiness.
Predicted flaky likelihood by binary sizeand tool

Category r2 Smallest bucket
(33rd percentile) 90th percentile 95th percentile All tests 0.82 -4.4% 4.5% 9.0% Java WebDriver 0.81 -0.7% 14% 21% Python WebDriver 0.61 -0.9% 11% 17% An internal integration tool 0.80 -1.8% 10% 17% Android emulator 0.05 18% 23% 25%

There's virtually no correlation between binary size and flakiness for Android emulator tests. For the other tools, you see greater variation in predicted flakiness between the small tests and large tests compared to RAM; up to 12% points. But you also see wider differences from the smallest size to the largest; 22% at the max. This is similar to what we saw with RAM use and another of the most useful outcomes of this research: binary size accounts for larger deviations in flakiness than the tool you use.
Conclusions

Engineer-selected test size correlates with flakiness, but within Google there are not enough test size options to be particularly useful.
Objectively measured test binary size and RAM have strong correlations with whether a test is flaky. This is a continuous function rather than a step function. A step function would have sudden jumps and could indicate that we're transitioning from one type of test to another at those points (e.g. unit tests to system tests or system tests to integration tests).
Tests written with certain tools exhibit a higher rate of flakiness. But much of that can be explained by the generally larger size of these tests. The tool itself seems to contribute only a small amount to this difference.
We need to be more careful before we decide to write large tests. Think about what code you are testing and what a minimal test would look like. And we need to be careful as we write large tests. Without additional effort aimed at preventing flakiness, there's is a strong likelihood you will have flaky tests that require maintenance.
Footnotes
  1. A test was flaky if it had at least one flaky run during the week.
  2. I also considered number of libraries built to create the test. In a 1% sample of tests, binary size (0.39) and RAM use (0.34) had stronger correlations than number of libraries (0.27). I only studied binary size and RAM use moving forward.
  3. I aimed for around 100 buckets for each metric.
  4. r2 measures how closely the line of best fit matches the data. A value of 1 means the line matches the data exactly.
  5. There are two interesting areas where the points actually reverse their upward slope. The first starts about halfway to the first vertical line and lasts for a few data points and the second goes from right before the first vertical line to right after. The sample size is large enough here that it's unlikely to just be random noise. There are clumps of tests around these points that are more or less flaky than I'd expect only considering binary size. This is an opportunity for further study.
  6. Distance from the observed point and the line of best fit.
  7. Other web testing tools get blamed as well, but WebDriver is our most commonly used one.
  8. Some of the predicted flakiness percents for the smallest buckets end up being negative. While we can't have a negative percent of tests be flaky, it is a possible outcome using this type of prediction.
Categories: Blogs

Exploring Five Pair Programming Techniques

A collaboration by Antonio González Sanchis and Mario Moreira
Extreme Programming (XP) introduced a programming technique that’s been famous since its early days: pair programming. It sounds simple, two people working together with the same computer. However, what people don’t know is that there are many ways in which you can approach this technique. Let’s explore a few:

Driver-navigator:This is the best known style of pairing. It consists of one person ‘driving’, taking the keyboard and coding or doing the work, while the other one is ‘navigating’. The Navigator’s job is to pay attention to the work being done by the driver while keeping the big picture in mind.  They should guide the driver in the right direction. It is very important that driver explains every decision they make, otherwise the navigator might lose interest and may stop paying attention. It’s healthy to switch roles every now and then.

Ping-pong: For this approach you might need two keyboards connected to the same computer. In contrast to the driver-navigator mode, in ping-pong both people can be driving at any point in time. A good strategy for this approach is to have one person writing the tests while the other one tries to pass them. As in the previous approach, you should be switching roles often.

Backseat navigator: A typical approach when there’s a new team member in the group or a junior colleague. The navigator (usually the more senior team member) tells the driver, who has the keyboard, what to do to solve a problem with every little detail. The navigator should provide insights on how to solve the problem but also on why that’s the best solution in mind. It’s a good way to ramp up new members in your team as they build relationships with other people in the team while learning about the working style.

Tour:If there is an even number of team members or someone who you used to pair with went on holidays, you are not going to stop working. However, when those people return or you start pairing with someone, you don’t want them to start right away. Therefore, a good way to start would be to do a ‘tour’ through the code or work you did in their absence. The person who doesn’t know the most about the work takes the keyboard and the mouse and starts driving through the code while the ‘expert’ acts as a navigator and explains every detail.

Distributed pairing: Often people tend to think pairing is for collocated teams but that’s not entirely true. While pairing face-to-face is definitely much more effective than remotely, there are many tools and ways colleagues can pair. A good suggestion would be to use a videoconference messaging tool with a screen sharing app that allows the navigator to see in real time what the driver is doing.
These are not the only pairing styles that you might encounter but they are the best known.  They are also the easier techniques to try when you want to start pair programming. Also, examples seem to focus on software development but pairing is applicable to almost every field.  For example if you are in marketing and are launching an email campaign, you can pair with someone to navigate through the email while you write it, etc.

There might be anti-patterns when pairing. For example, you might find cases when the navigator is ‘sleeping’. This means the navigator can be checking emails, texting or even coding in parallel without paying much attention to the driver. Try to identify the root causes of this behaviour before you jump in into action. Usually it might be because the driver is not explaining properly the reasons why he is doing a specific task in a particular way.

Lastly, keep in mind that while pair programming may work in every role or department, it might not be suitable for every work. There will be times when you just want to discuss the approach with the team and go do it by yourself. That’s also a way of getting feedback, which in the end is the main purpose of pair programming: getting feedback early.  

If you are interested in trying pair programming, the first thing to do is review the list of possible techniques (above) and select one that you will experiment with.  Then identify your pair.  Brainstorm on how you might approach pair programming, specify how long you will try your experiment, and begin.  
------------Learn more about Antonio González Sanchis at https://www.linkedin.com/in/antoniogonzalezsanchis/
Read more of Mario's article at: http://cmforagile.blogspot.com 
Categories: Blogs

My Experience at the Quest 2017 Conference

I was honored to be asked to present a half-day tutorial and track session at QAI's Quest 2017 Conference last week in Chicago. 

I have spoken at testing conferences since 1989, with only two years since then that I have sat out a year. So, I've been to a lot of these rodeos worldwide. In fact, I chaired the QAI International Software Testing Conference (1995 - 2000), so I know what it takes to keep everything on track.

It's from that background that I write this. My intent is not to take away from any other conference. I also have conferences on the schedule yet for this year. 

By the way, this blog post was not solicited. I just feel that when someone does a great job, recognition is deserved. 

I was blown away by this conference for several reasons:

Outstanding management - Tom Ticknor, Nancy Kastl, Anna Zucker and the rest of the team did a great job of making sure everything ran like clockwork. The weather turned rather bad on Wednesday (40F, 40 - 50 mph winds and rain), so the planned dinner cruise was still held on the boat, but it was enclosed and we never left the dock. It was also very challenging to get a few hundred people on busses and over to Navy Pier, but we all made it and a good time was had by all!

Outstanding volunteers - All the track hosts and other volunteers did a great job. In my experience, you can't pull off a great conference without great volunteers to make sure all the little things are handled. Thanks much to Kenneth Brown, who was my track host for both sessions. You rock!

A strong sense of community - I felt like I was among friends - and I was. In fact, in some cases, it was like old home week to see friends from the past QAI and CQAA events. I was able to strike up conversations with everyone I met. This was not a huge conference in terms of attendance, but for me it was just the right size - not too big, not too small.

Wide representation - This was not just a local Chicago event. There were people from much of the USA there, as well as from about a dozen other countries.

Great sessions - From the keynotes to the track sessions, I did not experience a bad session. Good content plus good speaking ability equals a great session. Also, the selection of topics was wide enough to cover just about anyone's topics of interest. 

Solid content - Speaking of content, one of the things that concerns me about the current state of testing is that we are taking our eyes off the main thing - testing and quality. Most testers think QA and testing are the same thing (they aren't). One thing that was striking to me was that in every session I attended, the concepts were solid, with very practical ways to apply the ideas. 

Awesome food - I can't say I ever had filet mignon for lunch at a conference before, but I did last week. I don't fly for the dining experience, and I don't expect a lot from conference fare, but really - bacon and eggs for breakfast every day? That's good stuff. Plus, the dinner cruise food was very good. Kudos to the Renaissance hotel for the preparation and for the Quest team for the menu selections.

Informative expo - The reality is most vendors go to the larger conferences because that is where the numbers are. But, how many of those people are the decision makers? I spoke with most of the vendors in the expo and found that they all had great offerings. I expect there were a fair number of decision-makers at the conference.

The negatives? I can't think of any.

I highly recommend this conference to anyone looking for any of the above things. If you are in the software testing or software quality field and live in the mid-west, attending the Quest Conference is a no-brainer. I hope to attend again in the future!




Categories: Blogs

PowerShell Get-ChildItem -Include, -Exclude, and -Recurse parameters

Testing tools Blog - Mayank Srivastava - Fri, 04/14/2017 - 11:43
‘Get-ChildItem’ cmdlet gets you the contents of any given path and then you can perform your next desired actions on … More
Categories: Blogs

Rice Consulting's IQBBA BA Certification Course is Accredited by the ASTQB

Press Release

Rice Consulting Announces Accreditation of New IQBBA Business Analyst Certification Training Course

Learn how to improve the quality of IT projects by understanding and documenting user needs in clear and understandable ways.

Oklahoma City, OK, April 14, 2017:  Randall Rice, internationally-recognized author, consultant and trainer in software testing and business analysis is excited to announce the accreditation of his newest course, IQBBA Foundation Level Certification Course by the American Software Testing Qualifications Board (ASTQB).

This is a course designed for business analysts and others who are looking for effective ways to gather and document user needs for projects in their organization. This course teaches people best practices in how to deliver projects that meets user needs and expectations.

The course is based on the Certified Foundation Level Business Analyst (CFLBA) Syllabus from the International Qualification Board for Business Analysts (IQBBA). Accreditation verifies that the course content covers the certification syllabus and glossary. In addition, the reviewers ensure that the course covers the materials at the levels indicated in the syllabus.

“Regardless of the project methodology in place, unless the user need is fully understood and articulated, a complete and correct solution cannot be delivered. Failure to capture the user needs is one key reason that around 30% of projects never reach successful implementation,” explained Randall Rice. 

ASTQB president, Debbie Friedenberg, says, "We believe that the IQBBA's internationally-recognized Business Analyst certification is of great importance to the profession, especially as we see the Business Analysis continuing to gain importance in the coming years."

This course is currently available on an on-site basis and in online e-Learning format. For further details, visit http://www.riceconsulting.com

To schedule a course to be presented in your company, contact Randall Rice at 405-691-8075 or by e-mail

Randall W. Rice, author and trainer of the course is a Certified Tester, Advanced Level and is on the Board of Directors of the ASTQB. He is the co-author with William E. Perry of two books, “Surviving the Top Ten Challenges of Software Testing” and “Testing Dirty Systems.”

Categories: Blogs

Search Party

Hiccupps - James Thomas - Wed, 04/12/2017 - 06:36

Last month's Cambridge Tester meetup was puzzling. And one of the puzzles was an empty wordsearch that I'd made for my youngest daughter's "Crafternoon" fundraiser. At Crafternoon, Emma set up eight different activities at our house and invited some of her school friends to come and do them, with the entrance fee being a donation to charity.

The idea of the wordsearch activity is simple: take the blank wordsearch grid and make a puzzle from it using the list of words provided. Then give it to someone as a present.

If you fancy a go, download it here: Animal Alphabet Wordsearch (PDF)

(You're free to use it for your own workshops, meetups, team exercises or whatever. We hope you have fun and, if you do, please let us know about it and consider donating to an animal charity. Emma supports Wood Green.)

After Crafternoon, I offered the puzzle to Karo for the Cambridge Tester meetup and she wrote about in Testing Puzzles: Questions, Assumptions, Strategy. It's fun to read about how the testers addressed the task. It's also fun to compare it to what the children did. Broadly, I think that the kids were less concerned by a sense of expectation about the outcome - and that's not a remotely original observation, I appreciate.

Everyone who took part had some "knowledge in the head" about the task (conventions from their own experiences) and there is some "knowledge in the world" about it too, such as whatever instructions have been given and the guidelines for the person who is gifted the completed wordsearch.

Some of the testers gently played with convention by, for example:
  • filling in all blank cells with the letter A
  • using symbols outside of the Roman alphabet
  • mixing upper and lower case in the grid
  • ...

But the kids in general went further by:
  • writing more than one letter in a cell
  • writing letters outside of cells
  • writing words around corners
  • leaving some cells blank
  • crossing out the words from the list if they couldn't fit them in the grid
  • spelling something wrong to make it fit
  • ...

In our jobs we're often thinking about how a product could be used in ways that it wasn't intended. It's an education watching children trample all over a task like this, deriving their own enjoyment from it, unselfconsciously making it into whatever works for them at that moment, constrained much more by the practical restrictions (pen, paper, the location of Crafternoon, ...) than any theoretical ideas or social norms.

While I was thinking about this - washing up last night, as it happens - I was listening to Russell Brand on The Comedian's Comedian podcast. He's a thoughtful chap, worth hearing, and he came out with this beautiful quote:
Only things that there are words for are being said. A challenge ... is to make up different words if you want to say different and unusual things.And that's fitting in a blog post about finding words, but it generalises: the children were willing and able to invent a lexicon of actions that was permitted by the context they found themselves in. As a tester, are you?
Image:Disney
Categories: Blogs

Code Health: Google's Internal Code Quality Efforts

Google Testing Blog - Tue, 04/11/2017 - 02:52
By Max Kanat-Alexander, Tech Lead for Code Health and Author of Code Simplicity

There are many aspects of good coding practices that don't fall under the normal areas of testing and tooling that most Engineering Productivity groups focus on in the software industry. For example, having readable and maintainable code is about more than just writing good tests or having the right tools—it's about having code that can be easily understood and modified in the first place. But how do you make sure that engineers follow these practices while still allowing them the independence that they need to make sound engineering decisions?

Many years ago, a group of Googlers came together to work on this problem, and they called themselves the "Code Health" group. Why "Code Health"? Well, many of the other terms used for this in the industry—engineering productivity, best practices, coding standards, code quality—have connotations that could lead somebody to think we were working on something other than what we wanted to focus on. What we cared about was the processes and practices of software engineering in full—any aspect of how software was written that could influence the readability, maintainability, stability, or simplicity of code. We liked the analogy of having "healthy" code as covering all of these areas.

This is a field that many authors, theorists, and conference speakers touch on, but not an area that usually has dedicated resources within engineering organizations. Instead, in most software companies, these efforts are pushed by a few dedicated engineers in their extra time or led by the senior tech leads. However, every software engineer is actually involved in code health in some way. After all, we all write software, and most of us care deeply about doing it the "right way." So why not start a group that helps engineers with that "right way" of doing things?

This isn't to say that we are prescriptive about engineering practices at Google. We still let engineers make the decisions that are most sensible for their projects. What the Code Health group does is work on efforts that universally improve the lives of engineers and their ability to write products with shorter iteration time, decreased development effort, greater stability, and improved performance. Everybody appreciates their code getting easier to understand, their libraries getting simpler, etc. because we all know those things let us move faster and make better products.

But how do we accomplish all of this? Well, at Google, Code Health efforts come in many forms.

There is a Google-wide Code Health Group composed of 20%contributors who work to make engineering at Google better for everyone. The members of this group maintain internal documents on best practices and act as a sounding board for teams and individuals who wonder how best to improve practices in their area. Once in a while, for critical projects, members of the group get directly involved in refactoring code, improving libraries, or making changes to tools that promote code health.

For example, this central group maintains Google's code review guidelines, writes internal publications about best practices, organizes tech talks on productivity improvements, and generally fosters a culture of great software engineering at Google.

Some of the senior members of the Code Health group also advise engineering executives and internal leadership groups on how to improve engineering practices in their areas. It's not always clear how to implement effective code health practices in an area—some people have more experience than others making this happen broadly in teams, and so we offer our consulting and experience to help make simple code and great developer experiences a reality.

In addition to the central group, many products and teams at Google have their own Code Health group. These groups tend to work more closely on actual coding projects, such as addressing technical debt through refactoring, making tools that detect and prevent bad coding practices, creating automated code formatters, or making systems for automatically deleting unused code. Usually these groups coordinate and meet with the central Code Health group to make sure that we aren't duplicating efforts across the company and so that great new tools and systems can be shared with the rest of Google.

Throughout the years, Google's Code Health teams have had a major impact on the ability of engineers to develop great products quickly at Google. But code complexity isn't an issue that only affects Google—it affects everybody who writes software, from one person writing software on their own time to the largest engineering teams in the world. So in order to help out everybody, we're planning to release articles in the coming weeks and months that detail specific practices that we encourage internally—practices that can be applied everywhere to help your company, your codebase, your team, and you. Stay tuned here on the Google Testing Blog for more Code Health articles coming soon!

Categories: Blogs

Automated Tests for Asynchronous Processes

thekua.com@work - Sun, 04/09/2017 - 15:31

It’s been a while since I’ve worked on a server-side application that had asynchronous behaviour that wasn’t already an event-driven system. Asynchronous behaviour is always an interesting challenge to design and test. In general, asynchronous behaviour should not be hard to unit test – after all, the behaviour of an action shouldn’t necessarily be coupled temporally (see forms of coupling).

TIP: If you are finding the need for async testing in your unit tests, you’re probably doing something wrong and need to redesign your code to decouple these concerns.

If your testing strategy only includes unit testing, you will miss a whole bunch of behaviour which are often caught at high level of testing like integration, functional or system tests – which is where I need asynchronous testing.

Asychronous testing, conceptually, is actually pretty easy. Like synchronous testing, you take an action and then look for a desired result. However unlike synchronous testing, your test cannot guarantee that the action has completed before you check for the side-effect or result.

There are generally two approaches to testing asynchronous behaviour:

  1. Remove the asynchronous behaviour
  2. Poll until you have the desired state
Remove the asynchronous behaviour

I used this approach when TDD-ing a thick client application many years ago, when writing applications in swing applications was still a common approach. Doing this required isolating the action invoking behaviour into a single place, that, instead of it occurring in a different thread would, during the testing process, occur in the same thread as the test. I even gave a presentation on it in 2006, and wrote this cheatsheet talking about the process.

This approach required a disciplined approach to design where toggling this behaviour was isolated in a single place.

Poll until you have the desired state

Polling is a much more common approach to this problem however this involves the common problem of waiting and timeouts. Waiting too long increases your overall test time and extends the feedback loop. Waiting too short might also be quite costly depending on the operation you have (e.g. hammering some integration point unnecessarily).

Timeouts are another curse of asynchronous behaviour because you don’t really know when an action is going to take place, but you don’t really want a test going forever.

The last time I had to do something, we would often end up writing our own polling and timeout hook, while relatively simple is now available as a very simple library. Fortunately other people have also encountered this problem in java-land and contributed a library to help make testing this easier in the form of Awaitility.

Here is a simple test that demonstrates how easy the library can make testing asynchronous behaviour:

package com.thekua.spikes.aysnc.testing;

import com.thekua.spikes.aysnc.testing.FileGenerator;
import org.junit.Before;
import org.junit.Test;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import static java.util.concurrent.TimeUnit.SECONDS;
import static org.awaitility.Awaitility.await;
import static org.hamcrest.Matchers.startsWith;
import static org.junit.Assert.assertThat;

public class FileGeneratorTest {

    private static final String RESULT_FILE = "target/test/resultFile.txt";
    private static final String STEP_1_LOG = "target/test/step1.log";
    private static final String STEP_2_LOG = "target/test/step2.log";
    private static final String STEP_3_LOG = "target/test/step3.log";

    private static final List<String> FILES_TO_CLEAN_UP = Arrays.asList(STEP_1_LOG, STEP_2_LOG, STEP_3_LOG, RESULT_FILE);


    @Before
    public void setUp() {
        for (String fileToCleanUp : FILES_TO_CLEAN_UP) {
            File file = new File(fileToCleanUp);
            if (file.exists()) {
                file.delete();
            }
        }
    }


    @Test
    public void shouldWaitForAFileToBeCreated() throws Exception {
        // Given I have an aysnc process to run
        String expectedFile = RESULT_FILE;

        List<FileGenerator> fileGenerators = Arrays.asList(
                new FileGenerator(STEP_1_LOG, 1, "Step 1 is complete"),
                new FileGenerator(STEP_2_LOG, 3, "Step 2 is complete"),
                new FileGenerator(STEP_3_LOG, 4, "Step 3 is complete"),
                new FileGenerator(expectedFile, 7, "Process is now complete")
        );

        // when it is busy doing its work
        ExecutorService executorService = Executors.newFixedThreadPool(10);
        for (final FileGenerator fileGenerator : fileGenerators) {
            executorService.execute(new Runnable() {
                public void run() {
                    fileGenerator.generate();
                }
            });
        }

        // then I get some log outputs
        await().atMost(2, SECONDS).until(testFileFound(STEP_1_LOG));
        await().until(testFileFound(STEP_2_LOG));
        await().until(testFileFound(STEP_3_LOG));

        // and I should have my final result with the output I expect
        await().atMost(10, SECONDS).until(testFileFound(expectedFile));
        String fileContents = readFile(expectedFile);
        assertThat(fileContents, startsWith("Process"));

        // Cleanup
        executorService.shutdown();
    }

    private String readFile(String expectedFile) throws IOException {
        return new String(Files.readAllBytes(Paths.get(expectedFile)));

    }


    private Callable<Boolean> testFileFound(final String file) {
        return new Callable<Boolean>() {
            public Boolean call() throws Exception {
                return new File(file).exists();
            }
        };
    }
}

You can explore the full demo code on this public git repository.

Categories: Blogs

Review Mirror

Hiccupps - James Thomas - Thu, 04/06/2017 - 22:47

Months ago, Rich Rogers asked on Twitter for volunteers to review the book that he's writing, and I put my hand up. This week a draft arrived, and I put my hands up.

After the shock of having to follow through on my offer had worn off, I pondered how to go about the task. I've reviewed loads of books, but always with the luxury of pulling out just the things that interest me. I can't recall giving detailed feedback on something book-length before and I wanted to do a thorough job.

I liked the idea of an approach that combined reaction and reflection. Reaction is an "inline" activity. I can read the book and provide my System 1 responses as I go. Because they are instinctive I don't need to interrupt my flow significantly to capture them. My reactions can then be data for my reflection, where my System 2 processes try to make sense of the whole.

That seemed reasonable. But I'm a tester so I framed it as a mission:
Explore the manuscript using both reaction and reflection to provide Rich with a review that's more than skin deep.The reactions were easy to deliver as comments in the online editor that Rich is using. I applied a few ground rules: no reading ahead, no reading of any passage more than three times, no reflection. I broke only the last rule, and only once: there's a pivotal chapter in the book that didn't hang together well for me, even after repeated reading, and I took a few moments to give an overview when I'd finished it.

Once I'd got to the end of the final section, I put the book away and did nothing on it for day or so. Reflections that had been forming while I read began to solidify and I started looking for a way to organise them. As usual, I wrote, and in writing I saw a pattern. I had three classes of notes:
  • feelings: I did not attempt to justify
  • observations: I had tried to justify
  • suggestions: I could both justify and explain
This wasn't a breakdown that I could say I'd seen elsewhere but it felt like a natural one. (Writing this later, I wonder whether it's influenced by PROOF - Past, Results, Obstacles, Outlook, Feelings. I'm a FOP, perhaps?)

In any case, I delivered my reflections to Rich. He didn't agree with everything, but seemed happy enough on the whole:
Very grateful to @qahiccupps for reviewing my book and for making some excellent suggestions. Some work for me to do yet @PublishHeddon— Rich Rogers (@richrtesting) April 6, 2017
And now I get something for myself: reflecting on what I did.
Image: https://flic.kr/p/4X3PW5
Categories: Blogs