I’ve been having a lot of rants about Cucumber of late, as it’s the new shiny thing for agile teams.  Does anyone else have issues with it?  I’ve asked all of my programmer friends to convince me of its worth, and they’ve all failed so far.  I’ve not seen it adding any value above building a good API (and I see it bringing a lot of negatives relative to other possible approaches).

In my experience, I’m seeing -

- Customers/non-programmers never write the tests (because they have very little interest in specifying everything in given-when-then.  They just want to tell us what they want.  And it doesn’t make much sense to specify everything in that format anyway).
- Customers/non-programmers write the tests but it focuses the test effort on writing those kinds of tests, rather than other testing that seems to add more value.
- The tests are written in English, but what the test actually does depends on how the developers convert the english phrases into code (so there’s no guarantee it tests what the customer intended anyway).
- Avoidance of conversation (ie. tests as contracts).
- Cucumber and the related tools (through the toy examples they provide) encourage developers to put lots of implementation detail into the tests (and sometimes to do a lot more testing through the GUI/http layer rather than pushing some of that testing down).
- Refactoring sucks as we lose IDE support.
- Much heavier test artefacts (when a lot of the teams I work with are already struggling with the weight of their agile test automation).
- A continued focus of tests on what the system does and not why it does it and the business outcome.
- Anecdotally, I’m not seeing better outcomes than people were getting with other approaches.  I realise it may be leading to better designs, but then I’d expect to see improvements somewhere in the process.  That looks like it would require better modelling skills than I’m seeing in most of the cucumber tests on my projects.

Most of the examples that I’ve seen where people are claiming success are fairly small applications.  I’m not seeing the approach scale that well.  Yes, most teams could do write their cucumber tests better, but even then, in my experience, other approaches would be more effective and more efficient.

Any thoughts?  If there’s interest, I’ll try and post some examples of what I think those tests might look like if we pushed them into other forms.

A frequently asked question for Watir seems to be whether there are any “record and playback” tools.  While most of the references point to a somewhat old tool, two viable alternatives seem much harder to find for some reason.

The first is Webmetrics‘ Global Watch Script recorder, available at http://www.webmetrics.com/products/script_recorder.html

The second, is CubicTest.  This uses SeleniumIDE to record the test, but can generate a script for Watir.

(Update) A third is the Testwise Watir Recorder, a Firefox plugin which I looked at a while back, but am unsure of it’s current state.

These will only get you so far, and have their own limitations, the most obvious of which is some corporate applications which only work in Internet Explorer. If a recorder is what you need though, there are a few options and these may get you further.  At any rate, hopefully they’re now easier for people to find.

Jonathan Kohl pointed me at Sikuli, a Python-based tool for automating applications using image recognition.  Unlike most tools, which attempt to identify objects via public APIs, Sikuli looks at the pixels on the screen and attempts to identify objects based on how they look.

This isn’t exactly a new approach, as commercial tools have had this feature for a long time as a means of creating custom objects.  So after playing with Sikuli, I wondered whether I could take advantage of it as a library to augment my Watir scripts in Ruby.

It turns out, the answer is ‘yes’, with a caveat.  You need to use JRuby (although you could probably do it in Ruby using the Ruby-Java bridge – It just looked a lot harder), and you also need to use Watir-Webdriver, a new implementation of Watir’s API which is used in a number of other automation frameworks.

Below is a simple example script and instructions to get you started.  It navigates to a website, then clicks on the flash control there. I’ve so far only tested this on Windows.  It should work on OSX and Linux, but perhaps not quite so easily (I’m waiting on some feedback).  Check the instructions for ‘Install OpenCV’ at the page with instructions for calling Sikuli from other tools.

Sikuli can be used to automate Flash components, any challenging AJAX elements of your web application, to dismiss pop-ups or probably even to inspect visual elements of the page (though I’d want to do this minimally).  It’s a little slow, but an interesting and immediately useful add-on to Watir or your favourite java-based testing tool.

#Install Java, or install the JRuby/JRE bundle at the next step
#Install JRuby 1.5.1 - http://jruby.org/download
#Install Sikuli - http://groups.csail.mit.edu/uid/sikuli/download.shtml
#Install watir-webdriver (eg. jgem install watir-webdriver)
#Update ssh - jgem install jruby-openssl
#Copy sikuli-script.jar to \jruby-1.5.1\lib
#Get the test image
#Download http://www.software-testing.com.au/images/flashmap_middleeast.png and put it in the image folder as below
#See http://sikuli.org/trac/wiki/How to use Sikuli Script in your JAVA programs for examples

require ‘rubygems’
require ‘watir-webdriver’
require ‘java’

java_import “edu.mit.csail.uid.SikuliScript”
java_import “edu.mit.csail.uid.Region”
java_import “edu.mit.csail.uid.Screen”

start_page=’http://www.lonelyplanet.com’
image_folder=”c:/sikuli_icons/”

$screen=Screen.new
$browser=Watir::Browser.new :ie
$browser.goto start_page

$screen.click(“#{image_folder}flashmap_middleeast.png”,0)

This news article where a hairdresser’s client went bald after a power loss in the salon reminded me of a test I frequently forget to run.

Power off your system or server while in the middle of testing and look for problems.  You’ll most frequently find issues if you aim for a power outage -

- in the middle of a network communication (eg. client-server apps or network games).

- in the middle of a multi-step transaction.

- in the middle of disk operations.

The last one’s pretty brutal, but sometimes you need to know your application can survive it gracefully.  Have fun!

As much as I love using Ruby, it’s become more frustrating over time.  Developers are feeling this, and as a tester, I’ve had my share as well.  This article, passed on by James Ladd sums things up for me (for testing, as well as development).

The best example of this in the Watir space is probably here – http://wiki.openqa.org/display/WTR/Pop+Ups.  I recently lost half a day because apparently Watir’s ‘click_no_wait’ doesn’t work in the release version of the 1.8.6 Windows one-click installer.  Of course, you have to use the beta release that came out after the 1.8.6 release, which is a bit trickier to find now that the one-click installer is not really supported (there’s a new Windows installer).

Let’s hope things start to stabilise again soon.

I was surprised to learn today that symbolic links are possible in Windows XP.  I’m not sure I’ll use them that often, but it will help me in replicating some linux build environment instructions.

Though not supported out of the box, you can download the tool that makes it happen at Microsoft’s sysinternals website.

I found it via this site.

Once you’ve downloaded it, add it to your path and then simply:

junction <linkname> <existing directory>

So if you had a folder c:\Program Files\Apache Software Foundation\tomcat and you wanted to reference it as c:\tomcat, you would run:

junction c:\tomcat c:\Program Files\Apache Software Foundation\tomcat

Windows 7 and Vista apparently support this via the ‘mklink’ command.

Safe from hackers, safe from users.  Movable type takes Captcha to the next level!

I’ve been finding popups in Watir a pain of late, with most of the solutions on watir.com not working for me.  Finally today, I found something that worked.  Steve Swanson’s solution did the trick.

However, before it could work, I had to figure out that the AutoIT dll wasn’t correctly registered.  If you try the solution above and find it to be behaving mysteriously, then try the steps on Watir.com’s FAQ page.

Today I found out that my post on Watir and TinyMCE let someone dump their Quick Test Pro licences.

A victory for common sense…

Nothing against QTP, but if you’re testing web applications, you should probably be looking at other tools.

According to the dev team, this is the muppet representation of me as a tester -

But as well as straying into Sam’s stern, righteous, “everything’s wrong” attitude on (rare, I hope,) occasions, I think aspects of my testing persona could just as well be represented by some different muppet outlooks:

Animal – The performance and stress tester.  High-volume test automation.

Beaker – The black-thumb tester.  Something baaaad is gunna happen.

Kermit – Maintainer of relationships and manager.

Miss Piggy – Hiiiiiiya!  Throw in some occasional defending of the weak as well.

Swedish chef – Just mix it all up and wait for chaos (and Borking) to ensue.

Gonzo – “Do you really think this will work?” – “No! Isn’t it great?!”

Dr. Bunsen Honeydew – The analytical tester.

Scooter – The gofer, assistant ( to developers).

The Count – Obsessive attention to detail and number wrangling.

Floyd – The creative tester.

Which muppet suits your testing personality?

Continuing the ‘what tool’ theme from last week, today’s topic is ‘Diff’.  

I frequently install windows versions of various Unix command line utilities via the Gnu Utilities for Win32 project.  Diff is particularly handy not just for the programming side of automation, but also for comparing output files from automation as well as database queries.  Occasionally though, I need the niceties of a graphical tool that handles side-by-side comparison of file differences a bit more nicely.  So I installed KDiff3 (http://kdiff3.sourceforge.net/), and it seems pretty good, supporting three-way comparisons.

I also looked at Winmerge which has a portable version.  It seems to have a nicer file diff view, but KDiff has a nicer view of folder differences.  Given that it’s portable, it will go onto my tester toolkit (although there’s some evidence KDiff may be portable enough for my needs).

Are there other diff tools I should know about?

I had a nagging feeling that I was missing a couple of posts after my wordpress death and subsequent server move. Last night I’d flushed that thought from my mind, but today I found out what they were, and thanks to Google’s cache, they’re back up again.

The first was a piece on implementing a moving progress bar for a Windows command prompt in Ruby.

The second was a solution to a JMeter documentation issue.

It’s nice to have them back, as they probably weren’t up long enough for anyone to read last time.

Now to find those comments…

The topic of interviews came up while chatting with a friend recently, and I noticed that the Dr Dobb’s interview I did a couple of years back is no longer available.  I’m reposting it now to have my own copy and to save me from having to dig it out of the Internet Archive in the future

Originally featured in ‘The book of testing – Thoughts from a Braidy Tester’ at http://www.ddj.com/blog/debugblog/archives/2007/12/five_questions_41.html

by Michael Hunter
December 18, 2007

By The Braidy Tester
Posted: December 18, 2007 07:30 AM
Once upon a time Jared Quinert tested console games. Then he led teams of testers of console games. Then he built the testing department for a console game which never actually shipped. Currently he is spreading the context-driven testing love throughout the corporate world as a consultant with Aegeon, one of those Web 2.0 companies.

Here is what Jared has to say:

DDJ: What was your first introduction to testing? What did that leave you thinking about the act and/or concept of testing?
JQ: I started as a games tester at the beginning of 1995, catching the tail-end of ROM-based games and the transition to CD-based PC and console game titles. Due to the expense of a product recall, those games needed to be rock-solid. Back in those days, our test approach was 100% exploratory, manual testing.

On my first real project, a bug report was thrust into my hand at some point, and I was told to wander around and ask the programmers what they had been working on and what they had changed in the build they had just given us. They would share information about things they thought were complex, risky, or otherwise. I would tell them what kinds of things our team was finding in the product.

After four or five months I was thrown onto a title as test lead, managing one other tester. Suddenly managers were wandering up to me and asking ‘How’s it going? When will we be done? Can we ship it?’ Pretty quickly, I needed to figure out how to answer those questions, and thus I learned about developing coverage models to support exploratory test management.

The company I was at also developed the first third-party Windows game for Microsoft. Microsoft came to us wanting a heavily scripted test approach. Our test group wasn’t even aware that people tried to test things by writing all their test ideas down. We definitely felt we weren’t going to test *our* product that way. After initially rejecting the idea, I did begin to think about what problems they might be trying to solve with their approach. Some of the ideas were adapted and found their way into our regular testing to solve particular problems. So while it was great to get the testing grounding that I did, I’m also grateful that my view of things was challenged fairly early on in my career.

The big things my first job left with were:

• The idea of testing as a service role, and information provider. This came from the relationships we had with developers and producers.
• The idea that exploratory testing, done well, can deliver quality far above what I’ve generally seen using heavily scripted testing approaches.
• The knowledge that if talented people do exploratory testing day in, day out, they get really good at it.
• The idea that face-to-face communication is critical. We moved offices, were split up across multiple floors of the building, and bad things started to happen.
• The knowledge that having skilled testers allows you to travel much lighter.
• The idea that testers have to tell a compelling and realistic story when reporting bugs. We were often representing the concerns of important people who didn’t have a voice on our team: Gamers of different skill levels, with different preferences, publishers and support people.

My understanding now is that not all games testing groups rely on smaller, skilled teams, so I could quite easily have come out of a games testing role with a very different view of testing.

DDJ: What has most surprised you as you have learned about testing/in your experiences with testing?
JQ: The big surprise was how useful the skills I developed as a games tester were when I moved to corporate environments. I think this was largely due to the testing philosophy instilled by my first role. I was also surprised by how much worse I perceived the testing in those environments to be. Using exploratory approaches, it was really easy to find bugs that had been lurking in systems for years. Given the potential financial losses involved in many of these systems, I had expected to encounter testing even more rigorous than my games experiences.

Testing for internal audiences I find is a much vaguer experience than consumer product testing. The politics are more complex, and problems are solved in different ways. Learning to be an effective tester in different environments is a huge, probably career-long task.

DDJ: What is the most interesting bug you have seen?
JQ: The most interesting games-testing bugs were usually needle-in-haystack kinds of problems. I remember one of the first bugs I found, which was a crash that happened only after playing seventy percent through a game, then applying an obscure button combination while attacking an enemy which only existed in this one room. I then surprised myself by playing the game back through to that section and reproducing the problem first time. I think that as the programmers later explained to me why it had crashed, I connected the dots and the bug value of null pointers was suddenly burned into my brain.

I think another thing that never ceases to surprise me about good testers, and that’s how much bad luck seems to follow them around when they’re testing. Bugs seemingly fall into their laps. While this may seem like an unfair advantage, I think it needs to be backed up by good investigative skills and the tester’s ability to pull the pieces together and retrace their steps.

DDJ: What do you see as the biggest challenge for testers/the test discipline for the next five years?
JQ: I see a few challenges. Firstly, that business right now seems particularly keen to reduce the cost of testing. Secondly, developers are becoming interested in testing, and getting better at it. At least in my part of the woods, much of what passes for testing is heavily requirements-based, and fairly clerical. Between outsourcing and enthusiastic developer test automation efforts, traditional tester roles are going to be under increased pressure to deliver value. If we want to have successful, satisfying careers as testers, I think we are going to need to skill up and figure out how to keep adding (enough) value.

The third key challenge is figuring out how to get some experimentation into the workplace and push the boundaries of what we know about testing, in order to test better. Persuading businesses to take on appropriate business risk for an appropriate reward seems like a hard sell from my view on the ground.

DDJ: Is there anything else you would like to say?
JQ: I’ve sort of undertaken a mission to change the testing landscape here in Melbourne, Australia. It’s a slow process, but a community is growing. If anyone feels that they can contribute, I’d love to hear from them!

——————————

Find the original copy at http://web.archive.org/web/20071220220538/www.ddj.com/blog/debugblog/archives/2007/12/five_questions_41.html

I’ve added a couple more agile haiku. The essence seems to be getting a bit less essential, so I think at some point a refactoring is going to be in order.

When my testing gets technical, there are a lot of things that I only have to grapple with infrequently. Regular expressions are one of those in-again, out-again things for which my expertise varies depending on when you ask me. Today’s Ruby hacking saw me find RegExr which helps you build and test regular expressions, as well as having a lot of ready-to-go samples built into it.  You can also download it as an Air application if you need it locally.

I always seem to end up at regularexpressions.info as well to get help, but had issues with their regular expression tester today.  While the bugs were a hassle, it helped me in a way, because testing your regular expressions with a few different regular expression testers is always a good idea if you think you might need to use them in multiple environments or tools.

Coupled with JEdit or Ruby, regular expressions are a powerful part of your tester toolkit.

What other tools for regular expressions are you using?

A good testing habit when working with web apps is to monitor the log files of servers as you test.  In some cases this is easy, especially where there’s a single application server.  With the trend toward more service-oriented architectures, and server clusters for high-traffic applications, the environments I’m working in tend to have many log files spread over multiple servers and folders.  My old friend, ‘tail -f error.log’ becomes a bit more difficult.

I’d heard about Ruby’s SSH library quite a while back, but only recently have my spare time and my memory conspired to work together. The library lets you connect to a server via SSH, execute commands and see the result. So today I quickly hacked out a script to let me watch multiple (Linux) log files at the same time.  This might work for other environments that support SSH, but I only have it connecting to Linux servers at this time.

Code is below, very hacky, and tailored to my needs.  If you have any questions, leave a comment, but hopefully this helps someone get started!

To install, use the instructions for NET::SSH version one at http://net-ssh.rubyforge.org/ssh/v1/index.html. This will actually install version two though, so you’ll need the docs and examples from http://net-ssh.rubyforge.org/sftp/v2/api/index.html

require 'net/ssh'

class Logfile
  def initialize(name,server,filename)
    @name=name
    @server=server
    @filename=filename
    @current_file_size=0
    @ssh_server='server'
    @username='username'
    @password={:password=>'password'}
  end

  #You'll need to customise this method so that it generates the
  #full path of the log file you're interested in.  This one is for Splunk logs.

  def full_log_path
    today=Time.now
    day=today.day.to_s.rjust(2,'0')
    month=today.month.to_s.rjust(2,'0')
    year=today.year.to_s
    todays_log_folder="#{year}-#{month}-#{day}"
    full_path="/var/log/#{@server}/#{todays_log_folder}/#{@filename}"
    return full_path
  end

  def get_new_lines
    Net::SSH.start(@log_server,@username,@password) do |session|
      new_file_size=session.exec!("wc -l #{full_log_path}").split(" ")[0].to_i
      lines_to_get=new_file_size-@current_file_size

      # Don't generally need to get everything if the error logs are being flooded
      lines_to_get=100 if lines_to_get > 1000 

      new_logs=session.exec!("tail -n #{lines_to_get.to_s} #{full_log_path}")
      @current_file_size=new_file_size
      return new_logs
    end
  end

end

logfiles=[]
logfiles.push Logfile.new(:log_name1,"log_app_folder1","log_file_name1")
logfiles.push Logfile.new(:log_name2,"log_app_folder2","log_file_name2")

5.times do
  logfiles.each do | logfile |
    changes=logfile.get_new_lines
    puts changes if !changes.nil?
    $stdout.flush
  end
  sleep 10
end

Given the lack of posts over the last 12 months and the various problems with my hosting, I’m surprised I scraped in.  But it’s nice to know anyway -

http://www.testingminded.com/2010/04/top-100-software-testing-blogs.html

Thanks to all of you who’ve been following along for the last few years.