When you are running an integration test of any sort, one of the things you need to deal with is configuring the other component(s) you are interacting with; either for step 0 or later. There are a couple ways that you can deal with this, each with their pros and cons. And unsurprisingly, there is a order I would approach these.

The easiest thing to do is create a script that executes all your setup in the other system(s). This is great since it is actually running through the end user process of creating the data in the system. Good or bad, it is ‘real’. But let’s face it, Selenium tests are not really zippy to begin with and what if you have a half dozen of these scripts to run? Well, your feedback loop is getting longer and longer. I’m not sure how much parallelization of tests will even solve the setup problem. This slowness of execution is offset though by the speed of creation (when prototyping with Se-IDE).

What if you could have the benefit of using properly generated data but without the speed penalty of using a browser? Well, if you are using something like Rails you can make use of its RESTful dna. Properly constructed, creating data becomes as easy as POSTing a properly formatted string to the server and voila!, you have data. Yes, the user isn’t creating it directly, but if the app is properly constructed then all the same business rules will be triggered as well.

But using a RESTful interface still means a network hit as well as being at the mercy of another system’s state. You can get rid of that by appropriate use of a Test Double. In this case, what you likely want is a Stub. A Stub returns canned data in response to certain criteria in the application. This breaks the very notion of ‘integration’ testing by removing the integration, but lets you quickly test the ‘as long as the other end continues to send information in the format I expect’ scenario. And that is a pretty important one. You have to remember to remove the stub every so often to make sure that it is still be getting the same data though.

You could in theory jump straight to the Stub step if you choose, but I think there is value to going through all the steps. First, you verify what is being sent by the app. Next you verify that the business logic is not in the presentation layer by replacing the browser by an API and then you supercharge the script execution.

And all the while you are learning about the application you are testing — and learning is kinda the point of testing.

I’m working on automating up a client’s AJAXy application right now and managed to get myself very confused frustrated yesterday over retrieving a value that was updated through a JS event.

For context, there are a bunch of fields that can have numbers entered into them and on each key press a running total is updated via Javascript and replaced in the DOM.

Because the field that gets updated is an input one, the content of it is stored in the value attribute.

This is a RoR application, so I’m using the selenium-client gem. To get an attribute named ‘value’ using it you would do the following.

@selenium.get_attribute "//input[@id='currency_count_total_value']@value"

And that works fine — until it is updated through Javascript. On the page the field will be updated but that snippet will always return the original value since the page hasn’t actually been updated — just the DOM.

So how do you get the updated value? Well, after an embarrassing length of time the magic is to use the get_value method.

@selenium.get_value "currency_count_total_value"

Now, in theory they should do the same darn thing. Reality is different. Yesterday evening would seem to imply that get_attribute works on the most recently updated html retrieved from the server whereas get_value works directly on the DOM.

At some point when I wasn’t paying attention Selenium grew the ability to take screenshots. The two commands of interest are captureScreenshot which takes an image of what is viewable in the the browser and captureEntirePageScreenshot which captures the entire content of the browser; visible or not.

Screenshots and automation have a dubious history. Too often they get used to determine error conditions which leads to ‘the test failed because the pixel was off by one even through it actually passed functionality wise’. That is exactly opposite to how I would recommend you use this capability.

Instead I think you should incorporate it two ways.

1. Ack! Something went wrong! – Any time an assertion fails, take a capture of what is being displayed on the screen. This could save you the time of running back through scripts to get to the error point and if it is an intermittent fail you can capture those states.
2. As an aid for humans – The ability to drive different browsers is one of the great selling points of Selenium. But any tool like this is great for functionality, but really bad for layout and presentation problems. By wonderful coincidence though, humans are great at it. So why not make a script to capture each page in your application and stash it somewhere for a human to look at later?

This doesn’t mean that people have not been successful doing screenshot comparisons before. At CAST 2007 in Seattle, Harry Robinson demonstrated how Google Maps used image comparison when figuring out the driving directions algorithm. And at GTAC 2009 there was a presentation on Fighting Layout Bugs – Techniques to automatically verify the work of HTML and CSS programmers which Jeff Fry summarizes quite nicely. Here is the recording of that session.



So yes, it is possible to do automation using screenshots to determine pass/fail conditions. But for most projects they infrastructure and time to set it up is not worth the return. Instead, use them as tools to assist in figuring out what went wrong in a script or to capture information for later human analysis.

• Selenium RC 1.0.3 is released
• Selenium IDE 1.0.5 is released!
• Mogo is making web testing easier; invitation alpha now on
• Step six of Oracle ADF Development Essentials is Testing Your Oracle ADF Faces Application with Selenium. I would have switched it for step seven which is CI, but whatever.
• FlexMonkium is a FlexMonkey -> Selenium bridge from the authors of FlexMonkey. I’m not sure how they are getting around the need to have an enterprise license for recording, but I’ll be talking to them over the next day or two.
• Learning Python is not about Selenium, but about why learning a scripting language has helped Lanette Creamer do her job better and includes ‘Since I’ve been working more with Python I can now understand Java better! I can read other people’s code and follow it more often. I’m pleased that I can get test ideas from what I think is missing in the code, not just what I see that is there.‘. If you are using Selenium, you should know a scripting language.
• While not new, 10 Tips for Building Selenium Integration Tests saved me some time last weekend.
• If you are using Rails and Sauce Labs OnDemand, the folks at Pivotal Labs have written a saucelabs-adapter
• Chris Bedford at the San Francisco Java User Group presents talking about Selenium.
  
  
• 2010 CWE/SANS Top 25 Most Dangerous Programming Errors – do your scripts check for these?
• Jason Huggins at the Denver Agile User Group talking about the Future of Automation
  
  
• The next London Selenium Meetup is April 20 and will be on Selenium 2 — going to record it guys?
• Announcing Selenesse, the modern Fitnesse -> Selenium bridge
• Flow Control and WebDriver Se-IDE plugins
• SeleniumHQ has a new blog
• If you are using Se with Rails, you really owe it to yourself to use mechanize
• The CoScripter Reusable History plugin was this week’s Sikuli. The only thing I could see steal, erm, borrow-able is the exporting to human friendly text part. Who is going to write that formatter for Se-IDE?
• Google Summer of Code is on the horizon. What Se projects should we submit?
• If you want the bleeding edge Se-IDE builds, you can get them out of the OpenQA Continuous Integration server. (Click the current build, then Artifacts)

I’ve just old (young?) enough to say that I have spent the majority of my life online. I remember when search engines were ‘new’ and a time where Search Engine Optimization (SEO) was not even a thought. So I’ve seen SEO be born and enter its unruly teenage years and so far I can say pretty confidently that companies offering SEO services for high dollar per hour amounts are a scam.

SEO is however a reality. The internet is so vast that discoverability is getting harder and harder even if you write content that people want. And because it is a reality, it is yet another thing that testers need to keep in mind when testing web properties.

The Art of SEO is the new tome from four SEO ‘experts’. The fact that experts is in quotes highlights one of my main objections to any book on SEO: there are no experts and it is a constantly moving target. The rub is that none of the major search engines provide their full algorithms for search discoverability or indexing so how can anyone claim to be an expert in it? Also, these algorithms are constantly evolving — often in a direct response to things happening in the SEO realm. That said though, if you are going to have someone write a book on such a subject, these four seem like a good choice.

I’m not going to even try to hide the bias I have towards the SEO industry as a whole, so why on earth would I get a 575 page book on the subject? Simple. I wanted to make sure my ‘SEO is a scam’ claim was correct — and I maintain that it is. But that doesn’t mean you shouldn’t buy this book. While SEO itself is a scam, ‘The Art Of SEO’ is not one. Especially if you are new to the area.

There is nothing in this book that could not be found online, but the ironic value in the book is that you don’t have to search for it. If you test an internet based web application, here are thing things I think you need to know about SEO — all of which are covered in the book.

• Spidering – How spiders work, how to control their access to your site, and how to make their job easier
• Structure – There are ways to structure content to legitimately enhance its searchability without being scammy
• Redirects – How to deal with moved or removed content in such a way as to inherit prior page value
• Rich Media – Flash, images and audio all present challenges
• Types of Search – Horizontal, Vertical and Local search all have different needs

What is also important is the couple pages about ‘Grey / Black Hat’ techniques. These are the tricks that raise your search ranking for a bit, but are often frowned upon by the search engines themselves and could get your delisted. The less that reputable SEO outsourcing companies often employ these techniques so people need to know what they are and stay away.

There are a few things I don’t agree with in the book (of course). Primarily, the notion that runs through the book is that SEO should be first and foremost when planning a website. That is suits the needs of the users almost seems secondary. Being a book on SEO though from people who make their living on SEO, this bias is understandable. I would counter that if you forget the user experience though then it doesn’t matter that people found your site as they won’t use it.

‘The Art Of SEO’ is not really a read-cover-to-cover book. Instead I would recommend picking a topic (like redirects), skimming the places where it is discussed and then see if your site needs tweaking. It is not exactly a heavy read so you should be able to get maximum value from it pretty quickly — I knew 97% of the content already and went through it in 2 swimming lessons and a snowboarding lesson.

Do I recommend ‘The Art of SEO’? That depends on your prior knowledge in the area. If you have been creating content online for awhile and have picked up how to do SEO naturally then probably not. But if you are new to testing web applications or management of them, then there is a decent amount of value in the book and it would be a good addition to your bookshelf. Especially when you compare the $56.99 Canadian price tag to what so called SEO experts charge to do stuff to your website that you are more than capable of doing for yourself.

I’m a mentor for a class at U of T this semester where a group of students are trying to crack the ‘Flash in Selenium’ nut. They are at the point where they are having planning / status meetings and I had to be downtown anyways so attended this week’s.

The first task they had to do was to individually produce the Elevator Pitch for their project. They had 3 minutes to write the pitch and then they had to read them out loud. The discrepancies, and there were some are signs of divergences within the team. These divergences are warning signs that need to be paid attention to.

Immediately.

I advocate elevator pitches for individuals and their careers but had not seen them used in a project beyond the initial kick-off until now. I think this has officially been added to my retrospective repertoire.

After talking to a number of people, Se-IDE is going to be growing a new pane in the Option panel to manage your plugins. Actually, it already has as of 1.0.5. In order for a plugin to utilize this new panel it has to register itself. Ideally this would be a mandatory thing, but overlays don’t allow that (so far as I know) which means it is up to the plugin developer to play nice.

Registering a plugin is easy. In one of your XUL files where you have an API object, just call the addPlugin method with the id of your plugin. This id is the one you specified in your install.rdf file.

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
 
<overlay id="preflight_extension_loader_overlay"
         xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
         xmlns:html="http://www.w3.org/1999/xhtml">
    <script type="application/x-javascript" src="chrome://selenium-ide/content/api.js"/>
    <!-- <script type="application/x-javascript" src="chrome://preflight/content/preflight.js"/> -->
    <html:script type="application/javascript">
        var ide_api = new API();
        ide_api.addPlugin("preflight@adam.goucher");
    </html:script>
</overlay>

Eventually this new pane will display the plugin’s version/author information, options and even controlling whether it is enabled or disabled.

So it has been just over 2 months since the last Smattering of Selenium and a lot has happened; not including starting my own Selenium consultancy / startup called Element 34 and inked a deal for a book on Selenium. Here are the browser tabs that I have had open for far too long that would be relevant to Selenium users.

• Selenium IDE – 1.0.4 Released – the first release of IDE in a while. The next one should be out this week.
• Sauce Labs has releases their commercially supported versions of part of the Selenium suite: Sauce IDE and Sauce RC
• Not to be out done, BrowserMob has been very busy as well with new features, Se2 support and another cloud location
• While on the subject of BrowserMob, here is the BrowserMob Proxy which will let you do a bunch of games to your traffic
• If you are using xUnit.net with Se-IDE you will want to look at Web testing with Selenium and xUnit.net
• Screenshots are generally overused for the value they deliver in automation, but here is a good use of them
• And here is how to do screenshots with the robot framework
• One trick I have heard about to make interaction with ‘black box’ technologies like Flash, Flex and Silverlight is to turn on accessibility. Here is how to enable accessibility in Flex
• The project with the most presence right now in the ATDD/BDD space is Cucumber, but it does come with a learning curve. Here is a presentation on Behaviour driven development with cucumber
• Now, if you are using Cucumber with Selenium you will likely be using either Webrat or Capybara. Well, it seems that they are considering merging
• Speaking of ATDD, here is how to do it with PHP and Se
• Se2 examples are starting to surface; here David show us Se2 with IronPython — this is something I will likely need towards the end of the month for a client
• Here is another plugin for Firefox to add to your stable: Firefox Alert Closer — which does, well, what the name suggests

I’ve had this podcast in the queue for a long time. Privacy has become one of those things that testers need to always have at the back of their mind when testing. Understanding Privacy is a book by Dan Solove was on the podcast circuit promoting and had a number of excellent points around the subject in this interview.

• People don’t understand the implications of giving up information
• Because we don’t know how information might be used in the future, holders of the information doesn’t even know yet, making an informed decision is really hard
• What does privacy even mean? Usually its a vague an amorphous concept
• By contrast, the things that privacy is often traded off against, such as Security, are easy to define so they win out
• Privacy isn’t one thing, but many things
• Past attempts to define Privacy have been too broad (‘Right to be let alone’) or too narrow (‘intimate’ or ‘client confidentiality’)
• Because of the diverse areas affected by Privacy, there is not just a single solution to it, but a basket of solutions for each problem
• It is not about being paranoid, but about doing things with your eyes wide open
• Unlike a car accident, the problems that Privacy issues cause might manifest many years down the road. The lack of immediacy is a key problem in education and recognition of risks
• Most people think that because a company has a Privacy Policy it means that the company won’t share you information. Umm, no.
• Don’t think that this isn’t just a Consumer problem. This is hard to manage and build out from a developer / implementation perspective.
• There are four broad categories of problems around Privacy. The idea is that with such a taxonomy you can understand which problems (and there is often not just one) that need to be addressed.
  • Information Collection
    • Surveillance – when it inhibits personal information or used in an unrelated manner
    • Interrogation
  • Information Processing
    • Identification – linking people to information streams (think Gattaca)
    • Aggregation – combining bits of information to make something that is greater than the individual parts
    • Security – is the information kept securely?
    • Secondary Use – when data gathered for one purpose is used for others
    • Exclusion – denial of information regarding how data is used
  • Information Dissemination
    • Disclosure – spreading of facts
    • Breach of Confidentiality – betrayal of trust
    • Exposure – not reputation; but deep embarrassment
    • Accessibility of information – obscure data being made less obscure
    • Blackmail
    • Appropriation
    • Distortion
  • Invasion
    • Intrusion – spam for instance
    • Decisional Interference

Fortune has their annual list of the 100 Best Companies To Work For out now. What I think is more important that where companies ended up, or their trending in the ranking is what the blurb that accompanies each entry has. The reason? That is the ‘why’ of the choice. The why gives insight into the culture, the very fabric of the company. Having worked at a number of organizations, a toxic culture can be no good at all.

The big patterns this year are: caring about the people who work for you, investing in people, reward community involvement and low turnover.

Here are the ’sound bites’ I lifted from the list.

• Trust between our employees and the company
• Salaries were frozen, but profit sharing continued
• 11% of the workforce, have been here more than 15 years
• Culture of collaboration and openness
• Good people can do their best work
• “No asshole” rule
• “If you tell anybody what to do here, they’ll never work for you again.”
• On-site child care
• Your time off is your time
• No wonder full-time turnover rate is just 3%
• Always “do what is right.”
• Mentors
• More than 1,000 new employees were added in the past two years, 40% from referrals
• Culture of openness
• Employee referrals account for 85% of new hires
• Restocked its vending machines with healthy snacks and nuts, and employees get checks ranging from $100 to $300 for trying to get fit
• Treats me as a peer
• Fourteen “junior boards of directors” give employees at all levels input into decision-making
• Employees who volunteer are rewarded with vacation
• The most generous sabbatical program in American business
• “No job is seen as better than another job, and no person is seen as better than another person.”

For the last couple years there has been a growing collection of people rallying around the software craftmanship. From an outside perspective, it appears that they are dipping into the classic apprentice – journeyman – master model for growth (without the inclusion of guilds which is often the point of fear with this model). From its title, Apprenticeship Patterns – Guidance for the Aspiring Software Craftsman would be aimed only a the beginning of the road and only for developers. Both would be wrong. This book is of value to anyone in a knowledge role and at any point in their development.

Dave Hoover and Adewall (Ade) Oshineye have collected together a series of of almost 40 ‘patterns’ that you can use, and likely already are, to assist your growth. Each pattern follows the same structure by defining the context, giving a sample scenario where the pattern might solve a particular type of problem, a solution and action that the reader can take immediately to start implementing the pattern. Because of this structure it is easy to jump at random throughout the book and [start to] digest a pattern in a single (brief) sitting. The real power of the patterns for people starting out is in the Action section I think, though I admit that I have not followed through on most. These are the ‘get off you butt and do something about your growth’ motivators.

Having embarked on my path already I found that I was already implementing a number of these patterns which I think completely validates their inclusion in the book. But it also meant that the Solution section was what got my attention the most with lots of ideas how to refine my implementation of the patterns. Flipping through my copy of Apprenticeship Patterns I can find only 2 or 3 patterns that I did not underline or star or otherwise mark up.

My favourite patterns (both from the book and the real world)?

• Breakable Toys – Experience is built upon failure as much as (if not more than) success
• Find Mentors and Kindred Spirits and Rubbing Elbows – Seek out those who have gone ahead of you and strive to learn from them
• Record What You Learn and Share What You Learn – Start writing. Start writing now!

People who know me likely won’t be that surprised with that list. Your list, almost by definition will be different as you have walked a different (though perhaps similar) road in your journey.

My only knock against the book is likely its price. It is 130 pages and $37.99 CDN. That is steep, though I suppose in-line with current pricing structures since Beautiful Testing is 320 pages for $62.99. To their credit, the patterns could at one point be all found online (minus some polish that comes from the editing process) but I couldn’t find them at time of writing. But a book like this really should be consumed in dead-tree format where you can make notes in the margins. (If someone finds the wiki link again, please post it.) And at the risk of sounding like I’m pimping my book, Amazon has been long running a deal where you can save if you buy Apprenticeship Patterns and Beautiful Testing together. While likely related to release dates and ish similar breadth of content, I’m ridiculously happy to see it.

The Agile community is in the midst of an obsession of Shu Ha Ri, and much to my annoyance I admit that there are different ways that people learn at different stages of their learning. I don’t know if there is going to be ‘Journeyman Patterns’ follow-up, but I hope there is. But even if there is, Apprenticeship Patterns is staying on the bookshelf where it is nice and handy.

I expect that the majority of plugins that people create are going to be around adding custom functionality or formats, but sometimes you will want to tie Se-IDE to you service. (Much like Sauce Labs does with Sauce IDE.) Typically that tie-in needs some sort of credentials.

There are two different ways to do this. First, you could just store things as a preference and be done with it, but that has a bit of a problem around it. Namely, it gets stored in the clear in your profile’s prefs.js file and (depending on your permissions regime on disk), anyone could read it and get your credentials. This is not quite ideal.

The better way to store your credentials is through Firefox’s Login Manager.

Again, Preflight doesn’t have need for a password right now, but if it did, here is how you would protect it.

Add a new overlay to selenium-common.xul in the chrome.manifest

overlay chrome://selenium-ide/content/selenium-ide-common.xul chrome://preflight/content/overlays/preflightCommonOverlay.xul

And in the overlay, add a js file that is going to have out secure password manipulation code.

<overlay id="common_overlay" xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
  <script type="application/x-javascript" src="chrome://preflight/content/js/secure.js"/>
</overlay>

Here is the contents of secure.js. I’ll explain them afterwards.

var passwordManager = Components.classes["@mozilla.org/login-manager;1"].
                       getService(Components.interfaces.nsILoginManager);
var nsLoginInfo = new Components.Constructor("@mozilla.org/login-manager/loginInfo;1",
                      Components.interfaces.nsILoginInfo, "init");
 
function storeUsernameAndKey(username, key) {
    var logins = passwordManager.findLogins({}, "chrome://preflight", null, 'Username and API Key');
 
    if (logins.length == 0) {
        var extLoginInfo = new nsLoginInfo('chrome://preflight',
                               null, 'Username and Password',
                               username, key, "", "");
        passwordManager.addLogin(extLoginInfo);
    } else {
        var extLoginInfo = new nsLoginInfo('chrome://preflight',
                               null, 'Username and Password',
                               username, key, "", "");
        passwordManager.modifyLogin(logins[0], extLoginInfo);
    }
}
 
function retrievePreflightUsername() {
    var logins = passwordManager.findLogins({}, "chrome://preflight", null, 'Username and Password');
    if (logins.length == 1) {
        return logins[0].username;
    } else {
        return "";
    }
}
 
function retrievePreflightPassword() {
    var logins = passwordManager.findLogins({}, "chrome://preflight", null, 'Username and Password');
    if (logins.length == 1) {
        return logins[0].password;
    } else {
        return "";
    }
}

This is almost straight copy-and-paste from the Mozilla docs, but with an additional layer of wrapping added. The part that you will want to change is the chrome url. It should be the url for your plugin.

The reason why I wrapped things like this is that it allows us to do things like

var u = retrievePreflightUsername()

from anywhere in your addon’s js. (This works because we included secure.js at a very high level in the app.)

Some things to be aware of when using this

• This does not protect your credentials from other plugins. They can just as easily make the same calls as you did and get the information. All it does it protect it from prying eyes on disk.
• This method of dealing with password was introduced in Firefox 3.0. As a result, if you are using this you need to set the minVersion in install.rdf appropriately.

The world is full of people who don’t read English, so it makes sense that your application should cater to more of the world than just them. Firefox provides a means to display these other languages through its use of locales.

There are a couple of ways to play with locales, but the easiest way is the Quick Locale Switcher plugin.

Se-IDE currently ships with support for four locales

• en-US – US English
• fr-FR – France french (as compared to Canadian french)
• ja-JP – Japanese
• pt-BR – Brazilian portuguese

The reason I list these here is that, like most things, you need to make a chrome.manifest change for each locale you support. And since the plugin your custom plugin supports those four, yours should too.

locale preflight en-US chrome/locale/en-US/
locale preflight fr-FR chrome/locale/fr-FR/
locale preflight ja-JP chrome/locale/ja-JP/
locale preflight pt-BR chrome/locale/pt-BR/

Now, Firefox knows where to look for its language strings.

Believe it or not, if you have been following this series you are already used to making use of the localization framework. Recall that we have been putting the following line in our .xul files?

<!DOCTYPE prefwindow SYSTEM "chrome://preflight/locale/options.dtd">

Through a bit of internal magic, the ‘locale’ part of that gets switched into whichever locale you have selected so in my ‘normal’ case the url is really chrome://preflight/chrome/locale/en-US/options.dtd. With that url loading in the overlay we can now pull locale specific strings from it.

Right now, it only has one string in it.

<!--en-US-->
<!ENTITY performpreflight "Perform preflight checks">

To reference that string you use the entity name preceded by a & as shown in this snippet of overlay.

<hbox align="center">
  <label control="name" value="&performpreflight;"/>
  <checkbox preference="pref_perform" id="perform" />
</hbox>

Ideally, every string that is presented to the user in your overlays would be handled like this. A good way to test this is through a technique I call LOUD which I will let you research on your own a bit.

But referencing DTDs in xul is only half of the L10N story as far as our plugins go. Plugins are a combination of XUL and JS.

The way you localize the JS is very similar to the XUL only instead of using .dtd files you use a .properties file. Now my Preflight Checks plugin is still too simple to have JS that displays something to the user, so I can’t really make use of this. But essentially, this is what you need to know…

• Property files have the syntax of key=value
• In your overlay, use the stringbundle element to inject it into the DOM
  

  <stringbundle id="options.strings" src="chrome://preflight/locale/options.properties" />

• Get a reference to the bundle from the DOM and then extract the string
  

  var strbundle = document.getElementById("options.strings");
  var nofilesfound=strbundle.getString("notFoundAlert");

That’s it. It can be a bit of a pain when creating screens to now have to put text into multiple files when working, but it is an absolute pain to have to find all your strings after the fact, so just get in the habit of writing L10N clean code from the start.

Oh, and here are two links that you will want to have handy when doing this step are

• XUL Localization
• Property Files

So I went to the monthly gathering of TASSQ this evening. I figured that I should probably go to a meeting or two before I’m slated to talk at the March one and now that I’m independent I should make myself known in various circles. What I discovered appeared to the a bastion of ‘old school’ qa (not testing).

Now, I know this post is going to sound like a bash, and it’s not really meant to be one. It is more rooted in incredulousness. I know it is hard to get up in front of people and expose your ideas and ways of doing things to others. But you have to remember, that when you do, you open yourself to this sort of thing.

Oh yes, and it is also important to realize (as most who read this blog likely do already) that I’m philosophically aligned with the Contextual, Agile and Craftsman schools of thought.

What was the name of the talk?

Factory School Testing Revisited / Redacted / Re-energized. aka Kickin’ it ‘Old School’

Gah! Right there I knew I was in trouble.

Anyhow, here are the notes / impressions / ideas I had during it.

• One of the stated goals was to have a topic that resonated with folks in the room. I’ve only been to a couple TASSQ meetings before, but if this is the topic of interest then it kinda says that the group is not concerned with new ideas.
• I haven’t worked in a regulated environment in a long time so I could be wrong, but don’t the regulatory bodies care about types of testing you are going to do before hand and the results after? I seemed to get the impression that one of the large arguments for creating a large body of test cases ahead of time was to show the ‘depth’ (or something) of your ‘to be done’ activities. Well, that doesn’t prove anything. What matters is what has been done. I could say that as part of the testing I’m going to build a rocket and go to the moon! Results are what matter.
• Also provided on the same slide was the notion of resource variability. ‘We have high turnover of QA people so we need stacks of test cases to bring the replacements up to speed’ is my paraphrase. Well, if you have high turnover, then I would say that is a smell that you are doing something wrong. And what about turnover in test management? I’m guessing it has the exact opposite pattern — the entrenched stay entrenched and those that challenge or deviate from plan are removed.
• He had a nice metaphor about Christmas and testing. Testing is like Christmas morning; when you wake up there are gifts lying under the tree. But they are wrapped and you can’t see what is under the paper. And sometimes you like what you get, but sometimes it doesn’t fit and you have to return it. This can sometimes hurt the feeling of the giver, but often they understand. And there is no guarantee the replacement you get is going not have a page missing or some other thing wrong with it that you don’t discover until the next day. So the cycle repeats itself.
• I’m not the best speaker, but there is likely a business opportunity waiting to be exploited in teaching testers how to present to small-ish audiences of strangers. How to speak clearly, to the audience, without fidgeting and with a nice slide deck. I wonder how my Lessig-esque deck is going to go over in March. Care to bet the comments that come back are that I need more bullets?
• Not once were the people involved referred to as ‘testers’. It was always ‘QA’. What you name something is critically important.
• What I think was most scary was that I can remember a time when everything that was said would have rang true to me. I’m not saying that Rapid Software Testing threw me from my horse as I was likely travelling down that road anyways, but apparently the big Canadian banks need some heavy introductions to Context techniques and ideas.
• I sat next to Fiona Charles who offered up to a discussion of metric a rule-of-thumb I hadn’t heard before. 3:3:1 – For every three test cases you write, you will find one bug. For every three bugs you find, one will be severe enough to warrant fixing. If you application is producing more bugs that the model, then you need to look at the quality of the application. If however you are finding less bugs than that, you need to look at the quality of the test cases.
• If metrics and counting test cases executed is so important, isn’t it rather ironic to encourage testers to go ‘off-script’? You number are wrong(er!) then by your own admission.
• I did like that it was stressed during the metric part that it is not the number exactly that matters on the graph. It is the pattern that it depicts that counts. Oh, and that it might take 20 minutes to generate the numbers daily; of which 15 minutes of that is writing the summary and story (context) around them.
• My big issue with the Factory is around its reliance on test cases. So I asked a couple questions at the end about it (in as non-obnoxiously as I could). And through that I had to explain the Mine Field problem. Sigh. I know I associate with people who value constant learning as a core attribute, but c’mon! Learn the core theories of your craft! Someone in the audience added that in real-time/safety-critical systems they cover the system 100% thus removing all the mines. Except, that is impossible. You’ve covered every scenario and path you could conceive of at the time. But unless you test every scenario, every interaction, by every person, ever to use your product, ever, then you don’t have 100% coverage. Even if you have 100% code coverage, which I admit is likely what he meant, you can’t account for oracles like emotions at the time of use.

The Stack Overflow podcast #71 has finally bubbled to the top of the queue. It is kinda two different podcasts smooshed into one. The first half is clips from the speakers from the SF DevDay event, and the second half is the usual gang of suspects goofing off around a microphone. The redeeming part of this particular episode was Jeff Atwood’s on-stage portion. Here are the bits you should care about. (Or at least I did…)

• An interesting interview idea – Prepare a list of the smartest people in your field and present that list to the candidate. How many people can they identify? Do they argue for people you missed or that shouldn’t be present on the list? Seems like an interesting way to weed out the people who do the job just for the mortgage
• The Stack Overflow Team Motto: Feel free to fail early, often and frequently as possible in really painful ways
• Don’t phone in
• Be passionate for the ‘craft’. But not just the craft of constructing the software, but for delivery of the solution
• Would you sign your name on the product? The Amiga 1000 hardware developers did.
• Jeff recommends Coders at Work and provided some of the rationale behind it through quotes:
  • Douglas Crockford – Don’t major in CS; major in English
  • Joshua Bloch – Recommends Elements of Style. If you can be a really good communicator, thats the kind of program you will write.
  • Joe Armstrong – Advice to universities: produce graduates that can write and argue coherently
• Before you ask for help on something talk through the problem to a stuffed animal first
• Can you deploy something new everyday? Do you make it even a tiny bit better every day?
• The Train of Shame – the error logs, in real-time(ish) to show you what is busted. So important that you figure out and implement this first before you do any production code. Start and end you day by watching this.
• Take bits from different stacks as necessary. Don’t just stick with only one solution family
• The big engineering problem is going from 1 to 2 servers. After that, it is ‘easy’ to add more.
• With technology, use the lightest thing you can get away with

Part of the reason for the 1.0.4 release of Se-IDE last week was support for the Firefox 3.6 series of browsers. Now, Mozilla has a fairly aggressive upgrade process, but you can’t always trust that people have the same / current version of the browser. That means you need to test against multiple versions of Firefox.

This post explains my trick of having multiple Firefox versions on a single Mac OS machine.

1. Download as many releases as you want / care about
2. Rather than install it into Applications like you normally would, install it into a folder in Applications that relates to the version.
3. Start each one and uncheck the Preference that would have the browser check for updates

Now you have many different versions of Firefox installed. There are likely to be some headaches around plugins, etc. but those could likely be mitigated by having a different profile for each one.

What about with Selenium? To do that we need one extra step

4. Mac applications are directories called Something.app (so Firefox.app). So what you want to do is create a symlink from /Applications/your-version-of-firefox-dir/Firefox.app to /Applications/Firefox.app

Now you can run your Se-RC tests on different versions of Firefox by changing where the symlink points to. This is because how Se-RC finds Firefox on the system. Here is the Firefox 2 locator. Three is very similar.

private static final String[] USUAL_UNIX_LAUNCHER_LOCATIONS = {
        "/Applications/Firefox-2.app/Contents/MacOS",
        "/Applications/Firefox.app/Contents/MacOS",
        "/usr/lib/firefox", /* Ubuntu 7.x default location */
};

In this trick we’re making use of the middle location.

Making the rounds through twitter right now is Sikuli. I became aware of it through someone claiming it a Selenium killer. Well, not quite.

The ‘magic’ of Sikuli is that it used the actual content on the screen to find elements rather than a native API or XPath by taking screen captures and looking for image areas to interact with.

Yes, that’s correct, it used the way an object is rendered on the screen to determine whether it is interacted with or not. Didn’t industry decide this was a bad idea a number of years ago? Here is why this project fails the sniff test for me.

• What about when I create a script on one resolution and play back on another?
• What about different record and playback OSes?
• Accessibility enhancements on/off?
• Image reuse? Without this you have to change every location when something changes. And things always change.
• How do you data-drive something that requires you to have image captures of the run? Data driving is rather important when it comes to minimizing code duplication yet increasing test data.
• And since you have to have visited each screen in a certain manner whilst writing the script, you start to run afoul of both the Minefield Problem and Pesticide Paradox
• And and if you can’t data-drive a script, is there any chance that scripts would be written in a model-based manner? Highly unlikely.
• Can you record actions? I would wager that Selenium’s success can be largely attributed to the Record functionality of Se-IDE. A quick spin through their site implies that manual script creation is the only option. People who have done the market research have told me that most QTP users stay in the keyword, non-script, part of the environment.

Of course, the goal of Sikuli isn’t to solve these problems. It is a research project between two groups at MIT. This is said with some derision in my voice. Yes, some things are well suited to make the transition from Academia to Commercialization. Google came out of Stanford for instance, but it is/was at its core, a new algorithm to (re) attempt to solve an understood problem. The issue with this is that the divide between Academia and the Real World in Testing is mammoth. I’m not sure why this is exactly, but I often just shake my head when reading papers on testing by people ensconced in a University somewhere. Yes, there are some exceptions to this, but they are exceptions rather than the rule.

Now, contrary to how it may seem, I’m not completely bashing Sikuli, more just pouring some cold, hard reality against it. I understand its appeal; since it uses screen captures is can interact with anything that displays on the screen. That is close to a Holy Grail if ever there was one, but they way in which they accomplish it is terminally flawed I fear.

One place I see Selenium users stubbing their toes on time and again is around automating HTTPS connections. I’m about to tell you that you should save yourself a lot of pain and just not do it.

Having a secure connection in production is an extremely good idea. Especially at places like checkout and authorization. But in testing? Not so much.

There is exactly one situation where you need to be running HTTPS in testing, and that is if you are using certificates as an authentication means. For every other test scenario, in test, you can run things just as HTTP. Not only can you watch the network traffic in the clear over HTTP but you don’t have to worry about the trust issues certificates introduce.

And there is almost always trust issues.

Why?

Because ‘real’ certificates cost money. The whole notion of security is based around trust, and the way to establish that trust is through money and verification. If you look in your browser, it will list somewhere all the CA (Certificate Authority) root certificates that it trusts by default. Those vendors pay non-trivial amounts of money to get in there. And then they charge non-trivial amounts of money end consumers to sign a certificate with their root certificate. Production sites will (almost) always go that route since your customers are likely to be a little concerned about all the browser warnings about things being wrong with the certificate.

In test though, machines are often rebuild and used for multiple purposes so companies are loath to spend the money on real certificates. When manual testing, this isn’t anything more than a nuisance dialog to dismiss, but when automating, it is much more annoying.

The problem is that the error box is not part of the application, but is part of the browser. In Selenium 1.x there are some workarounds, and I believe Selenium 2.x and WATiR solve this by low-level control of the browser.

The easier solution though, is to just not use HTTPS in the testing environment. Sure, your testing machines won’t be as ‘production-like’ as they might otherwise be, but you have to ask yourself, does that materially affect the quality of your testing?. In most situations, it won’t. In fact, I would argue that it increases the breadth of testing available to you; both manually and in an automated fashion.

edit: To be clear, the workaround mentioned are just some of the possible solutions in Selenium 1.x for dealing with HTTPS pages. There are a lot of situations where it will Just Work. Especially in RC where the server is a proxy and certificates get injected on the fly. I would still say in those cases to turn off HTTPS though; it is really hard to figure out what is going on at a network level when it is encrypted.

Been a busy couple days in Selenium-land. On Tuesday I pushed a new version of Selenium-IDE (see the official announcement) and yesterday Sauce Labs pushed Sauce IDE. I’ve not been coy about my relationship with Sauce Labs and don’t want this to be a generic fluff post, but I think Sauce IDE will eventually be the model for building a commercial product around Selenium-IDE. And the reason for this is/will be packaging.

Firefox extensions are, at their core, just a zip file with an extension Firefox knows what to do with and have a minimum common set of configuration files (install.rdf, chrome.manifest and such). One of those configurations in the Se-IDE and Se-IDE plugins is to set the em:type to 2 which tells Firefox that it is an extension (rather than a theme or whatnot).

The ‘classic’ way for a businesses to commercialize an open source project is to essentially fork it and start along their own, potentially divergent path. One of the goals of starting to develop an API for Se-IDE was to explicitly avoid all this splintering.

The Firefox system for extension is extremely helpful in this regard. There are very few things a commercial entity cannot do to Se-IDE through overlays. This means they don’t need to fork. I fully anticipate that situation where extra helper methods, or even some refactoring will be necessary to succeed long term, but in general, companies don’t need to change the code Se-IDE code. They just write an extension.

There is still a bit of a delivery problem though. My model for the plugin idea for Se-IDE has been the relationship between Firebug and YSlow. When you go to the YSlow page it says to install Firebug, then install YSlow. This is fine for the alpha geek demographic, but is less than appealing from a corporate perspective. What you want is a single installer.

The folks at Firefox thought of that too! Enter the multi-item installer.

What this lets you do is package multiple .xpi files in a single .xpi file. Which means, in a commercialized Se-IDE perspective, you can ship the version of Se-IDE unmodified as well as your modifications as an extension in a single file. Here is the install.rdf of a multi-item xpi banged out to package Firebug with a couple plugins as a make-sure-it-works experiment.

<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
     xmlns:em="http://www.mozilla.org/2004/em-rdf#">
 
  <Description about="urn:mozilla:install-manifest">
    <!-- nsIUpdateItem type for a Multiple Item Package -->
    <em:type>32</em:type>
 
    <em:id>firebug@goucher.ca</em:id>
    <em:name>Firebug Multi-installer</em:name>
    <em:version>0.0.1</em:version>
    <em:creator>Adam Goucher</em:creator>
    <em:description>Proof is in the pudding</em:description>
 
    <em:targetApplication>
      <Description>
        <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id> <!-- firefox -->
        <em:minVersion>2.0</em:minVersion>
        <em:maxVersion>3.6.*</em:maxVersion>
      </Description>
    </em:targetApplication>
  </Description>
</RDF>

Now, Sauce IDE isn’t quite there yet, but that is the endgame. Right now, it is about 60% there; Se-IDE was forked, but the modifications were made in a plugin-esque manner so they could be separated once it was all working. It’s working now, so the splitting will commence shortly meaning the fork will be retired and an official Se-IDE xpi will be bundled as part of the multi xpi.

Plugins are where a lot of future development of Se-IDE will be focused. Heck, some existing functionality might be extracted into an extension to clean up the code / core. Plugins are only half the story though when it comes to commercial entities adopting Se-IDE as the automation platform they build upon. They need simple distributions to appeal to the mass market. Thankfully, there are ways to solve that problem.

So let’s start getting those commercial distributions of Se-IDE going!

As part of its 100 Best Global Brands series, Business Week had an article on The Great Trust Offensive. Trust is an idea we need to be very aware of, not just from an outward corporate perspective, but from internal corporate and personal career ones as well.

Here are three key quotes:

• In the world of branding, trust is the most perishable of assets
• If you aren’t open with me, then I won’t trust you
• Stay relevant to the current mood or risk been seen as spin merchants

Part of why I think agile-ish things work is that by breaking down the silos and building an environment where communication can happen, then trust has a chance of being achieved.

The article ends with a warning though which resonates with my theory that agile will fail to reap maximum benefits if any part of the organization is actively resisting (intentionally or otherwise).

Trust-related marketing only works if there is a message that people want to believe in. You cannot spin an audience that doesn’t want to be spun.