At a recent customer visit, we were asked how they could track not just who did a deployment and when, but also "why". AnthillPro doesn't know why you do the things you do, so we needed to add an input to the deployment workflow so that the deploying engineer could record why the deployment was taking place.

This short example shows how a workflow property can be used to automatically generate build life notes within AnthillPro.

Read more...

As you can imagine there is a pretty lively debate going on over the Zune date math issue here in the hallways and on our internal mailing lists. There are plenty of places one can find analyses of the bug itself, like here, but I am more interested in the testing implications. <?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" />

One take: this is a small bug, a simple comparator that was ‘greater than’ but should have been ‘greater than or equal to.’ It is a classic off-by-one bug, easily found by code review and easily fixed then forgotten. Moreover, it wasn’t a very important bug because its lifespan was only one day every leap year and it only affected the oldest of our product line. In fact, it wasn’t even our bug; it was in reused code. Testing for such proverbial needles is an endless proposition, blame it on the devs and ask them not to do it again. (Don’t get your knickers in a twist, surely you can detect the sarcasm.)

Another take: this is a big bug, in the startup script for the device and thereby affected every user. Moreover, its effect is nothing short of bricking the device, even if only for a day (as it turns out, music is actually a big deal on that specific day). This is a pri-1, sev-1, run-down-the-halls-screaming-about-it kind of bug.

As a tester can I take any view but the latter? But the bug happened. Now we need to ask what can we learn from this bug?

Clearly, the code review that occurred on this particular snippet is suspect. Every code review I have ever been part of, a check on every single loop termination condition is a top priority, particularly on code that runs at startup. This is important because loop termination bugs are not easily found in testing. They require a “coming together” of inputs, state and environment conditions that are not likely to be pulled out of a hat by a tester or cobbled together using unthinking automation.

This brings me to my first point. We testers don’t do a good job of checking on the quality of code reviews and unit testing where this bug could have been more easily found. If I was still a professor I would give someone a PhD for figuring out how to normalize code review results, unit test cases and system test cases (manual and automated). If we could aggregate these results we could actually focus system testing away from the parts of the system already covered by upstream ‘testing.’ Testers would, for once, be taking credit for work done by devs, as long as we can trust it.

The reason that system testing has so much trouble dealing with this bug is that the tester would have to recognize that the clock was an input (seems obvious to many, but I don’t think it is a given), devise a way to modify the clock (manually or as part of their automation) and then create the conditions of the last day of a year that contained 366 days. I don’t think that’s a natural scenario to gravitate toward even if you are specifically testing date math. I can imagine a tester thinking about February 29, March 1 and the old and new daylight savings days in both Fall and Spring. But what would make you think to distinguish Dec 31, 2008 as any different from Dec 31, 2007? Y2K seems an obvious year to choose and so would 2017, 2035, 2999 and a bunch of others, but 2008?

This brings me to my second point. During the discussions about this bug on various internal forums no less than a dozen people had ideas about testing for date related problems that no one else involved in the discussions had thought of. I was struck by a hallway debate between two colleagues who were discussing how they would have found the bug and what other test cases needed to be run for date math issues. Two wicked smart testers that clearly understood the problem date math posed but had almost orthogonal approaches to testing it!

The problem with arcane testing knowledge (security, y2k, localization all come to mind) is that we share our knowledge by discussing it and explaining to a tester how to do something. “You need to test leap year boundaries” is not an ineffective way of communicating. But it is exactly how we are communicating. What we should be doing is sharing our knowledge by passing test libraries back and forth. I wish the conversation had been: “you need to test leap year boundaries and here’s my library of test cases that do it.” Or, “counting days is a dangerous way to implement date math, when you find your devs using that technique, run these specific test cases to ensure they did it right.”

The testing knowledge it took to completely cover the domain of this specific date math issue was larger than the set of folks discussing it. The discussion, while educational and stimulating, isn’t particularly transportable to the test lab. Test cases (or models/abstractions thereof) are transportable and they are a better way to encapsulate testing knowledge. If we communicated in terms of test cases, we could actually accumulate knowledge and spread it to all corners of the company (we have a lot of apps and devices that do date math) much faster than sitting around explaining the vagaries of counting time. Someone who doesn’t understand the algorithms to count time could still test those algortihms using the test assets of someone else who did understand it.

Test cases, reusable and reloadable, are the basis for accumulated knowledge in software testing. Testing knowledge is simply far too distributed across various experts’ heads for any other sharing mechanism to work.

Are you looking to find out how virtualization (server virtualization, like that offered by VMware, Citrix and Microsoft) can help your software engineering process? Lets look at various stages in the software development process (regardless of the software engineering methodlogy - waterfall/agile/… that you use). By the way, this was

Uses of virtualization spread across the entire ALM cycle

something I wanted to post about and incidentally I also found a recent question about this raised on StackOverFlow.

Do take a look at this list below and let me know use cases that I am missing.

1. Requirements gathering and Definition
   • Use virtualization during your rapid prototyping and proof of concept (POC) development. Rapidly build up machine configurations, deploy your ‘Hello World’ app and test out the POC.
2. Design and Development
   • Easily create environments for intermediate (work in progress) demos to customers and management
   • Provide a consistent environment to all your developers
   • Unit test your code on various target OS systems easily
   • Some development organizations may use VDI to ease the management of developers’ desktops
   • Snapshot and save your developers’ environments - easy back up and recovery in case of machine failures
3. Software Testing
   • Probably the first area in software engineering to leverage virtualization, there are umpteen uses of virtualization in testing. Regression tests, performance testing, testing under varied environment conditions/permutations/combinations, integration testing etc. are all cases where virtualization is useful. You will find this question posed on SQA forums as useful reference.
   • Virtual lab management is a management layer above the base hypervisor that takes the use of virtualization in software labs to the next level. You will find this other post of mine as useful reading in this context — “What virtual lab automation offers over and above a hypervisor“.
     
4. Software Build and Release Management
   • Get virtual machine build farm environments on the fly and on demand. Various virtual lab automation solutions like VMLogix LabManager offer integrations with build tools like IBM Rational Build Forge. Read more about VLA integrations over here. Read more about the Build Forge integration over here.
   • Save your build environments so you can easily revert back to the build configuration of an older version of the software. This is particularly useful during patch releases (I found this use as one of the comments in StackOverFlow).
5. Source and Version Control
   • Maintain your source control systems on a virtual machine. Use the snapshot and save features to capture (backup) your source control systems easily. Revert back to the last known state in case of machine failures etc.
6. Software Maintenance
   • Leverage virtual lab management solutions to support your various customers using the software. Easily replicate customer scenarios and reproduce the defects in the software. Share the environment with the developers. This may be useful related reading about the various users of VLA software.
7. Software Staging and Deployment
   • Products like VMware Stage Manager and VMLogix StageManager help users manage multi-machine configurations in the pre-production staging and system readiness. These management products over the virtualization platforms help users collaborate in a workflow during the staging process and eliminates the risk of service deployment/upgrades to a large extent.
8. Software Trials and Evaluations
   • Virtualization can help by making software trials and evaluations simpler. Software can be packaged as an appliance and users can immediately evaluate the software features rather than going through the challenges during installation, setup, configuration etc. This is a post that talks about virtual lab automation in software trials and evaluation.

- Srihari Palangala

      

It’s been a little while, but my theme hasn’t finished yet. In this case I am revisiting an idea I posted about some time ago. Due to the clear benefits of continuous integration for large teams, sometimes people forget the ways in which it can be useful for small teams. Even teams as small as one.

For small teams, the benefits are less about communication (although that is still an important factor as soon as you have more than one developer), and more about automation. Calling out the points made in my last post, these benefits include:

• Build and release automation: nobody should waste time on manual build and release processes, least of all teams already short on development resources. Setting up a continuous integration process enforces a fully automated build, which naturally extends to automated releases.
• Building across multiple environments: as a small team with a correspondingly small number of development environments, it’s unlikely your code is naturally exercised on all the platforms your support. A continuous integration server that supports distributed builds can make it easy for you to test across platforms.
• Testing frequency: a small team of developers will naturally have a lower overall frequency of testing than a large team, simply because there are less people exercising the code as they work. A continuous build can help close the gap, helping to uncover Heisenbugs that might otherwise escape into the wild.

Perhaps the largest benefit is a shift to the continuous integration mindset of working — coding in short iterations and checking in at least daily. The solo developer, with nobody else waiting for their changes, can easily bash away at the keyboard for weeks between checkins. Sometimes this works fine, but other times you end up losing perspective and sometimes motivation as you keep digging and digging away. Nothing keeps development on track better than taking baby steps and constant assessment of progress, and the very feeling of progress feeds your motivation.

This is what continuous integration is all about — baby steps, constant feedback.

—
Small team looking for a CI solution? I’ve got good news: we offer small team licenses for free, with no obligation whatsoever!

Welcome to the new year!

2009 will be the year I publish a new book on testing and the year I ship my first testing tool since Holodeck oh so many years ago.

I’m thinking about calling the book exploratory testing. Believe it or not the title isn’t taken. Any thoughts on that title? Am I too cheeky by claiming it? By happy coincidence the material I am preparing is on manual testing of a strikingly exploratory nature.

I don’t get to name the tools though as they will be shipping as part of Visual Studio 2010 and are currently code named after various islands in the Puget Sound. My boss (yes, there is some poor unlucky soul here at the empire who is forced to claim me as a direct) is in the process of blogging about those tools our team is building. Allow me to introduce Amit Chatterjee at his corner of MSDN. Please send along your condolences for him having to be the one to write my review. <?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" />

Also, based on reader feedback, it will also be the year that I expose more insider details about testing culture and practice at Microsoft. Those are the most read of my blog posts and I’m going to take that as a hint. Although, much of the thunder has already been aired by my colleagues in their new book How We Test Software at Microsoft. That book's a good read and a high bar for me to match when my own book comes out in a few months.

We recently upgraded the burndown chart in the Tickets tool to include all of the key planning information on one page.  The goal of a burndown chart is to help you hit your release date.  If you run an agile development process with fixed release dates (this is almost the definition of an agile process), then please read on to learn how this simple and powerful feature can help you.

I have advocated doing without schedule estimates and task estimates, based on the following logic.  Maintaining estimates is a lot of work, and we can accelerate the schedule by applying that work to other tasks.  If we can correctly sort our tasks into priority order and work on the highest priority tasks first, then the estimate isn't needed because it doesn't change anything we do.  This logic applies if you run your own business and don't need to negotiate your deliverables with your boss or clients.  If you do need to figure out what to promise those guys, then you probably need estimates, and burndown charts will help you a lot.

Burndown charts can be simple, and they can be complicated.  Ours is the simple kind.  We do not waste any energy with the original estimates, only with the work remaining.

To use the burndown feature, you need to estimate"Work hours remaining" whenever you work on, or edit, a ticket.  This field appears on the ticket edit and comment forms.  You also need to assign tickets to a particular Milestone, or iteration, that you will burn down.

I have attached a picture of a milestone burndown below.

The orange line at the top is the burndown chart.  It shows how many work hours remaining you had estimated on each of the days in this iteration.  The goal of this process is to get to zero on our intended release date, January 10.  This chart shows a typical burndown history. 

December 30: In my management role, I entered some tasks adding up to about 20 hours. 

January 1: I showed the tasks to the other developers, and they added some tasks, and put in their own estimates, so now the total jumped up to about 50 hours.  (We didn't work on New Years, this is just an example.)

It is important that you get estimates from the developers who will do the work.  Project managers always have an incentive to underestimate.  They are optimists and they want to get a lot of work done.  Developers are also notoriously bad at estimating, but at least they have an incentive to make the estimates accurate.  They want a decent life without too much overtime or questions about missed estimates.

January 2 and 3: We did some work on these tasks.  The work "burns down".

January 4: We did some work, but we also found some bugs. The bugs add to the total amount of work, so we don't get much net gain.

You can look a this chart and see that the slope of the line is approximately correct to get to zero on January 10. However, if it stops sloping down at the right pace, then you need to change your plan.  That's why we show the two bottom panels.

Under the burndown chart is a user workload chart.  You might want to re-allocate some of the work if one person is overloaded and that is causing a bottleneck.

Under the user workload chart is a ticket details list.  This shows the biggest tasks at the top of the list.  You can probably break up or move some of those tasks.

 

Going into the subject of Test Project Strategy and more specifically around MTPs there are 2 points where we are not always sure what to do and what are our responsibilities as testers and test managers:

Product Risks & Project Risks


The first step is to understand these are 2 different entities:

Product Risks are areas in the AUT where there is a high risk you will find (important or numerous) defects, usually due to changes or other internal factors.

Project Risks are situations that may or may not happen (risks), if they materialize they usually cause delays in the project's timelines, and the source of these risks may be internal or external.


Product Risks
As testers one of our tasks is to manage Product Risks.

We are paid (at least in part) to be aware of all Product Risks, to make sure the rest of the project team is also aware of them, and to coordinate this information with Project Management to make sure our schedules are taking these risks into account.
In addition, we are expected to plan our testing strategy based on these risk, scheduling more tests (and earlier tests) on areas with higher risks in order to find these issues faster.

My preferred method for handling Product Risks starts by listing them as part of my MTP and reviewing them with all the Project Stakeholders. During these reviews I try to get inputs into the relevance of these risks, as well as information about additional risks I may be unaware off. Once the project starts we review and update all product risks as part of the coordination meetings with the rest of the project team.

Examples of Product Risks are:
- Complex features affecting multiple areas of the existing product, like an upgrade/migration of the system.
- New Technologies used in the product; for example a new DB server, a new programming language, a new integration, etc.
- New Developers or Development Teams, who may lack experience and thus pose a higher risk to the existing product.
- Tight Schedules, that make people work in a rush and commit more mistakes.
etc


Project Risks
The responsibles for Project Risks are the Project Managers, who are also in charge of the project's schedule. The QA, as part of the overall team, is responsible for the Project Risks within our work areas.

Project Risks are usually managed in an Excel Sheet (or Google Docs Spreadsheet) where we list and manage all the risks for the project. For each risk we collect and manage the following information:
1. Risk Name & Description
2. Risk Index (we use this field in order to sort our list) - calculated by multiplying its probability by its consequence
3. Risk owner
4. Date of relevance - when does the risk starts been relevant and prevention actions can start taking place
5. Prevention Actions - how to avoid this risk
6. Contingency Plans - what do we do if the risk materializes

The most usual project risks related to the QA & Testing work are:
- Delays in the delivery of the AUT for testing
- Lack of technical knowledge on specific areas of the product
- Lack of testing environments and/or data that effectively simulate real customer usage
etc.


Risks are a big part of any manager's work. We are expected to be in the lookout for the things that may happen and prepare accordingly.

Most of the task related to Risk Management are not complex but they require good understanding of the project and product as well as the strict discipline required to keep following and managing these risks throughout the whole lifecycle of the project.

Over the past several months (since May, 2008 in particular), I have been blogging about the use and impact of virtualization in software engineering; and virtual lab automation in particular. Thank you all for reading and your interest in this blog. From all at VMLogix, our best wishes to you for a very happy and prosperous 2009! As we enter the new year, I wanted to share with you the some summary information from the blog.

Top posts of 2008

This blog has had ~13,000 views and has 93 published posts to date. The top 5 posts (based on views) on this blog have been:

1. Citrix XenServer, Microsoft Hyper-V and VMware ESX Server Impact on Virtual Lab Automation (1096 views)
2. Management tools available for Hyper-V (568 views)
3. Test environment network configurations made easy with Virtual Lab Automation (448 views)
4. Importance of Linked Clones in Virtual Lab Management (358 views)
5. IBM Rational Quality Manager/RTLM Demos (215 views)

I also spent some time going through the posts and pulling together a set of reading recommendations if you are new to virtual lab automation and the technology. Here is the list, I hope you find it as a useful index into the blog:

Key reading recommendations/posts if you are getting started with Virtual Lab Management (Regardless of your hypervisor choice — across XenServer, VMware or Hyper-V)

About Virtual Lab Automation as a Technology

• Importance of Linked Clones in Virtual Lab Management
• Enterprise Labs - Best Practices with Virtual Lab Automation
• What Virtual Lab Automation offers over and above a Hypervisor
• Test environment network configurations made easy with Virtual Lab Automation
• Network fencing - Yet another key feature from Virtual Lab Automation Products
• Automation in Virtual Labs — What exactly do we mean by it?
• Features and functionality in Virtual Lab Automation solutions
• Seven Tips on Virtual Test Lab Automation
• Getting by with desktop virtualization for your testing needs? Here is what you are missing
• Reigning in and Arresting VM sprawl in your Test Environments

How VLA would fit in your ecosystem (other products, many lab users etc.)

• Test automation tools and Virtual Lab Automation
• Virtual Lab Automation - Who are the many users?
• Virtual Lab Automation Product Integrations
• Test Lab Management - Can it help you make a better decision on when to ship software?
• Virtual lab automation to manage your software assets in the lab
• Creating a Private Cloud to run your Virtual Test Environments
• Self serve IT enabled in the software lab through virtual lab automation
• Software trials and evaluations - How can they be made more effective?

Virtual Lab Management Technology Benefits

• Quantifying the savings on your test cycles from your virtual lab automation deployment
• Control and reduce the IT cost spending in your software test lab
• A Framework to Estimate Cost Savings from Virtual Lab Automation Solutions
• The Benefits of Software Lab Consolidation using Virtual Lab Automation
• Keeping Test Lab Operational Costs Low
• Virtual Lab Automation as a Strategic Purchase in your Software Labs

In 2009, we will continue to maintain this blog focus on the management and use of virtualization in various aspects of software engineering. We do look forward to staying engaged with you.

Once again, best wishes for a great 2009!

- Srihari Palangala

      

In my last post, I had mentioned that I would pick a specific area of testers challenges and share with you how we are addressing it. In this part of the series, I want to address heads on a very key issue – respect for testers.

There are testers from all walks of life and background. Some very technical, some experts on the product they are testing, some are specialists and often they are also developers.  While testing itself is recognized as one of the most critical components of application delivery, it’s worth a wonder whether the great majority of testers do get the respect they deserve. Why is it an accepted fact of life that testing gets crunched when a project is running over time? What would really happen if a testing organization said “We cannot sign off on this release” and put their foot down? If these situations intrigue you, I invite you to wonder with me, why they haven’t changed in the history of software development.

One key aspect of quality assurance (as the word is described often) is the assurance that the software is of a certain quality. Do all testing teams out there feel like they in-fact have an impact on Software Quality and can assure it, or do they just focus on finding bugs? In an interesting informal survey at a very well represented testing conference, a lot of testers wondered if they added value to their organization. I invite you to wonder with me why this could be the case.

Testers work hard, they are smart, and often are the closest a project team can get to the end user before actually releasing their product. But let’s examine the following issue closely.

Tester files a bug

Developer reviews it – marks it as  “Need More Info”

Tester collects a lot of data and adds to bug

Developer marks it as “No Repro”

Tester has spent 20 hrs on this bug already, and developer has marked the bug no-repro (after having spent some chunk of his/her time).

Each time this happens, the developer is frustrated that the testers are not doing enough and the tester is frustrated because he/she is always on the defensive – it’s like the bug is not a bug unless a developer finds it (innocent until proven guilty). How frustrating is that for a tester who just spent a good portion of his week on this bug?

<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /> 

To this date, what has happened is that testers have created a world in which developers have little to no space. They don’t use the same tools the developers use, they don’t see the bug in same way a developer does, and they are not empowered enough to share this data across the teams. Why is this hard to change? The fact is, it is not. Except that, for the longest time we’ve built teams and tools that separate developers and testers. At Microsoft, we are excited to bring these two teams closer together and deliver better interaction. We want to break the silos between dev and test teams and build bridges.

 

In this series of posts, I want to give you specific details on HOW we are doing this. To best describe this, I want to segment the testers in the world into three broad buckets. This will allow me to not only break up the blogs, but also introduce to you the various solutions that we are bringing to market in the near future to address each of the areas.

 

1.       Engineering Support / Developers: These typically are developers who write code for pure customized automation testing or hard core libraries to be used by others. They would likely stay on a development career and do unit testing and deep white box testing

2.       Specialist Testers: These typically are people with engineering backgrounds, are formally trained in QA and continue to write code/scripting for automation, performance, security and other advanced testing.

3.       Generalist Testers: These typically are testers that DO NOT have a computers/engineering back ground, have at some point used the application that is under test and are motivated to make sure the users of the software get the best product they can deliver. They typically DO NOT aspire to be developers or technical coders. They are very savvy when it comes to testing, and deeply understand end users of the application.

Our research shows that 70% of testing done in the Industry is manual testing done by generalist testers, with the other two categories making up the remaining 30%.

While Microsoft is a fairly new and unknown player in the test market, we have shipped our Unit testing, and Load and Performance testing solutions in 2005 as Visual Studio Team System Test Edition (VSTS) and then continued investing in the VSTS 2008 release. These testing solutions are successfully being used by many and we continue to get great feedback from our customers.

For Visual Studio Team System 2010 release, Quality is one of the two pillars for the release (driving Business and IT alignment being the other). We have announced a wave of products that make up the test offering that solve critical customer pain points. The following figure shows how our offerings map across the development and testing teams and especially maps to the different competencies across testers in an organization.

 

Caption: A single snapshot of the Microsoft Test Offerings in Visual Studio 2010

Our test offering is built on top of Team Foundation Server that serves as a single repository allowing different team members access to information they seek via tools, programmatic access, reporting, and other means. With virtualization playing a very key role in development and test environments, Lab Management capabilities are built in a fully integrated way as a core ALM capability that allows developers to focus on coding and testers to focus on quality while eliminating bug ping pong and tedium of setting up complex environments.  

While it is a commonly available toolset, our customers are asking us for integrated solutions that allows test case management to co-exist with all other work items and project assets that govern the success of a project. We have built our test case management framework grounds up to allow different team members a complete view of test cases and all related artifacts.

Microsoft Test Runner allows generalist testers to execute their manual test workload very easily via “Capture & Replay” technology and empowers them to log bugs that are rich in information. Rich bugs allow developers to quickly recreate bugs and avoid the dreaded bug ping pong that is no one’s favorite.

“Coded UI” testing allows you to bring quality earlier into the cycle by making additional tests as part of your build verification tests.

This hopefully gives you a good overview of what we are doing. In future posts I will go into more details of each of the areas. I do hope you will come back to read more specifics about each of these areas of testing.

 

Here’s wishing you, and your families and friend, a Very Happy and Prosperous 2009!

I have occasionally posted on the topic of genetic programming, as a possible way to drive rapid software development.  Some of these posts have drawn requests for more material.  For those who are interested, I posted a summary of my (very old) research and source code here.   The material includes Doodle Garden, a windows application that evolves pictures, the GPQuick C++ framework, sample code for using a Meta-GA to evolve financial trading rules, and a discussion of advanced techniques like distributed clusters, Meta-GA, and symbiosis.

 

Our release this week continues the tradition of great holiday upgrades.  Read on for news about integrating with other systems used by your team, our upgrades to the Tickets tool, Wiki markup improvements, burndown charts, and portfolio management.

Webhook tool

A Webhook is how modern Web applications talk to each other.  Assembla's new Webhook tool will post selected stream events (commits, ticket edits, wiki edits, etc.) to the URL of your choice.  I want see what users do with this highly programmable capability.  It's immediately useful for two main reasons:

• You can use the webhook to trigger a continuous integration build process.  The Webhooks tool replaces the old post-commit hook, with the advantage of serving all of our repository types - svn, git, mercurial, github, and external svn.
  
• Distribute your news to other systems with REST API's.  For example, we provide preconfigured formats to send your events to Basecamp, Present.ly, and Twitter. 

If you are sending events to Twitter, you get an effect that is like the Twitter tool, but you can play with the form of the message, and you can send direct messages.  Follow the instructions on the bottom of the page to compose your messages.

Wiki markup

You can now type links to any wiki page name, tickets, files, revisions, and images.  There is a link guide on the bottom of the wiki edit page.  A popular request at feedback.assembla.com reminded us that people were not satisfied with our CamelCase links to wiki pages.  We implemented the suggestion to use Mediawiki style [[ link stuff ]] links.  The double-bracket delimiter is unique enough to add to all of our formats (WYSIWIG and textile), and the textile format that we use for tickets. 

Burndown chart

We have long featured a "Burndown" link on the Tickets tool, but until now, it wasn't very useful.  This week's enhancements to the burndown chart give you a simple and complete way to find out the things that a burndown chart should tell you:  How much work do you have remaining?  How fast are you getting it done?  How is it allocated to your workers?  What are the big tasks?  To use it, enter a "work hours remaining" estimate whenever you edit a ticket, and follow the instructions on the burndown page.

We also fulfilled a customer request and added work hours remaining as a field in the customer reports. 

I'm going to post a separate article about how to use the burndown chart.

Simplified Ticket report / filter UI

Even I got lost making Ticket reports.  The simplified "Manage filters" and "Search" forms In the Tickets tool solve the problem.  We have renamed the reports "filters", out of deference to our Jira users.

Portfolio Payer management

If you manage a portfolio, you are probably also responsible for the subscriptions in member spaces.  We now show the payer in the list of portfolio projects, and we give you subscription management links when you click through to the member space.

Watcher permissions - step toward the customer permissions

A customer account with restricted permissions is the number 1 and 3 request at feedback.assembla.com.  We are implementing this by upgrading our read-only "Watcher" permission level for team members.  In this release, you can select permissions for the Watcher role - All, Edit, View (default for watcher).  In our next release, you will be able to set different permissions for each tool.

As we come to the end of the calendar year it seems to be shaping up that 2008 will have been a really tough year for a lot of people. So we're doubly thankful that for eValid it has been a fairly good year.

Recall that the Chinese word for "crisis" is composed of two symbols: one for "danger" and the other for "opportunity."

Crisis = "dangerous opportunity" is the way that's usually interpreted. And, we aim to take advantage of the opportunities that the current situation presents!

So this may be a good opportunity for those of us who are in the quality and productivity business. It seems to us that products and technologies that make life easier, get the work done quicker and more efficiently, make better use of resources, and do the job better are going to be extremely important as the world economy gets back on track.

Here is a summary of what you can expect from eValid in the coming days and weeks:

• V9 Release, sometime after the first of the year with many new features and performance improvements. (All V8 support will be ending a few months after V9 is released.)
  
  

• There'll be several special offers for regression suite creation, for package pricing involving server loading work, and for monitoring script development.
  
  

• Realigned product bundle compositions that will see the EPI feature added to both the WebMaster Bundle and to the Regression Test Bundle. Look for other changes in product composition and organization, too.
  
  

• Revisions of some of the pricing, including introduction of SAAS-type pricing and special virtualization support.
  
  

• Enhanced DOM programming support: you'll now be able to do ANYthing you can imagine doing with DOM scripting based on "from life" recording.
  

We wish you all the very happiest of New Years celebrations, and we wish for the very best for everyone in 2009.

As we come to the end of the calendar year it seems to be shaping up that 2008 will have been a really tough year for a lot of people. So we're doubly thankful that for eValid it has been a fairly good year.

Recall that the Chinese word for "crisis" is composed of two symbols: one for "danger" and the other for "opportunity."

Crisis = "dangerous opportunity" is the way that's usually interpreted. And, we aim to take advantage of the opportunities that the current situation presents!

So this may be a good opportunity for those of us who are in the quality and productivity business. It seems to us that products and technologies that make life easier, get the work done quicker and more efficiently, make better use of resources, and do the job better are going to be extremely important as the world economy gets back on track.

Here is a summary of what you can expect from eValid in the coming days and weeks:

• V9 Release, sometime after the first of the year with many new features and performance improvements. (All V8 support will be ending a few months after V9 is released.)
  
  

• There'll be several special offers for regression suite creation, for package pricing involving server loading work, and for monitoring script development.
  
  

• Realigned product bundle compositions that will see the EPI feature added to both the WebMaster Bundle and to the Regression Test Bundle. Look for other changes in product composition and organization, too.
  
  

• Revisions of some of the pricing, including introduction of SAAS-type pricing and special virtualization support.
  
  

• Enhanced DOM programming support: you'll now be able to do ANYthing you can imagine doing with DOM scripting based on "from life" recording.
  

We wish you all the very happiest of New Years celebrations, and we wish for the very best for everyone in 2009.

In the past I posted articles on the importance of differentiating between the severity and priority of a bug and about how to report a bug correctly; but there is one subject that I left out and got questioned about it a couple of days ago:

how to give bugs the correct (and objective) severity?

I know of at least 2 ways to set the priority of each bug correctly, the hard way and the easy way.


The hard way: to define the Severity each time from scratch

Sadly enough most groups work this way, asking their testers and/or bug-reporters to set the severity based on a scale (of either 3 or 5 levels) according to their objective understanding of the harm the bug causes the system or end-user.

What's the problem with this approach?

1. It's not objective, so two people may look at the same bug and set 2 different severities.

2. It usually gathers bugs towards the middle of the scale.
From personal experience if you leave your severity definition to the testers' gut-feeling, the bug distribution at the end of the project will look like a normal curve.
At first you may think this is OK but think again... There is no logical explanation for this behavior, on the contrary for many or most projects you expect to see shifts to the right or left based on factors such as the maturity of the team, application, technology, etc.
My explanation is that we like to do things on a balanced way, so unconsciously we try to balance even the severities of the bugs we report (don't blame your mother, its genetic...).

3. Your severity appreciation can shift with time.
You may have seen this in the past projects, as we advance in our testing we may look at a bug we reported in the past and feel that we gave it a severity that is too high compared to the ones we are reporting today.
This may be related to the old saying: "just when you though things can't get any worst, they usually do".


The easy way: create a Severity Look-up Table

In my opinion the best way to give the correct severity is by having a look-up table to help us classify our bugs. By thinking ahead of time about most bug cases and creating a table we are able to classify bugs more accurately and based on a standard system.

An example of a Severity Look-up Table I've used in the past is the following:

S1 - Critical
A defect that causes the complete system to stop functioning or that will result in unrecoverable data-loss. The bug has no workaround.
Additional specific bugs:
- Spelling or Grammar Mistakes
- Important bugs that were reported from customers and we assured them will be fixed.

S2 - High
The defect causes a part of the system to be inaccessible or to stop functioning. The bug has no workaround or it has a workaround that will not be easily found by users.
Additional specific bugs:
- High visibility GUI issues.

S3 - Medium
The defect causes a non-critical failure on the system but it will allow users to continue working. The bug has a workaround that is relatively easy to find and will be acceptable by most users.

S4 - Low
The defect causes no dysfunctions to the system and may even be unnoticed by most users . The bug has an easy workaround or may not even require a workaround at all.

S5 - Enhancement Request or Suggestion
No need to explain.

Notice that on some levels I added specific cases that even if they don't fit the definition of the severity will still make it to that level. This is a practice that will help you include your specific issue themes into the Severity Look-up table.

Each Organization / Team / Project may need to review or fine-tune their Severity Look-up Table once in a while based on their individual definitions, requirements and customer needs.


I remember once I told a group I was training about this practice. A young tester got up and said that she thought using a Look-up Table would make her look inexperienced and unprofessional.
My reply was that I still use lists like this in almost every project I work, and that it would be less professional to use another approach if she knew that this one was easier and gave better results...

Graeme Rocher has never been a fan of Maven, and (as far as I can tell) he still isn’t.     In “Grails & Maven Kiss and Make-up with Grails 1.1 Beta 2“, Graeme writes:

So Grails 1.1 Beta 2 is out. Rejoice! There are many new features that are detailed in the release notes. However, one of the main ones in this beta is the new support for Maven….Regular readers of my blog will probably be aware of my long history as one who, ahem, is not particularily fond of Maven. Granted I am still not [particularly] fond of Maven, but it is the Christmas period and in the spirit of “why can’t we all just get [along]” I am proud to say that Grails integrates nicely with Maven now

Merry Christmas, indeed.

In the same spirit of “why can’t we all just get along”, I’ve had a similar shift in my opinion of Groovy over the years, I used to talk about Groovy as the wrong choice when compared to something like Jython or JRuby, but after having used Groovy in a number of projects, I ready to admit defeat.  While I’m still interested in Java inoperability with Ruby and Python, it is often much more straightforward to use Groovy than it is to deal with the “impedance mismatch” between something like JRuby and Java.  Groovy has turned into a mature and valuable language that offers tight integration with the JVM thanks to the continued efforts of Graeme and Guillame at G2One (recently acquired by SpringSource).

GMaven provides a great option for people interested in writing Maven Plugins in Groovy and also as a quick way to extend a Maven build with a Groovy script.

As the clock winds down on 2008 and before we turn our eyes toward ‘09, we wanted to take a moment to wish everyone in the greater uTest community — testers, customers, investors, partners and employees — a safe and happy holiday season.

Thank you for contributing to an exciting and successful ‘08.  Next year will be one of great opportunity, challenge and growth for uTest.  And with your help, we’ll someday look back on 2009 as a breakthrough year for all of us, as well as the beginning of mainstream adoption for community-driven software testing.

Happy holidays from Team uTest.

addthis_url = 'http%3A%2F%2Fblog.utest.com%2F%3Fp%3D91'; addthis_title = 'Happy+Holidays+and+Sincere+Thanks'; addthis_pub = '';

Yesterday, in Central Maven Repository Traffic: Investigation and Analysis, I wrote about the analysis involved in tracking down the increasing load on Central. By identifiying some misbehaving tools, we were able to reduce the traffic from a 98 Mbps average down to 60-80 Mbps.  In this post, I discuss the next step toward a Central Maven repository that can scale to meet the load generated by the millions of developers using an ecosystem of tools which rely on the Central Maven Repository.

Where We Left Off…

As a refresher, here’s what the load looked like at the end.   To summarize, Central was experience load problems on httpd, we subsequently moved to Nginx which fixed the load problem but caused us to regularly saturate a 100 Mbps pipe.   After a few weeks of investigation, we discovered that much of the traffic was due to a misconfigured product that was repeatedly downloading the Nexus index.   We notified the responsible project and blocked access to the index until the problem was fixed.   The end product of all of that work was a Central Maven Repository that had fewer saturation events and which was operating in a consistent 60-80 Mbps range in the middle of a work week.

We still experienced significant amounts of traffic on Monday morning as users fired up M2Eclipse and downloaded the updated weekly index. We tried shifting the index creation from Sunday night to Friday afternoon, to help smooth out some traffic over the weekend, but ultimately this didn’t help much. We ended up having to QOS the transfers of the zip down to something like 30 kbps to keep the rest of the repository available during these loads. This wasn’t a great solution as it significantly increased the time for users to get at the index.

Considering Amazon S3

One of the ideas we had been persuing was to use Amazon S3 to host central. For those of you unfamiliar with S3, it is a Cloud based system for storing and serving data and it is a part of the Amazon Web Services products (EC2, S3, Cloudfront).  The nearly unlimited bandwidth and option to use Cloudfront to bring the data physically closer to the users is a definite bonus.   It was clear to us that we need something that could scale indefinitely as we continue to see increased adoption of tools that rely on the Central Repository.  The drawback with S3, is that we don’t have a direct ability to monitor the traffic and uncover any abuse or misconfigured tools like we do now.   We all agreed that moving to something like S3 was the future.

Despite our frequent pleas to use Repository Managers and not scrape the repo, we continue to find people doing just that nearly every day. The bandwidth on S3 is not free and opening it up without the ability to protect the bottom line from abusers is not a great idea.    Once you start designing systems on an Internet Scale you start to realize that bandwidth isn’t free.

Moving to “The Cloud”: Amazon S3

We decided first to take baby steps and see if S3 could help us with a very specific problem: The index downloads.   Instead of downloading multiple GB of repository data and creating an index locally, we think it makes more sense for people and tools to download a repository index once a week.   This index weighs in at 30 MB, and while that might not seem to be very large at first glance… multiply 30 MB by 50,000 downloads, and you’ll quickly start to serve TB of data.  This is exactly what was happening, and it seemed like an easy target to offload to Amazon S3.

We found a handy Ruby script called S3Sync that we use to synchronize the maven2/.index folder over to a “repo1.maven.org” bucket on S3. Nginx is then configured to send temporary redirects (302) for all /.index/ requests over to S3. By doing this, we are still able to manage traffic to the Index, yet offload the bulk of the data transfer to the S3 network.

So how did it work? Unbelievably well. Take a look at the week prior to the shift to S3 and week after the shift to S3:

Take a look at the Weekly traffic before the switch to S3, realizing that the weekly traffic numbers mask the periodic 100 Mbps saturation that we were seeing in the hourly graphs. A new index was published, and we saw a spike of download activity during the morning of December 1st.   After that our weekly traffic gradually diminishes to a background level of between 40 Mbps and 80 Mbps on an average weekday.     Again, note that even those good days had periods of complete saturation, after blocking the offending tools we saw an improvment, but we wanted to offload the bulk of our index downloads to S3 to gain further improvements.

Now, take a look at the weekly traffic graph after moving the index to S3.   Unless you note the difference in Y Axis scale, this might not seem as impressive.    The number we’re most interested in decreasing is the 95th percentile value, it describes the 5-minute average bandwidth which 95% of our traffic falls within.      If our 95th percentile number is very close to 100 MBps it means that we’re likely to saturate the 100 MBps quite often.   If our 95th percentile is down around 20 MBps, we’re much more likely to have a stable and available repository.  After moving the index to S3, we have a 5x decrease in the 95th percentile from 98.7 MBps to 12.3 MBps.   We went from a weekly bandwith average of 49.28 MBps to 8.24 MBps, and our total transfer for the week went from 3.81 TB to 629.7 GB.

Before moving the index to S3, we were suffering from slow response times and an index download which would take a few minutes during peak traffic.   After we moved the index to S3, response times improved by 300%, and the index download now takes a few seconds to complete.   Just moving the index to S3 yielded dramatic improvements for the Central Maven Repository.

The Result: Greater Speed, Higher Availability

In the 2 weeks since we moved the index to S3 it has served over 4TB of data and 730,000 requests! Even though the bandwidth bill for the S3 service is significant, we estimate it to be half of what we save on the Central connection traffic.    In other words, we’ve increase availability and reduced the overhead costs associated with the Central Maven Repository.

A note of caution: To protect the system from abuse, we may have to change the urls on S3 from time to time, so don’t point directly at the S3 url, continue to request the data from Central and you’ll be ok.

We won’t stop here in finding ways to optimize the repository experience for users. One thing that will be rolled out shortly is Incremental Index support. This will enable tools that use the Nexus Indexer API to grab only chunks of the index that have changed. This should have a significant impact again on the amount of traffic. We also continue to investigate the possibility of leveraging the cloud to host the entire repository so stay tuned.

Note: If you liked the screenshots used in these past two entries, check out Jing Project. Jing is a great free and easy to use tool for capturing screen images or videos and marking them up. Both OSX and Windows versions are available.

Some good feedback from Henri Gomez From the nexus-user mailing list on December 23rd:

We [had] a major crash on Archiva 1.1.3 after our Linux SLES VMWARE partition has been upgraded.  We couldn’t get it to work with Sun JDK 1.5 / 1.6 (seems to be a major problem with Linux/Sun JDK/VMWare) and Archiva didn’t support IBM JDK. I had to switch to Nexus 1.2.0.2 and after 4 hours of works, it just works great. Nexus WAR 1.2.0.2, Tomcat 6.0.18 and IBM JDK 6.0 and everything works now fluently. A big thanks to all Nexus developers and thanks also to made it available as a simple webapp.

While we think it takes less than 4 hours to install Nexus, Henri was probably taking his time to preserve data from his old Archiva installation.   If you are interested in Sonatype Nexus, you can download the 1.2.0.2 release and get started today.

Every once in a while something or someone comes along and fundamentally redefines a game or an outlook. Humanity believed the earth was the center of the Universe until Galileo came along to suggest and prove otherwise. In the game of cricket, Sachin Tendulkar came along and batting has not been the same since. So what’s the game changing event happening in the software development industry?

There are many key new trends occurring in the software industry, from Virtualization to Software + Services and more. I would like to focus your attention on an area of the software development lifecycle that has stagnated for a very long time - testing - and we at Microsoft are looking to change the rules of the game!

Testing is a critical part of the software development lifecycle that has long since been dominated by practices and tools that segregate the testing teams from the development teams. In a world where testers and developers don’t collaborate effectively, software projects, end users, and ultimately the businesses are the real losers.

We have been learning from our customers and partners as to why such an important area (viz..testing) of software development lifecycle, is falling short on providing developers what’s needed to make software more reliable. We have listened to our customers and have recently announced a series of capabilities as part of the VSTS 2010 product line that address this critical gap.

Over the past couple of years, we spent a lot of time with testers, trying to understand their pain points, understanding how their daily life unfolds with challenges not only interacting with development but also feeling helpless, and at the same time getting no respect. 

We have been working hard at Microsoft to bring a set of integrated technologies that we hope will change the rules of the game. We hope it will not only help address the pain points of these testers but fundamentally change the way they feel about testing – by enabling them to be respected as equals in the SDLC, by creating a tighter interaction between developers and testers, and by allowing them to embrace the changes that occur in software development projects.

I will blog about this change and the way we have been thinking about it and developing tools for it in a series called “Rethinking Testing”. This is my introductory post for the series. As part of the series, I plan to pick a specific area of testers challenges and share with you how we are addressing it. I hope you will follow this series to understand and visualize the complete picture of our solution that we believe will fundamentally change the way software teams have been thinking about testing & software quality.

I also take this opportunity to wish all reader and their familes and friends Merry Christmas, Happy Hanukkah, and a Very Happy New Year! Enjoy the holidays and talk to you soon …

<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /> 

Amit Chatterjee