Skip to content

Open Source

Do you care about your code? Track code coverage on new code, right now !

Sonar - Thu, 11/27/2014 - 06:40

A few weeks ago, I had a passionate debate with my old friend Nicolas Frankel about the usefulness of the code coverage metric. We started on Twitter and then Nicolas wrote a blog entry stating “Your code coverage metric is not meaningful” and so useless. Not only am I thinking exactly the opposite, but I would even say that not tracking the code coverage on new code is almost insane nowadays.

For what I know, I haven’t found anything in the code

But before talking about the importance of tracking code coverage on new code and the relating paradigm shift, let’s start by mentioning something which is probably one of the root causes of the misalignment with Nicolas: static and dynamic analysis tools will never, ever manage to say “your code is clean, well-designed, bug free and highly maintainable”. Static and dynamic analysis tools are only able to say “For what I know, I haven’t found anything wrong in the code.”. And so by extension this is also true for any metric/technique used to understand/analyse the source code. A high level of coverage is not a guarantee for the quality of the product, but a low level is a clear indication of insufficient testing.

For Nicolas, tracking the code coverage is useless because in some cases, unit tests leading to increase the code coverage can be crappy. For instance, unit tests might not contain any assertions, or unit tests might cover all branches but not all possible inputs. To fix those limitations, Nicolas says that the only solution is to do some mutation testing while computing code coverage (see for instance pitest.org for Java) to make sure that unit tests are robust. Ok, but if you really want to touch the Grail, is it enough? Absolutely not! You can have a code coverage of 100% and some very robust but… fully unmaintainable unit tests. Mutation testing doesn’t provide any way, for instance to know how “unit” your unit tests are, or if there is lot of redundancy between your unit tests.

To sum-up, when you care about the maintainability, reliability and security of your application, you can/should invest some time and effort to reach some higher maturity levels. But if you wait to find the ultimate solution to start, that will never happen. Moreover maturity levels should be reached progressively:

  • It doesn’t make any sense to care about code coverage if there isn’t a continuous integration environment
  • It doesn’t make any sense to care about mutation testing if only 5% of the source code is covered by unit tests
  • … etc.

And here I don’t even mention the extra effort involved in the execution of mutation testing and the analysis of the results. But don’t miss my point: mutation testing is a great technic and I encourage you to give a try to http://pitest.org/ and to the SonarQube Pitest plugin done by Alexandre Victoor. I’m just saying that as a starting point, mutation testing is already a too advanced technic.

Developers want to learn

There is a second root cause of misalignment with Nicolas: should we trust that developers have a will to progress? If the answer is NO, we might spend a whole life fighting with them and always making their lives more difficult. Obviously, you’ll always find some reluctant developers, doing some push back and not caring at all about the quality and reliability of the source code. But I prefer targeting the vast majority of developers eager to learn and to progress. For that majority of developers, the goal is to always make life more fun instead of making it harder. So, how do you infect your “learning” developers with the desire to unit test?

When you start the development of an application from scratch unit testing might be quite easy. But when you’re maintaining an application with 100,000 lines of code and only 5% is covered by unit tests, you could quickly feel depressed. And obviously most of us are dealing with legacy code. When you’re starting out so far behind, it can require years to reach a total unit test coverage of 90%. So for those first few years, how are you going to reinforce the practice? How are you going to make sure that in a team of 15 developers, all developers are going to play the same game?

At SonarSource we failed during many years

Indeed, we were stuck with a code coverage of 60% on the platform and were not able to progress. Thankfully, David Gageot joined the team at that time, and things were pretty simple for him: any new piece of code should have a coverage of at least 100% :-). That’s it, and that’s what he did. From there we decided to set-up a quality gate with a very simple and powerful criteria: when we release a new version of any product at SonarSource, the code coverage on new or updated code can’t be less than 80%. When this is the case, the request for release is rejected. That’s it, that’s what we did, and we finally started to fly. One year and an half later, the code coverage on the SonarQube platform is 82% and 84% on the overall SonarSource products (400,000 lines of code and 20,000 unit tests).

Code coverage on new/changed code is a game changer

And it’s pretty simple to understand why:

  • Whatever your application is, and may it be a legacy one or not, the quality gate is always the same and doesn’t evolve over time: just make the coverage on your new/changed lines of code greater than X%
  • There’s no longer a need to look at the global code coverage and legacy Technical Debt. Just forget it and stop feeling depressed!
  • As each year X% of your overall code evolves (at Google for example, each year 50% of the code evolves), having coverage on changed code means that even without paying attention to the overall code coverage, it will increase quickly just “as a side effect”.
  • If one part of the application is not covered at all by unit tests but has not evolved during the past 3 years, why should you invest the effort to increase the maintainability of this piece of code? It doesn’t make sense. With this approach, you’ll start taking care of it if and only if one day some functional changes need to be done. In other words, the cost to bootstrap this process is low. There’s no need to stop the line and make the entire team work for X months just to reimburse the old Technical Debt.
  • New developers don’t have any choice other than playing the game from day 1 because if they start injecting some uncovered piece of code, the feedback loop is just a matter of hours, and anyway their new code will never go into production.

This new approach to deal with the Technical Debt is part of this paradigm shift explained in our “Continuous Inspection” white paper. Another blog entry will follow explaining how to easily track any kind of Technical Debt with such approach, not just debt related to the lack of code coverage. And thanks Nicolas Frankel for keeping feeding this open debate.

Categories: Open Source

What about Microsoft Component Extensions for C++?

Sonar - Wed, 11/19/2014 - 08:32

After my previous blog entry about the support of Objective-C, you could get the impression that we’re fully focused on Unix-like platforms and have completely forgotten about Windows. But that would be a wrong impression – with version 3.2 of the C / C++ / Objective-C plugin released in November, 2014, support for the Microsoft Component Extensions for Runtime Platforms arrived in answer to customer needs. The C-Family development team closely follows discussions in the mailing list for customer support, so don’t hesitate to speak about your needs and problems.

So what does “support of Microsoft Component Extensions for Runtime Platform” mean? It means that the plugin is now able to analyze two more C++ dialects: C++/CLI and C++/CX. C++/CLI extends the ISO C++ standard, allowing programming for a managed execution environment on the .NET platform (Common Language Runtime). C++/CX borrows syntax from C++/CLI, but targets the Windows Runtime (WinRT) and native code instead, allowing programming of Windows Store apps and components that compile to native code. Also could be noted there is not much static analyzers capable to analyze those dialects.

So now the full list of supported C++ dialects looks quite impressive – you can see it in the configuration page:

And this is doesn’t even count the C and Objective-C languages!

You also may notice from the screenshot above, that now there is clear separation between the ISO standards, the usual Microsoft extensions for C/C++ (which historically come from Microsoft Visual Studio compiler), and GNU extensions (which historically come from GCC compiler). The primary reason for the separation is that some of these extensions conflict with each other, as an example – the Microsoft-specific “__uptr” modifier is used as an identifier in the GNU C Library. To ease configuration, the plugin option names closely resemble the configuration options of GCC, Clang and many other compilers.

But wait, you actually don’t need to specify the configuration manually, because you can use the build-wrapper for Microsoft Visual Studio projects just like you can with non-Visual Studio projects. Just download “build-wrapper” and use it as a prefix to the build command for your Microsoft Visual Studio project. As an example:

build-wrapper --out-dir [output directory] msbuild /t:rebuild

and just add a single property to configuration of analysis:

sonar.cfamily.build-wrapper-output=[output directory]

The build wrapper will eavesdrop on the build to gather configuration data, and during analysis the plugin will use the collected configuration without the headaches of manual intervention. Moreover, this works perfectly for projects that have mixed subcomponents written with different dialects.

So all this means that from now you can easily add projects written using C++/CLI and C++/CX into your portfolio of projects regularly analysed by SonarQube.

Of course, it’s important that the growth of supported dialects is balanced with other improvements, and that’s certainly the case in this version: we made several improvements, added few rules and fixed 28 bugs. And we’re planning to go even further in the next version. Of course, as usual there will be new rules, and improvements, but we’ll also be adding a major new feature which will make analysis vastly more powerful, so stay tuned.

In the meantime, the improvements in version 3.2 are compatible with all SonarQube versions from 3.7.4 forward, and they’re worth adopting today.

Categories: Open Source

SonarQube 4.5 in Screenshots

Sonar - Tue, 11/11/2014 - 18:18

The team is proud to announce the release of SonarQube 4.5, which includes many new features:

  • Computation of the SQALE Rating and the Technical Debt Ratio
  • Improvement to the Rules pages
  • Redesign of the Treemap

Computation of the SQALE Rating and Technical Debt ratio

With this version of the SonarQube platform, the SQALE Rating (letter grade) and Technical Debt Ratio move from the SQALE plugin into core. Now there’s an easy index for project comparison, and an easy way to see it too, the new “Technical Debt Synopsis” widget:

Improvement the the Rules pages

This version of the platform brings several improvements to the Rules space.

The first is the addition of a new Active Severity feature, which lets you search for rules by the severities they have in a given profile (rather than by default severity):

This version also makes the rule’s SQALE/Technical Debt remediation function visible. I.e. you can see now how much violating a rule will “cost” you in terms of technical debt. Just click on the SQALE characteristic to see it:

The ability to create new manual rules has also been added to the Rules space. Since markdown is now supported in rule descriptions, you can use it to add some formatting to manual rules:

Once created, you’ll find them in the “Manual Rules” repository:

Redesign of the Treemap

The treemap, one of the earliest visualizations of code in SonarQube got a complete overhaul in this version. Rewritten in JavaScript, you should find it more responsive, more intuitive, and just as beautiful as ever. Among the changes are a better mouseover, and a clickable breadcrumb trail at the bottom for zooming out:

That’s all, Folks!

Time now to download the new version and try it out. But don’t forget to read the installation or upgrade guide.

Categories: Open Source

SonarQube supports ECMAScript 6

Sonar - Tue, 10/28/2014 - 13:53

The 2.1 version of the JavaScript Plugin fully supports ECMAScript 6 (ES6). But what does that mean exactly ?

What is ECMAScript 6 ?

Let’s start from the beginning. The JavaScript language is one implementation of the ECMAScript language standard. (JScript and ActionScript are two more.) Each browser has its own JavaScript interpreter, and most modern browsers support the ECMAScript 5 (ES5) standard. That means that you can write JavaScript that adheres to the ES5 standard and have it work correctly for the vast majority of your users.

ES6 is the next version of the standard, and it provides great new features to allow you to write shareable, efficient, and readable code more easily. Luke Hobbans is a member of TC39, the committee behind ECMAScript. He’s written up a great summary, of the main features of ES6, including a short description of each, complete with code snippet.
Here’s a quick sample for 2 new constructs, variable and constant declaration, and arrow function:

Block scoped binding construct: let for variable declaration and const for constant declaration.

const a = 1;
a = 2;  // error - cannot be re-assigned

if (true) {
  let b = 1;
}
print(b);  // b is undefined

Arrow function: function shorthand using the => syntax.

let sum = (a, b) => a + b;
Can I use ES6 today ?

A new version of the standard also means that each browser needs to provide support for it, at least the major ones. It might take years before it happens, but you don’t have to wait for that to take advantage of the innovations in ECMAScript 6!
Thanks to the availability of ES6-to-ES5 transpilers, it is possible to use ES6 features today. A transpiler will translate your ECMAScript 6 code to ECMAScript 5 code so it can be run by today’s browsers. I like the Traceur compiler, which offers an online demo. You can enter ES6 on the left side and see the resulting ES5 code on right.
AngularJS has already taken advantage of a transpiler to make the move with AngularJS 2.0.

You can follow the progress of ES6 support in browsers and for transpilers in Juriy Zaytsev’s ES6 compatibility matrix.

Use SonarQube to analyze ES6 code

The SonarQube JavaScript Plugin 2.1 fully supports ES6. What does that mean?

It means that the plugin is able to:

  1. parse ES6 source code,
  2. compute all relevant metrics accordingly:
    • a classes count is computed when classes are used
    • class complexity is computed when classes are used
    • the function count includes generator functions
    • general complexity metrics take generators into account
    • the number of accessors is now computed
  3. analyse code against rules, all existing coding rules have been updated to cover the new features, e.g: “unused variable” will detect unused variables & constants declared with let and const.

In upcoming versions, we’ll be adding new rules specific to ES6 features. We’re still figuring out what those should be, but a set of rules about classes is likely. If you’ve got suggestions, please let us know at user@sonar.codehaus.com.

Wanna see a live demo? Check out Angular DI Framework using ES6 on nemo: Angular Dependency Injection v2.

Categories: Open Source

Mobile App for Jenkins User Conference Bay Area

Jenkins User Conference in Bay Area is this Thursday, and one of the new things this year is the mobile app.

There's an Android version as well as an iPhone version. I've installed it locally, and it's very handy for checking the agenda, get more info about speakers and sponsors.

Categories: Open Source

FreeBSD project use of Jenkins for OS testing

This is a guest post by Craig Rodrigues

The FreeBSD project produces a modern operating system derived from BSD Unix.

In the past 6 months, we have set up Jenkins at http://jenkins.freebsd.org/, to continuously build FreeBSD as developers add new code to the project. This has helped us identify and fix build breaks very quickly.

We have gone even farther by integrating Jenkins, Kyua, and Bhyve. Kyua is a testing framework for infrastructure software. Bhyve is the native hypervisor that comes with FreeBSD (similar to KVM on Linux).

We use the Build Flow plugin in this example Build flow to do the following:

  1. Build the FreeBSD kernel and userland on amd64 whenever someone checks in new code to http://svn.freebsd.org
  2. Create a bootable FreeBSD disk image with makefs
  3. Boot the image under bhyve
  4. Run these commands inside the bhyve VM:

cd /usr/tests; kyua test; kyua report-junit --output=test-output.xml

  1. Shut down the bhyve VM
  2. Imports test-output.xml into Jenkins.
  3. Produces a full native test report in Jenkins

The results of this work were presented at the Bay Area FreeBSD Users Group in this presentation in October 2014.

Jenkins has been very easy to set up and use under FreeBSD. We hope that by using Jenkins to run OS-level unit tests, we will be able to improve the quality of FreeBSD. For further information, please feel free to contact us at freebsd-testing@FreeBSD.org .

Categories: Open Source

CVE-2014-3566 "poodle" impact on Jenkins

Another day, another SSL vulnerability! Google has announced a vulnerability in SSL v3, and if you are using the "Winstone" servlet container built into Jenkins, and if you are using the HTTPS connector with the --httpsPort option (it is off by default), then you are vulnerable to this problem.

I've just issued a security advisory on this. If you haven't already subscribed to the Jenkins security advisory mailing list, this is a great opportunity to do so.

The advisory includes the target delivery vehicles for the fix and how you can address the problem in the mean time. Inside corporate intranet, where Jenkins is typically used, I suppose there's a degree of trust among participants to make this less of a problem. But if you run an internet facing Jenkins, be sure to deploy the fix.

(And as I write this, I've fixed all the https://*.jenkins-ci.org servers to disable SSLv3, so we are covered there)

Categories: Open Source

Suggest a Valuable Rule, Win a SonarQubeT-Shirt

Sonar - Wed, 10/08/2014 - 11:58

Is there a rule you’d like to turn on in SonarQube, but you just can’t find it? Well, wish no more, just tweet your missing rule and if its valuable, we’ll implement it.

With coverage of more than 15 languages, developing our own source code analyzers to deliver the most valuable coding rules and bug detection mechanisms is a key mission at SonarSource. In the past, we’ve mainly worked to offer better covererage of rules offered by other rule engines like JSLint, PMD, Toad, CodeSniffer, PHPPMD, CPPCheck, and so on. But the time has come to fly, and now we’d like to know what rules you’re dreaming about.

How to participate

To participate, just tweet the title of your rule and a link to a short description using the tags #1rule1tshirt. If we like it:

  • We’ll add this rule to our Rule Repository, the well from which all new SonarSource rules are drawn.
  • We’ll send you a SonarSource t-shirt.
  • And obviously we’ll do our best to implement this rule and make it available quickly.

That’s it !

Really want few additional details ?
  • The rule can target any language currently covered by SonarSource: Java, C/C++, JavaScript, C#, COBOL, Objective-C, Python, PL/SQL, PHP, ABAP, Android, Flex, Groovy, HTML, PL/I, RPG, VB.Net, XML.
  • You can use whatever’s convenient to host the rule description, such as a Google doc to provide a short description of the rule.
  • The goal of the rule can be either to reinforce a coding practice or to detect some bugs.

Whether or not a rule is valuable is subjective. Like any benevolent dictator, ours will be the final word. :)

Let the tweeting begin!

Categories: Open Source

Gradle-fy your Jenkins Plugin Project

(This is a guest post from Daniel Spilker)

Jenkins supports building plugins using Gradle for a while now. Last week a new version of the Gradle JPI plugin has been released to iron out some issues.

The Gradle JPI plugin enables a 100% groovy plugin development environment with Groovy as primary programming language, Spock for writing tests and Gradle as build system. Have a look at the Job DSL plugin for an example.

An existing Maven build can be converted to Gradle by using the build.gradle template from the Gradle JPI plugin's README. For instance, the POM from the Gradle plugin translates to this build.gradle file:

buildscript {
    repositories {
        mavenCentral()
        maven {
            url 'http://repo.jenkins-ci.org/releases/'
        }
    }
    dependencies {
        classpath 'org.jenkins-ci.tools:gradle-jpi-plugin:0.6.0'
    }
}

apply plugin: 'jpi'

group = 'org.jenkins-ci.plugins'
version = '1.25-SNAPSHOT'

jenkinsPlugin {
    coreVersion = '1.480'
    displayName = 'Jenkins Gradle plugin'
    url = 'https://wiki.jenkins-ci.org/display/JENKINS/Gradle+Plugin'
    gitHubUrl = 'https://github.com/jenkinsci/gradle-plugin'

    developers {
        developer {
            id 'gbois'
            name 'Gregory Boissinot'
            timezone '+1'
        }
    }                           
}

dependencies {
    compile 'org.jenkins-ci.lib:dry-run-lib:0.1'
}

Usage of the Gradle JPI plugin is similar to working with the Maven HPI plugin. Use gradle jpi to build the plugin file. gradle check runs the tests, gradle install copies the plugin into the local Maven repository, gradle uploadArchives deploys the plugin to the Jenkins Maven repository and gradle server starts a Jenkins development server with the plugin installed.

It is recommended to use Gradle 1.8 because that is the version used to build and test the Gradle JPI plugin.

For the next release it is planned to do some maintenance like fixing code style issues and adding tests. After that more issues need to be addressed to bring the plugin on par with the Maven HPI plugin, most notably fixing the test dependencies (JENKINS-17129) and publishing the plugin's JAR (JENKINS-25007). Updating Gradle to 2.x and getting the plugin on the Gradle plugin portal is also on the wishlist.

Categories: Open Source

CVE-2014-6271 impact on Jenkins

I suspect many of you have been impacted by CVE-2014-6271 (aka "shellshock" bash vulnerability.) We had our share of updates to do for various *.jenkins-ci.org servers.

Java application servers in general (including one that ships in Jenkins) do not fork off processes like Apache does to serve requests, so the kind of CGI attacks you see on Apache does not apply. We are currently unaware of any vulnerabilities in Jenkins related to CVE-2014-6271, and no plan to issue a patch for that.

That said, we did come up with one possible way attackers can exploit vulnerable bash through Jenkins, that you might want to be aware of.

When a build is parameterized, parameters are passed to the processes Jenkins launch as environment variables. So if you have a shell step (which uses bash by default), and if Eve only has a BUILD permission but not CONFIGURE permission, then Eve can exploit this vulnerability by carefully crafting parameter values, and have the bash runs arbitrary processes on the slave that run the build.

In most such scenarios, Eve would have to be an authenticated user on Jenkins. Jenkins also leaves the record of who triggered what build with what parameters, so there's an audit trail. But if your Jenkins fits this description, hopefully this serves as one more reason to update your bash.

Finally, to get notified of future security advisories from Jenkins, see this Wiki page.

Categories: Open Source

Jenkins in JavaOne 2014

There'll be several talks that touch Jenkins. The first is from me and Jesse called Next Step in Automation: Elastic Build Environment [CON3387] Monday 12:30pm.

Then later Tuesday, there's Building a Continuous Delivery Pipeline with Gradle and Jenkins [CON11237] from Benjamin Muschko of Gradleware.

Thursday has several Jenkins talks. One is The Deploy Factory: Open Source Tools for Java Deployment [CON1880] from Bruno Souza (aka the Java Man from Brazil) and Edson Yanaga. In this same time slot, guys from eBay are doing Platform Upgrades as a Service [CON5685], which discusses how they rely on automation to make platform upgrades painless. Then Mastering Continuous Delivery and DevOps [CON1844] from Michael Huttermann.

In the exhibit area, the Jenkins project doesn't have its own booth (JavaOne is too expensive for that), but I'll be at the CloudBees booth, so is Jesse Glick. Find us at the booth for any Jenkins questions or impromptu hacking session, which would really help us as we get distracted from the booth duties that way. Or just drop by to get stickers, pin badges, and other handouts to take for your colleagues.

And finally, Script Bowl 2014: The Battle Rages On [CON2939] gets an honorable mention because our own Tyler Croy is representing JRuby against other scripting languages, including my favorite Groovy. Hmm, who should I root for...

Categories: Open Source

More Jenkins-related continuous delivery events in Chicago, Washington DC, and San Francisco

The usual suspects, such as CloudBees, XebiaLabs, SOASTA, PuppetLabs, et al are doing a Jenkins-themed continuous delivery event series called "cdSummit." The event is free, has a nice mix of user/vendor talks, and has an appeal to managers and team leads who are working on and struggling with continuous delivery and automation.

I've spoken in the past events, and I enjoyed the high-level pitches from various speakers. The last two events at Paris and London filled up completely, so I suspect others have liked them, too.

If you live near Chicago, Washington DC, or San Francisco, check out the date and see if you can make it. RSVP is from here. If you do, be sure to pick up Jenkins stickers and pin badges!

Categories: Open Source

Jenkins Workflow Summit RSVP

As was discussed some time ago, the workflow summit is being organized, and it's open for RSVP.

Due to the overwhelming demand, I've increased the capacity this time to 50, but this is an unconference where everyone needs to participate, which means we really cannot have too many people without changing the dynamics of the event.

So please make sure you are willing to participate, as in not just listening and watching, but actually willing to speak. We expect you to bring something to the table — opinions, experiences, rants, presentations, feedbacks, etc. If you don't please let others take the seat, and rest assured we will give a presentation about workflow in JUC Bay Area.

If you understand the criteria, please RSVP is from here.

Categories: Open Source

Jenkins User Meet-up in Paris

My apologies for the last minute announcement, but there will be a Jenkins user meet-up in Paris on Sep 10th 7:00pm, which is just next week. The event is hosted by Zenika. You'll hear from Gregory Boissinot and Adrien Lecharpentier about plugin development, and I'll be talking about workflow.

It's been a while we do a meet-up in Paris. Looking forward to seeing as many of you as possible. The event is free, but please RSVP so that we know what to expect.

Categories: Open Source

JUC SF 2014 is Here!

JUC SF on October 23, 2014 is shaping up to be bigger and better this year.

Here’s what we have in store for you!

Three Tracks

We’ve received a record high of 40 stellar proposals this year. To accommodate the many community proposals, we’ve decide to add a third track to the agenda. JUC SF sessions are now available for you to view. We have speakers from Google, Target, Gap, Cloudera, Ebay, Chicago Drilling Company, and much more. Register now for early bird price. The early bird price is only good until September 21, 2014.

Live Stream

If you can’t attend the conference in person, Track 1 sessions will be available via live stream, it’s all free. Brought to you by CloudBees. Registration for JUC SF live stream is here.

Get Drunk on Code

Have a beer while learning how to write Jenkins plugin. Steve Christou, Jenkins support engineer will lead this lecture from 3:30pm to 6:00pm. He will teach everything from how to get started, to techniques like writing a new CLI Command, to writing your own builder.

Ask the Experts

Meet the Jenkins creator, committers, support engineers, and developers. We have dedicated time slot(s) for our attendees to get 1 on 1 access to our experts. Exact time is TBD. Ask them anything from plugins, configuration, technical support, to bug fixes.

Our current list of experts are:

  • Andrew Bayer
  • Gareth Bowles
  • Steve Christou
  • Jesse Glick
  • Kohsuke Kawaguchi
  • Dean Yu

Want to join our panel of experts? Contact Alyssa Tong aly13@gmail.com

Exhibit Mixer

Sixteen technology sponsors will be showcasing their newest technologies during the exhibition hour from 2:25 – 3:30pm. Grab a beer, visit with sponsors and see how they are using Jenkins.

This is just a taste of what you’ll see at JUC SF. We look forward to seeing you there!!

Categories: Open Source

Workflow plugin code walk-through

Jesse and I will walk through the source code of the workflow plugin, highlights key abstractions and extension points, and discuss how they are put together.

If you are interested in developing or retrofitting plugins to work with workflows, I think you'll find this session interesting.

The event will be on Google Hangout tomorrow. The time of the day is the same as usual office hours.

Categories: Open Source

Test Framework Feature Comparisons – What If We Cooperated?

NUnit.org - Sun, 04/07/2013 - 03:14

Software projects often publish comparisons with other projects, with which they compete. These comparisons typically have a few characteristics in common:

  • They aim at highlighting reasons why one project is superior – that is, they are marketing material.
  • While they may be accurate when initially published, competitor information is rarely updated.
  • Pure factual information is mixed with opinion, sometimes in a way that doesn’t make clear which is which.
  • Competitors don’t get much say in what is said about their projects.
  • Users can’t be sure how much to trust such comparisons.

Of course, we’re used to it. We no longer expect the pure, unvarnished truth from software companies – no more than from drug companies, insurance companies, car salesmen or government agencies. We’re cynical.

But one might at least hope that open source projects might do better. It’s in all our interests, and in our users’ interests, to have accurate, up-to-date, unbiased feature comparisons.

So, what would such a comparison look like?

  • It should have accurate, up-to-date information about each project.
  • That information should be purely factual, to the extent possible. Where necessary, opinions can be expressed only if clearly identified as opinion by it’s content and placement.
  • Developers from each project should be responsible for updating their own features.
  • Developers from each project should be accountable for any misstatements that slip in.

I think this can work because most of us in the open source world are committed to… openness. We generally value accuracy and we try to separate fact from opinion. Of course, it’s always easy to confuse one’s own strongly held beliefs with fact, but in most groups where I participate, I see such situations dealt with quite easily and with civility. Open source folks are, in fact, generally quite civil.

So, to carry this out, I’m announcing the .NET Test Framework Feature Comparison project – ideas for better names and an acronym are welcome. I’ll provide at least a temporary home for it and set up an initial format for discussion. We’ll start with MbUnit and NUnit, but I’d like to add other frameworks to the mix as soon as volunteers are available. If you are part of a .NET test framework project and want to participate, please drop me a line.

Categories: Open Source

Software Testing Latest Training Courses for 2012

The Cohen Blog — PushToTest - Mon, 02/20/2012 - 05:34
Free Workshops, Webinars, Screencasts on Open Source Testing Need to learn Selenium, soapUI or any of a dozen other Open Source Test (OST) tools? Join us for a free Webinar Workshop on OST. We just updated the calendar to include the following Workshops:
And If you are not available for the above Workshops, try watching a screencast recording.

Watch The Screencast

Categories: Companies, Open Source

Selenium Tutorial For Beginners

The Cohen Blog — PushToTest - Thu, 02/02/2012 - 08:45
Selenium Tutorial for Beginners Selenium is an open source technology for automating browser-based applications. Selenium is easy to get started with for simple functional testing of a Web application. I can usually take a beginner with some light testing experience and teach them Selenium in a 2 day course. A few years ago I wrote a fast and easy tutorial Building Selenium Tests For Web Applications tutorial for beginners.

Read the Selenium Tutorial For Beginners Tutorial

The Selenium Tutorial for Beginners has the following chapters:
  • Selenium Tutorial 1: Write Your First Functional Selenium Test
  • Selenium Tutorial 2: Write Your First Functional Selenium Test of an Ajax application
  • Selenium Tutorial 3: Choosing between Selenium 1 and Selenium 2
  • Selenium Tutorial 4: Install and Configure Selenium RC, Grid
  • Selenium Tutorial 5: Use Record/Playback Tools Instead of Writing Test Code
  • Selenium Tutorial 6: Repurpose Selenium Tests To Be Load and Performance Tests
  • Selenium Tutorial 7: Repurpose Selenium Tests To Be Production Service Monitors
  • Selenium Tutorial 8: Analyze the Selenium Test Logged Results To Identify Functional Issues and Performance Bottlenecks
  • Selenium Tutorial 9: Debugging Selenium Tests
  • Selenium Tutorial 10: Testing Flex/Flash Applications Using Selenium
  • Selenium Tutorial 11: Using Selenium In Agile Software Development Methodology
  • Selenium Tutorial 12: Run Selenium tests from HP Quality Center, HP Test Director, Hudson, Jenkins, Bamboo
  • Selenium Tutorial 13: Alternative To Selenium
A community of supporting open source projects - including my own PushToTest TestMaker - enables you to apply your Selenium tests as functional tests for smoke testing, regression testing, and integration tests, load and performance tests, and production service monitors. These techniques and tools make it easy to run Selenium tests from test management platforms, including HP Quality Center, HP Test Director, Zephyr, TestLink, QMetry, from automated Continuous Integration (CI) tests, including Hudson, Jenkins, Cruise Control, and Bamboo.

I wrote a Selenium tutorial for beginners to make it easy to get started and take advantage of the advanced topics. Download TestMaker Community to get the Selenium tutorial for beginners and immediately build and run your first Selenium tests. It is entirely open source and free!

Read the Selenium Tutorial For Beginners Tutorial

Categories: Companies, Open Source

5 Services To Improve SOA Software Development Life Cycle

The Cohen Blog — PushToTest - Fri, 01/27/2012 - 00:25
SOA Testing with Open Source Test Tools PushToTest helps organizations with large scale Service Oriented Architecture (SOA) applications achieve high performance and functional service delivery. But, it does not happen at the end of SOA application development. Success with SOA at Best Buy requires an Agile approach to software development and testing, on-site coaching, test management, and great SOA oriented test tools.

Distributing the work of performance testing through an Agile epoc, story, and sprints reduces the testing effort overall and informs the organization's business managers on the service's performance. The biggest problem I see is keeping the testing transparent so that anyone - tester, developer, IT Ops, business manager, architect - follows a requirement down to the actual test results.

With the right tools, methodology, and coaching an organization gets the following:
  • Process identification and re-engineering for Test Driven Development (TDD)
  • Installation and configuration of a best-in-class SOA Test Orchestration Platform to enable rapid test development of re-usable test assets for functional testing, load and performance testing and production monitoring
  • Integration with the organization's systems, including test management (for example, Rally and HP QC) and service asset management (for example, HP Systinet)
  • Construction of the organization's end-to-end tests with a team of PushToTest Global Professional Services, using this system and training of the existing organization's testers, Subject Matter Experts, and Developers to build and operate tests
  • On-going technical support
Download the Free SOA Performance Kit On-Site Coaching Leads To Certification
The key to high quality and reliable SOA service delivery is to practice an always-on management style. That requires on-site coaching. In a typical organization the coaches accomplish the following:
  • Test architects and test developers work with the existing Testing Team members. They bring expert knowledge of the test tools. Most important is their knowledge of how to go from concept to test coding/scripting
  • Technical coaching on test automation to ensure that team members follow defined management processes
Cumulatively this effort is referred to as "Certification". When the development team produces quality product as demonstrated by simple functional tests, then the partner QA teams take these projects and employ "best practice" test automation techniques. The resulting automated tests integrate with the requirements system (for example, Rally), the continuous integration system, and the governance systems (for example, HP Systinet.)
Agile, Test Management, and Roles in SOA
Agile software development process normally focuses first on functional testing - smoke tests, regression test, and integration tests. Agile applied to SOA service development deliverables support the overall vision and business model for the new software. At a minimum we should expect:
  1. Product Owner defines User Stories
  2. Test Developer defines Test Cases
  3. Product team translates Test Cases into soapUI, TestMaker Designer, and Java project implementations
  4. Test Developer wraps test cases into Test Scenarios and creates an easily accessible test record associated to the test management service
  5. Any team member follows a User Story down into associated tests. From there they can view past results or execute tests again.
  6. As tests execute the test management system creates "Test Execution Records" showing the test results
Learn how PushToTest improves your SOA software development life cycle. Click here to learn how.


Download the Free SOA Performance Kit

Categories: Companies, Open Source