One of the few plugins that I still personally maintain is Active Directory plugin. In the past few months, I've been making steady improvements in this plugin, thanks to various inputs and bug reports given to me from the ClodBees customers.
One of the recent fixes was to get the "remember me" feature finally working for Active Directory. This requires a relatively new Jenkins 1.556, but it eliminates the need to having to constantly type the password in.
Then I've rebumped the version of COM4J, which was causing a thread leak when Jenkins runs on Windows. If you are running a Windows deployment with lots of active users, this probably would have contributed to the instability of Jenkins.
And then lastly, a small but crucial improvement was made to the way we search group membership, so that we can avoid recursively searching AD. This should result in a significant speed improvement when you are logging into Jenkins through AD.
The latest version of the plugin as of writing is 1.37. I hope you'll have a chance to update the plugin soon.
One of the new efforts in Jenkins this year is the acceptance test harness for Jenkins.
We will be doing the Jenkins office hours next week to go over this and sync up and coordinate between people in the community that are trying to work on this.
It'll be April 23rd 11am PT (see what this time is in your time zone) on Google Hangout at http://jenkins-ci.org/hangout. If you are intereste in hacking Jenkins or if you are a large user of Jenkins who have acceptance tests, we are looking forward to seeing you there.
For those of you who haven't looked, this test harness allows you to write blackbox tests of Jenkins and its plugins. It was originally used to test LTS releases, but over the time, it acquired a number of features, such as ...:
- Docker support for launching complex fixtures to test Jenkins with.
- Pluggability to launch Jenkins under test (JUT) in many different environments
- Pluggability to provision Jenkins and slaves from EC2 to test large deployments
- Choice of cucumber or JUnit to write test scripts
We are working on porting over existing test cases, but we'd like to work with users to move their acceptance tests on top of this same harness. The idea is to pool those test cases in the community so that we can test Jenkins and its plugins as we develop them. For this to work, we want tests to have lots of metadata (such as what plugins it touches), and for the harness to have sufficient modularity that different people can run the same scenario against different deployments, including existing instance.
The team is proud to announce the release of SonarQube 4.2, which includes many exciting new features:
- Multi-language analysis
- Tags of rules
- New visual measure filter representations (bubble chart, pie chart and histogram)
- Improved Issues page
The most voted JIRA ticket ever is now fixed! Running an analysis on a multi-language project is now rather simple. Just point to the parent directory containing all the source code and that’s it. Then, from the very same place, you can browse issues on all your files whatever their language.
Thanks to the tagging mechanism, you can now classify coding rules, which should ease searching.
Bubble chart, pie chart and histogram are now available to display your filters in nice and meaningful ways.
The Issues page was redesigned to make it easier to search for and browse issues.
Starting with the next 1.554.x LTS, the release model will switch to the train model, where we commit to dates and get whatever we can ship by that date.
You can see the scheduled dates in our event calendar. Backporting window for 1.554.1 is almost closing, so if you want to have your favorite issues nominated for it, please see the process in the Wiki and hurry!
InfoQ has been running a CI server survey for more than a month now, and here is the current result:
Jenkins has gotten more than 70% of the votes, once again proving the wide adoption among developers. If you are one of those who picked Cruise Control into the "considering" section, I'd encourage you to look around a bit more.
You can still vote from their website or leave comments if you want.
By the way, the design of two axes make no sense to me; for example, I'd order the adoption axis to "considering -> migrating to -> using now -> moving away from", and the circle seems to imply two axes are somehow interchangeable, when it should probably be just in a checkerboard to indicate those are independent axes.
NIO-based Java Web Start (JNLP) slave handling is coming to 1.560. This will help you run a large number of JNLP slaves more efficiently. A connected JNLP slave used to occupy one thread on the master, but now it occupies none. Combined with the earlier change that eliminated threads from idle executors, now you can connect thousands of slaves.
All you have to do is to use the latest slave.jar from Jenkins 1.560. No other changes are necessary on users' part.
A bulk of this is implemented in remoting 2.38, and a good part of it was implemented about a year ago on the airplane on the way to Europe.
We plan to make CLI connections take advantages of this too, which helps those who use that a lot. That's not in 1.560, but hopefully it'll be in the near future. This change also paves a way for multi-participant bus-topology communication, which I think would be an useful building block for the work-in-progress master-to-master API.
Good taste prevents me from embedding a trumpet fanfare into this post, but it does seem warranted. After all, with the release of SonarQube version 4.2 last week, SonarSource has finally implemented the all-time highest voted ticket in the SonarQube backlog: multi-language analysis.
Okay, now for a tiny dose of reality. The ability to participate in a multi-langauge analysis must be implemented in each language plugin separately, so it may be a while before your project is fully inspected with just one analysis. But very soon you’ll be able to remove the sonar.language property, and SonarQube will automatically analyze every file with an extension it recognizes.
Of course, if you don’t want to turn on multi-language analysis yet, then all you have to do is retain the sonar.languages property, and you’ll get the legacy behavior: analysis only of the single language you specified.
I recently had an opportunity to visit a big Jenkins user on site, and one of the things they've told me is that building projects in the Maven job type is substantially slower than doing the same with the freestyle project type.
This is partly expected, because this job type does more for you. For example, it automatically archives your build artifacts, fingerprints all the relevant information, and so on. These are good things, and naturally, it cost time.
But the slow down they are seeing was substantial, and this is a complaint I've heard from others as well. So I started looking into it.
With a help of artificial delay induced to my network interface and several custom scripts to probe into the running processes, I was able to understand what was going on and make some good improvements.
First, in Maven plugin 2.0, we've made a change in the way we archive artifacts from Maven. Previously, the artifacts were copied between the master and the Maven JVM, and for a reason I'll mention later, this was very slow, especially in a network that has a large latency. With Maven plugin 2.0 and onward, artifacts are archived between the master and the slave JVM.
The second problem that I discovered was that the spy program we put inside Maven is causing excessive amount of unnecessary classloading. Some classes have static initializers that too eagerly refer to other classes, which in turn brings in other classes, and so on. Despite the jar file caching that we do, these classloading still sometimes requires precious roundtrips to the master, which costs in the order of 10s of ms. I was able to make various changes in Jenkins core to cut this down, and these fixes will land in Jenkins 1.559 (ETA is April 14th.) The classloading overhead is independent of the size of your Maven build, so this improvement is more for people who have lots of small Maven builds, like Jenkins building Jenkins plugins.
Now, on to the biggest fruit of this investigation I was able to discover and fix. Imagine the Maven JVM has a lot of data to send to the master, say you are archiving test reports or code coverage report. A good implementation would send these data as fast as ppossible to the master, paying respect to the limit of flow control to avoid overwhelming the master.
It turns out that the way we set up this communication channel was far from optimal. Instead of having the Maven JVM push data with flow control, we were relying on the master to pull data. That is, master has to send out a request to the slave to fetch the next batch of data (8KB), then once it receives that data, it sends out another request to fetch the next batch of data, and so on. If your network latency is 10ms, this scheme only lets us send 500KB/sec, even if you have a gigabit ethernet. No wonder it was so slow!
This fix is in in Maven plugin 2.2. See JENKINS-22354 if you want to know more about the actual diffs and such.
Unfortunately, none of these are available for those who are on 1.532.x LTS, but the next 1.554.1 LTS will be able to run the newer Maven 2.2 plugin. So the help is on the way!
In case of a connection loss, this type of slaves has been designed to automatically attempt to reconnect to the master. This makes sense because you want these slaves to remain online all the time, even if your janitor trips over the ethernet cable. Unfortunately, it also means that over the time, these slaves accumulate gunk, such as mutated static states, any left-over threads or memory leaks, or native libraries that are loaded into JVM.
To prevent that, a better approach is to restart the slave JVM (JENKINS-19055) and have the new JVM reconnect, instead of having the same JVM reconnect. That would ensure that the slave always stays clean. I've planned to make this change for a while now, and I'm happy to report that this change is finally landing to the upcoming 1.559.
Restarting JVM is easy on Unix, where I could just exec(3) to itself. We've been doing this for ages on masters, for example when you update a plugin and tell Jenkins to restart.
The hard part is to do this for Windows, where the most of the time was spent. I had to improve windows service wrapper to support self-restarting services, which turned out to be trickier because Windows service control manager doesn't provide "restart" as an atomic operation. It also kills not just the service process itself but all the processes in the group. So I had to double-fork the service wrapper into a separate process group just to restart a service from within itself.
In any case, the end result is that if you have installed a service through GUI, be it on Windows, Unix, or OS X, slaves will restart themselves every time it gets disconnected from the master.
I've also taken the opportunity to make jenkins-slave.exe on the slave self-updating. Every time it connects to the master, it gets the latest version from the master.
If you have installed Web Start slaves as services, make sure to update the local copy of slave.jar on these slaves to 2.37 or later. This "restart on reconnect" feature only kicks in when you are running this very recent version of slave.jar. And yes, we realize it'd be nice for slave.jar to update itself, which is tracked as JENKINS-22454. But that's a work for another day.
Jenkins User Conference (JUC) season is upon us! It’s a busy year for the Butler – he’s hosting conferences all over and looking for sponsors to help:
- Boston – June 18
- Berlin – June 25
- Herzelia, Israel – July 16
- Bay Area (California) – October (date TBD)
Mr. Jenkins and the JUC Organizing Committee want to invite you and your company to sponsor a JUC this year. Show your support for the Jenkins community and help keep costs low for attendees*. The funds go to are put to good use: conferences are two full tracks. Lunch, light breakfast, coffee and a coveted Jenkins t-shirt are also included.
Sponsors get all sorts of thanks from the Jenkins community:
- Your logo on the conference t-shirt and all other conference communication (emails, website, signage, etc.)
- A blog featuring sponsors
- Free passes
- Silver and Gold sponsors get a table to talk to folks and hand out swag
- Gold sponsors get either a speaking slot, happy hour sponsorship or a dedicated room for demos
- And more, but most especially, you get to support JenkinsCI.
Just let us know if you’re interested to get the details. We’d love to have you join us.
Friendly reminder: We are looking for speakers for all four cities. Call for Papers ends March 30 for Boston, Berlin and Israel. Submit your abstract now and come share your expertise with the Jenkins community.
We hope to see you at a JUC this year!
Lisa, Alyssa and the JUC Organizing Committee
*PS - Registration just opened for Boston and early-bird tickets are only $59.
Since I joined SonarSource full time at the beginning of this month, I’ve been thinking a lot about ducks and belly dancers.
That seems like an odd combination, but they have more in common than you might think. Most people have heard the old saw about being like a duck: stay calm on the surface and paddle like hell underneath. It’s pretty much the same for belly dancers; those big, poofy pants (or skirts) they wear aren’t simply a fashion statement; they’re intended hide how hard the dancer’s legs are working in the process of making the rest of the body move.
It’s pretty much the same story for SonarSource. As a member of the SonarQube community, and even as a part-timer at SonarSource, I had no idea how much work goes into each new version of SonarQube and the language plugins. Having spent the first two weeks of this month at the La Roche office, I’m beginning to understand.
As a community member, I was concerned mainly with SonarQube itself and the language plugins I used every day. Wearing the blinders of that narrowed focus, it always seemed like a very long time between releases. With the blinders off, and seeing more clearly everything that goes on, I’m astounded by how often new versions go out the door. From November through January, there were 18 major releases, and that doesn’t count the sort of internal refactoring that will make it easier to continue pumping out high-quality code in the future.
What I saw in Jira and on the mailing lists is the above-the-water portion of the duck (or the above-the-legs portion of the dancer, depending on which metaphor you prefer). What’s going on underneath is an incredible amount of testing (manual and automated), test maintenance, management of milestones and RC’s, and yes monitoring and improvement of quality.
As a community member, I had noticed how few real issues are turned up for each new release candidate. I had also noticed that sometimes RC1 of a new version isn’t the first release candidate to be announced to the community for testing. The exhaustive tests that SonarSource itself performs mean that most issues are caught internally before the community ever sees a new version – often in the milestone phase, but sometimes in the release candidate phase.
That’s because the focus at SonarSource is not just making quality monitoring software, but making quality software. Making it look easy is just a byproduct.
Over the past three years, the Jenkins User Conference is held annually in the Bay Area with a few events in different locations around the world. The Jenkins User Conference has established a reputation as a focal point for the Jenkins community to come together to share new ideas and best practices. Each year we have experienced the growth and expansion within the Jenkins community. This year we are taking this platform to other regions of the world, offering regional gatherings of Jenkins users and developers.
At the moment, we are working on the following JUCs and events for 2014:
- JUC Boston - June 18
- JUC Berlin - June 25
- JUC Israel - July 16
- JUC San Francisco Bay Area - October (TBD)
- JUC Australia/New Zealand - November/Dec (TBD)
- Copenhagen - September
- Brazil - November/December (TBD)
There are a few different ways to get involved:
- Register Now to Attend: JUC Boston. Registration pages for other JUCs/events are coming soon. Please check back here often.
- Be a Sponsor: There are a few ways to become a sponsor.
- Submit a Talk Proposal: deadline is March 30
- Submit a great idea for t-shirt design in the comment box below or email email@example.com
- Submit ideas for swag in the comment box below or email firstname.lastname@example.org
- Be part of the JUC committee: contact email@example.com
- Learn more about Jenkins User Conferences
- Sign up and stay up to date with the latest Jenkins newsletter
Looking forward to seeing you at a local JUC near you! :o)
If there’s a set of data you regularly look up in SonarQube, the Measures Service – and saved filters – are going to be your new favorite SonarQube features.
A user at my day job recently showed me a spreadsheet he’s using to track the metrics of the “worst offender” files in his COBOL project. I was afraid I already knew the answer, but I asked how he was getting the data for each file. It was one time I wasn’t happy to be right – he was doing it the hard way, manually recursing each branch of his project’s components tree to find the numbers.
That’s when I pointed him to the Measures service. It lets you search for any type of resource based on a host of criteria. He was looking for files in his project that exceeded a certain threshold. This doctored screenshot shows the kind of search I showed him how to run:
First he specified what he was looking for: files, and then the criteria by which to choose them: Components of the project, SonarQube in this case, with Coverage less than 90%. Just having the list of relevant files pulled neatly together thrilled him; I could tell by the way the corner of his mouth quirked up.
His mouth quirked again. He was really happy. Except…
We both started to speak, but he beat me to the gate, “Can you save…?” I knew he didn’t want to have to reconfigure this every time – who would?
I had him close column editing mode (a crucial step), and the “Save As” link reappeared:
He gave it a name, and knew that from then on, it would always be waiting for him in the saved filters menu:
I got the mouth quirk again.
Then, I showed him how to use a “Measure Filter as List” widget on his own private dashboard to display the saved filter automatically, and pointed out that he could make that dashboard the first page he saw in SonarQube.
He actually smiled.
It makes me enormously happy and proud to announce that the Selenium Conference 2014 will be held in Bangalore on the 4-6 September. I’m looking forward to seeing you there!
One of the plans we’ve had from the very beginning for SeConf was that it was going to be a conference for the community of people who make Selenium such a fun project to work on. One way to do this was to host the conference where the largest groups of people using Selenium are found. We kicked off the first conference in San Francisco mainly because of the large number of Selenium users there (and, I’ll be honest, because that’s where the organising team had the most experience and contacts!)
In Europe, that large pool was London, so we held the second conference there. We had originally planned for the third conference to be in New York, but that proved to be a little too expensive, so we moved it North a little to Boston. Essentially, the pattern is that we alternate between the US one year and The Rest of the World the other.
That brings us to the planning for Selenium Conference this year. We had a look at the data available to us, and noticed that there were two areas of the world that it would be great to take the conference to. Selenium Camp, hosted in Kiev each year, does a great job of catering to one of these groups, so that leaves the second.
It’s India’s turn.
Thank you to everyone who’s already poured so much heart and spirit into this conference. We’ll be putting up a call for papers and more details soon, so please stay tuned!