I'm going to visit Sao Paulo once again this weekend to attend the second annual Jenkins users meet-up. It's a whole day free event Saturday full of Jenkins goodness.
You'll hear from a number of active Jenkins folks, and I'll be presenting about what CloudBees (where I currently work) has contributed to the Jenkins project, including recent new OSS plugins and some services. I'm also stuffing my suitcase with lots of give-aways, including Jenkins stickers and popular Jenkins bobble heads. I don't intend to bring anything back to the U.S.
The morning half of the event is a cross-atlantic hackathon between Brazil and Copenhagen. you can check what's being planned on the western side of the ocean and the eastern side of the ocean. The afternoon half is a series of presentations. Please come join us. I'm really looking forward to seeing you!
I'll be in Sao Paulo for the whole Sunday and Monday as well. If you are interested in talking to me, please feel free to drop me a note.
It's been a month now, but I realized that I've never posted a wrap-up post of JUC 2013. So in the spirit of "better later than, never", here it goes.
First of all, I wanted to thank everyone who came. More than 400 of you came, and another 600 signed up for live streaming of events (and I know some people watched those live streams past midnight in their local time zone!). I did my part in signing bobble heads and answering questions, and I was able to finally put faces to some of the people who I actively interact in the community but never met before.
All the slides are video recordings are available online if you couldn't join us.
Alyssa said she's got a lot of feedbacks from folks, and she's already planning for the next year — if you are interested in sharing your thoughts on how to better do this next year, we've put it up for the next week's Jenkins project meeting agenda to talk about how to do it.
Finally, everything in the San Francisco Bay Area is incredibly costly, and events like this was really only made possible by generous sponsors, and we really want to make them happy so that they can help us make this event happen next year as well. So I thought the least I can do is to give them a spotlight and talk about who they are and what they do:
form you wish to use it - on-premise or in the cloud. Check out the many
resources available on the CloudBees website for Jenkins fans - whether
you use open source Jenkins, Jenkins Enterprise by CloudBees or Jenkins in the cloud. Gold Sponsors Appvance delivers technology and services to prove and improve
performance, security and scalability of websites, apps and mobile apps.
The largest brands in the world choose Appvance, from Pepsi to Best Buy
to Bell Alliant. Learn more.
Have questions on SDLC tools or agile process (especially Jenkins
Enterprise, CI or CD)? Leverage our 25 years of expertise for assistance
with CloudBees, Xebia Labs, Sonatype, JFrog, Atlassian, SVN, Git,
Rational, Microsoft TFS and many more. Visit www.BDS.com to learn more..
digital video advertising for the world’s largest brands. Jenkins has
become a core piece of our productivity tech stack here at BrightRoll,
and its importance is increasing. During the time that we've used it
we've seen a huge benefit to participating in the Jenkins community,
getting support from core contributors and plugin authors, and we try to
contribute back whenever we can. www.brightroll.com
The Jenkins User Conference is the only place you can actually feel the
Jenkins community and understand that being part of it is not just a
commitment, it is a privilege we are honored to share. Learn more about
JFrog, our Artifactory Binary Repository solution, and our new Bintray social platform for sharing, publishing and managing binaries.
LMIT Software is now GerritForge, the leader in Agile coaching and
Development Management. We are active contributors of Jenkins (see http://jenkins-ci.mobi) and Gerrit Code Review and we can enable their adoption and integration into the Enterprise Continuous Delivery chain.
provides end-to-end, real time visibility into the operations of network
connected applications wherever they run – across browsers, mobile
devices and servers. Sign up for a FREE account at
newrelic.com/cloudbees. With CloudBees DEV@cloud (Jenkins in the cloud) or Jenkins Enterprise by CloudBees, you can instantly connect to XebiaLabs Deployit (a fully automated deployment solution) and immediately begin reaping
the benefits of delivering continuously. Missed Andrew Phillips' JUC presentation, Preparing for Enterprise Continuous Delivery: 5 Critical Steps? View the slides here.
As reported in various places, there was an incident in early November where commits in our Git repositories have become misplaced temporarily by accident. By the mid next week we were able to resurrect all the commits and things are back to normal now.
As there are many confusions and misunderstandings in people’s commentary, we wrote this post to clarify what exactly happened and what we are doing to prevent this.Timeline
In the early morning of Nov 10th 2013, one of the 680 Jenkins developers had mistakenly launched Gerrit with a partially misconfigured Gerrit replication plugin, while pointing Gerrit to his local directory that contains 186 Git repositories cloned from the Github Jenkins organization. These repositories were checked out about 2 months ago and weren’t kept up to date. Gerrit replication plugin had then tried to “replicate” his local repositories back to GitHub, which it considers mirrors, by doing the equivalent of “git push --force” instead of regular push. Unfortunately, Gerrit replication plugin defaults to a forced push, which is the opposite of what Git normally does. The replication also happens automatically, which is why this mistake has impacted so many repositories in such a short time.
As a result, these repositories have their branch heads rewinded to point to older commits, and in effect the newer commits were misplaced after the bad git-push.
When we say commits were "misplaced", this is an interesting limbo state that's worth an explanation for people who don’t use Git. A Git commit is identified by its SHA1 hash, and these objects will never get overwritten. So the misplaced commits are actually very much on the server intact. What was gone was the pointer that associates a human-readable branch name (such as "rc") to the latest commit on the branch.
By Nov 10th 12:54pm GMT, multiple developers had noticed this, and within several hours, we figured out what happened. From Gerrit log files and with the help of GitHub technical support, he was able to figure out all the affected repositories, and later an independent script was written to verify the accuracy of this list.
Some of the Jenkins developers were closely following this development, and were able to restore branches to point to correct commits by simply pushing their up-to-date local workspaces back into the official repositories. Git makes it very easy to do this, and most of the popular plugins affected were restored in this manner within 24 hours.
At the same time, we needed to systematically restore all the affected repositories, to make sure that we have not lost anything. For this, we contacted GitHub and asked for their help, and they were able to mostly restore branch heads to their correct positions. We have also independently developed a script to find out exactly what commits branch heads should be pointing to, based on the GitHub events API that exposes the activities to Git repositories. This script found a dozen or so branches that fell through the cracks of GitHub support, and we have manually restored those.Mitigation in the future
The level of support we got from GitHub and our ability to independently verify lost commits and subsequently restore them made us feel good about GitHub, and we have gained confidence in our ability to recover from future incidents.
That said, what happened was a serious disruption, and it’s clear we’d better prepare ourselves both to reduce the chance of accidents like this and increase the ability to recover. To that end, we hope GitHub would expose a configuration option to disable forced ref updates. They already do this on GitHub Enterprise after all. Dariusz pointed out that CollabNet takes this one step further and offers ability to track deleted branches, tags, and forced updates. Something like that would have made the recovery a lot easier.
We are going to make two improvements to our process so that we can recover from this kind of problems more easily in the future.
Firstly, we’ll develop a script that continuously records the ref update events across the GitHub Jenkins organization. This will accurately track which branch/tag is created/updated/deleted by who. In case of an incident like this one, we can use this log to roll back the problematic push more systematically.
Secondly, we’ll allow people to control access to individual Git repositories, as opposed to give them all or nothing access to the entire array of plugin repositories.
The Jenkins developers decided to continue the current open commit policy despite the incident to preserve our culture, which helped us navigate through this incident without a single argument nor flame war.FAQ Does everyone in the organization have full commit privileges to all the repositories?
Yes, with some exceptions. To encourage co-maintenance of plugins by different people, and to reduce the overhead of adding and removing people from our 1100+ repositories, we use one team that gives access to most repositories, and put committers in this team.I prevent forced push in my Git repositories. I’m safe from this trouble, right?
No, unfortunately something like this can still happen to you, as you can also accidentally delete branches. If you want to learn from our mistakes, you should definitely enable server-side reflog, to track ref updating activities. “git config core.logAllRefUpdates true” on the server will enable this.Can’t you just have people with up-to-date copy push their repos and fix it?
This is indeed how some of the repositories got fixed right away, where some individuals are clearly in charge and are known to have the up-to-date local repositories. But this by itself was not sufficient for an incident of this magnitude. Some repositories are co-maintained by multiple people, and none of them are certain if he/she was the last one to push a change. Many plugin developers just scratch their own itch and do not closely monitor the Jenkins dev list. We needed to systematically ensure that all the commits are intact across all the branches in all the affected repositories.Can’t you just roll back the problematic change?
Most mistakes in Git can be rolled back, but unfortunately ref update is the one operation in Git that’s not version controlled. As such Git has no general-purpose command to roll back arbitrary push operation. The closest equivalent is reflog, which offers the audit trail that Git offers for resolving those cases. But that requires direct access on the server, which is not available on GitHub. But yes, this problem would not have happened if we were hosting our own Git repositories, or using Subversion for example.
This is a guest post from Alyssa Tong, who drives JUC organizations around the world.
If you missed JUC Palo Alto on Oct 23, 2013 the videos are now available.
We are off to planning JUC 2014. It is hard to believe this will be the 4th annual JUC in the Bay Area. The growth in the Jenkins community since the first JUC is astounding.
Every year we are in search of a larger venue to accommodate the larger crowd. For 2014, the challenge of finding a venue for a capacity of 500+ attendees at a low cost will prove even more daunting. We would love to hear your suggestions for low cost venues (in the Bay Area) so that we may continue to keep entry cost low while providing convenience and the highest level of Jenkins education to attendees. Please send suggestion(s) to firstname.lastname@example.org
We are proud to launch the call for volunteers to join the JUC organizing committee (OC). If you are interested in shaping the 4th edition of this great event, please send email to email@example.com
We encourage you to share this blog within your network in case other people would be interested in joining the JUC OC or have ideas for a great JUC 2014 location.
In the hope of streamlining account creation e-mail delivery and mailing list moderations, I have deployed SPF and DKIM over the weekend for e-mails coming out of @jenkins-ci.org, which includes account appliations, Confluence, and JIRA.
I've also used this opportunity to switch back the sender of JIRA notifications to firstname.lastname@example.org. It was originally this way, then changed to email@example.com when someone complained (on what ground I do not remember any more.)
To the degree that I have tested the setup, it is working correctly, but if you notice anything strange, please let me know.
In this Open Space Technology style event, we went over war stories from users. Just to show the degree of seriousness, some of those people run 1500+ slaves, and others run Jenkins in HA configuration with a data center fail over! We then picked various topics in the afternoon and discussed what people would like to see to make Jenkins scale further. Slides and raw notes from this meeting is available here.
The event allowed me to rethink and revisit what I thought we should do in coming days in the area of scalability.
The event was far more popular than we anticipated originally, and we had to turn down many folks. So I'm going to do a webinar to go over what we did, and what we talked about. If you are interested in this area, and want to see what's being considered and provide your thoughts, please join us on Nov 19th 10am PT.
(This is a guest post by Alyssa Tong, the lead coordinator of Jenkins User Conference)
Our 3rd annual Jenkins User Conference in the Bay Area being held next Wednesday in Palo Alto is booked fully to the capacity and we couldn’t be more excited for this event! It’s going to be an amazing day of learning, talking to technology experts, networking with other Jenkins users, seeing cool demos and finding out how you can contribute to the Jenkins open source projects.
This event is being held at the Oshman Jewish Community Center and registration begins at 8am. There will be breakfast and plenty of coffee to get you caffeinated. Welcoming announcement will begin sharply at 9am and the keynote address follows shortly after. We’re so excited to have thirteen sponsors investing in and supporting the Jenkins community in this continuous integration space.
New this year, there will be BoF sessions so be sure to sign up for your preferred discussion at check-in. Or suggest a topic by leaving your suggestion in the comments section below. Let us know what Jenkins topic(s) is near and dear to your heart.
For those who missed out on purchasing your ticket or are unable to attend, we are happy to offer the live stream of Track 1. You can choose to watch the entire track or just specific session(s). Either way don’t forget to chat and tweet. We will also tweet live from the conference so you can follow along that way as well. Follow @jenkinsconf for the latest updates.
Thank you to everyone for making this sold-out event possible.
Can’t wait to see everyone on Wednesday!