If you were unable to attend Sonatype’s last webinar on Build Promotion with Nexus Professional, you can view the presentation recording here. This webinar is an introduction to the Build Promotion capabilities of Nexus Professional, the industry leading repository manager to help companies acquire, manage, and report on open source software artifacts in their software development projects.

The Build Promotion feature in Nexus Professional allows an organization to create a temporary staging repository and to manage the promotion of artifacts from a staging repository to a release repository. This ability to create an isolated, release candidate repository that can be discarded or promoted makes it possible to support the decisions that go into certifying a release.

Upcoming Webinars:

Build Promotion with Nexus Professional

When: Thursday September 9th, 2010 at 10:30 am PDT, 1:30pm EST

Who: Blaine Mincey, Senior Systems Engineer, Sonatype

Register for this webinar

 

Sonatype will be holding a much anticipated webinar on Hudson Integration with Maven Studio for Eclipse next month.  On Tuesday September 28 Senior Systems Engineer at Sonatype, Blaine Mincey, will host a webinar that will show you how Hudson Integration will increase productivity, by allowing you to view the build results from your Hudson jobs within your own IDE.

This webinar will highlight features that are available with the Hudson Integration feature of Maven Studio for Eclipse. It will review the benefits of Hudson Integration directly from your Eclipse development environment.   Finally, it will demonstrate how you can track the status of interesting jobs in Hudson so you know immediately if something needs to be fixed without leaving your IDE.

Hudson Integration with Maven Studio for Eclipse webinar:

• Tuesday September 28, 2010
• 6:00 am PDT, 3:00 pm Europe Summer Time (GMT+02:00)
• To register, please click here

 

The last webinar in our series, Build Promotion with Nexus Professional, was a huge success.  Thank you to all who participated! Sonatype is happy to announce another free webinar will be held this Thursday at 6:00 am PDT, 3:00 pm Europe Summer Time (GMT+02:00).

This webinar is an introduction to the Build Promotion capabilities of Nexus Professional, the industry leading repository manager to help companies acquire, manage, and report on open source software artifacts in their software development projects.

If you release software, you will often need to test a release before deploying it to a production system or an externally accessible repository. For example, if you are developing a large, enterprise web application you may want to stage a release candidate to a production system and perform a series of rigorous tests before a release manager makes a decision to either return a system to development or deploy a system to production.

The Build Promotion feature in Nexus Professional allows an organization to create a temporary staging repository and to manage the promotion of artifacts from a staging repository to a release repository. This ability to create an isolated, release candidate repository that can be discarded or promoted makes it possible to support the decisions that go into certifying a release.

To register for this webinar, please click here.

 

I’ve seen my fair share of projects migrating from Ant to Maven, and, for a complex project, this migration path can take some time. You have to worry about dependency management, project structure, and retraining an existing team to use Maven and understand the core concepts behind the tool. When you make the shift, you are often affecting development infrastructure for an existing project, and you need to take into account development environments as well as developer’s ideas about how code should be organized and stored in source control. In this post, I’m going to discuss a common pattern I’ve seen in Ant to Maven migrations: how to migrate the monolithic project.

A Common Pattern: The Monolithic Project

Ant projects which have evolved over many years often lack modular structure. While it is certainly possible to create the equivalent of a multi-module Maven project in Ant, the usual progression in an Ant project is to store all of your source in a single tree and use extra targets to selectively compile different packages. This approach is shown in the following figure.

In the most extreme cases, there is a single project which contains an array of modules. Maybe you have an entire enterprise system all stored in a single project alongside a complex Ant build.xml file which contains a collection of tasks for each component. A monolithic Ant project usually produces a series of artifacts: a JAR containing some API for clients, a server-side web application, a utility library, etc. Developers have usually grown so accustomed to this approach that the idea of splitting up a complex system into a series of related submodules can see daunting.

Moving to Maven: Do we need all these projects?

One of the first questions I get in a Maven migration from Ant is whether it is really necessary to modularize a project. “Do we really need to create all these projects for Maven?” (Short answer: Yes, there is no avoiding this.) This is usually accompanied by a concern that Maven limits each project to producing a single artifact. These are two core assumptions of Maven: your projects will be modularized and each project should produce a single build artifact.

Now, while these are core assumptions, there are ways around them, and I’ve even seen people go to extreme lengths in an attempt to preserve this single, monolithic project. You can create an elaborate set of profiles to modify the build depending on the context in which it is run, and you can attach extra build artifacts to a project using assemblies. Even though Maven makes assumptions about code layout and project structure, it can do just about anything, and in this case, it can be used to approximate the monolithic, combined Ant project.

If you do this, if you try to trick Maven the first thing you’ll notice is that your monolithic project’s POM is going to be somewhat unwieldy: it is going to be massive. Instead of a simple, declarative picture of a project, you are going to have a POM that contains multiple assembly definitions. Each assembly definition is going to have to explicitly define the structure of your build artifacts and you are going to find yourself venturing into includes and excludes patterns for things like the Maven Compiler plugin. In other words, you are going to have to expend a huge amount of effort to bend Maven to your assumptions.

If you do this, you really won’t be using “Maven”. You’ll be using your own interpretation of Maven, and once you go down this road, you are going to start having problems using standard tools designed for Maven. In short, don’t do this. Don’t try to bypass Maven’s approach to modularity with assemblies and profiles. If you do, you are going to find yourself “swimming upstream”, and your first hint is going to be the size and complexity of your POM files. I’ve certainly created some complex POMs in my time, but I only need to do this for projects that really require customization because they are doing something unique (see the POMs for the Maven book builds, they are large and extremely customized).

If you are creating web applications, EARs, and simple Java applications you shouldn’t have large POMs. If you do, it is probably a sign that you have ventured too far “off the map”. While you might be building your project with “Maven”, there is likely something wrong with your approach. If you find yourself constantly challenging an assumption as basic as project modularity, then you need to either rethink using Maven entirely or rethink your project structure.

Adopting Maven Means Adopting Modular Structure

If you are going to adopt Maven, you must adopt a modular project structure. If you don’t you will be fighting an uphill battle with Maven and the tools that have been designed to work with it. You will be fighting not only with Maven, but you’ll be doing constant battle with your IDE and your repository manager.

If you are moving from Ant to Maven, do you yourself a favor. Adopt a modular project structure. Don’t approach Maven as a “toolbox” like Ant with tasks and targets. Approach Maven as a framework with expectations and assumptions. If you do this, you’ll have a much easier time adopting the tool.

 

Sonatype will be presenting at Eclipse Day hosted by Google in California this week.

Each year Google hosts an Eclipse Day and puts together an agenda that includes Eclipse and Google related topics. This year there will be presentations that feature Android, Helios, GWT, EGit, Linux Tools, Eclipse 4.0, EMF, XText and more. Sonatype’s founder, Jason van Zyl, will be presenting on Next Generation Development Infrastructure with Maven, m2eclipse, Nexus and Hudson.

This talk focuses on the Maven Ecosystem and the under-pinnings of technologies that are going to shape the Next Generation Maven Development Stack. While the Maven 3 release retains backward-compatibility with plugins written for Maven 2, the foundational technologies used in Maven are being recast to allow for more space for expansion as Maven starts to expand into newer spaces like OSGi and Polyglot development. In this talk, van Zyl discusses new, emerging technologies and how they fit into the overall approach to the development of a Next Generation Maven Development Stack. This presentation will include an overview of the following technologies:

• Moving Maven from Plexus to Guice
• Continued work to support OSGi development with Tycho
• Support for Polyglot development and alternative languages
• m2eclipse, the primary Eclipse IDE Maven-integration used at Sonatype which is also integrated with STS and JBoss Tools.

Eclipse Day hosted by Google:

• Thursday, August 26, 2010
• 9:00am – 5:00pm
• Googleplex
• 1600 Amphitheatre Parkway
• Mountain View, CA

 

Sonatype is pleased to announce version 1.5 of the Nexus Migration Plugin. This version contains an updated implementation of our Artifactory to Nexus migration tool. If you are currently running Artifactory and you want to migrate to Nexus, all you need to do is:

1. Create a System Export from Artifactory
2. Install Nexus with the Nexus Migration Plugin
3. Import Your Artifactory System Export to Nexus
4. Configure the Artifactory Bridge to seamlessly serve artifacts to existing clients

You can use the Nexus Migration Plugin with Nexus Open Source or Nexus Professional. To get started, download Nexus Open Source or Nexus Professional, then download the Nexus Migration Plugin. Read Appendix A of the Nexus Book and learn how to:

• Install the Nexus Migration Plugin
• Create a System Export in Artifactory
• Import an Artifactory System Backup into Nexus
• Configure Repository Mapping and Import Settings
• Map Legacy Artifactory URLs to Nexus

 

Thanks to all who attended Thursday’s webinar on Build Promotion with Nexus Professional.  To access a recording of this webinar, please click here.

If you missed this week’s webinar, the next Nexus Professional webinar will be held on Thursday August 26, at 6:00 am PDT, 3:00 pm Europe Summer Time (GMT+02:00).

This webinar is an introduction to the Build Promotion capabilities of Nexus Professional, the industry leading repository manager to help companies acquire, manage, and report on open source software artifacts in their software development projects.

To register or learn more, please click here.

 

Sonatype is pleased to announce the availability of our most popular book, Maven: The Complete Reference, as an epub. Click here to download the book as an epub and start reading this book on a mobile device.

If you have any questions or feedback, we encourage you to send email to book@sonatype.com. If you have any problems to report, please include the section number and the type of mobile device you are reading this book on. This book has been tested on iPad, iPhone, Sony Reader library and the Adobe Digital Edition reader.

If you are new to Maven, don’t forget to check out Nexus Open Source and Nexus Professional. If you develop with Maven, you should be using Nexus. Once you start using a Repository Manager, you won’t believe you ever developed without one.

 

Sonatype is hosting a free webinar this week on Build Promotion with Nexus Professional.  The webinar is hosted by Blaine Mincey, Senior Systems Engineer at Sonatype, and will be held on Thursday August 19 at 10:30 am PDT, 1:30pm EST.

The webinar is an introduction to the Build Promotion capabilities of Nexus Professional, the industry leading repository manager to help companies acquire, manage, and report on open source software artifacts in their software development projects.

If you release software, you will often need to test a release before deploying it to a production system or an externally accessible repository. For example, if you are developing a large, enterprise web application you may want to stage a release candidate to a production system and perform a series of rigorous tests before a release manager makes a decision to either return a system to development or deploy a system to production.

The Build Promotion feature in Nexus Professional allows an organization to create a temporary staging repository and to manage the promotion of artifacts from a staging repository to a release repository. This ability to create an isolated, release candidate repository that can be discarded or promoted makes it possible to support the decisions that go into certifying a release.

Register today!

 

The Nexus 1.7.2 release offers an improved search interface making it even easier to locate the libraries and artifacts you need in Nexus.  Sonatype has published a version of Nexus 1.7.2 on http://repository.sonatype.org which contains some dramatic improvements to the search interface.   Download the new Nexus Open Source or Nexus Professional release and start searching for artifacts.

What is new in the Nexus 1.7.2 search interface?

• Search results now link directly to the latest version of a matching artifact.
• Selecting a search result immediately displays information about the matching artifact.  You can browse artifact information from the search interface.
• (Nexus Professional) Archive browsing and artifact metadata are available from the search interface.
• Matching artifacts of different types (pom, jar, war, zip, etc.) can be downloaded from the search results page.

This release takes the effort out of searching for artifacts in Nexus.  Here are some sceenshots of the new interface now available on http://repository.sonatype.org and soon to be available in the 1.7.2 release of Nexus Open Source and Nexus Professional.

If you are searching for artifacts in Nexus, start with either the Advanced Search interface or the Welcome panel shown below.    This Welcome panel is the first thing you will see when you load Nexus in the browser, to see the welcome screen go to http://repository.sonatype.org.

To search for an artifact, just type in a keyword.  A keyword can be a groupId, an artifactId, or just the name of a project.   If you were searching for the latest version of the Guice library, you would type in “guice” in the Welcome panel and either press Return or click on the search icon to the right of the search box.    Searching for Guice will show the following result panel.

What’s new in this interface?

• Look at the Version column in the search results.   In this version of the search interface, we decided to list the most recent version.    If you need to view a different version, click on “Show All Versions”.   Clicking on “Show All Versions” will drill down into the list of available versions.
• Look at the Download column in the search results.   This Download column contains direct download links for the most recent version of the artifact.   To download a matching artifact, just click on a link in this column.
• Select a search result, and you will see the artifact in the Repository Tree in the lower left-hand quadrant of this interface.   This is helpful to give you context for an artifact.   An artifact could be present in more than one repository.  If this is the case, click on the value next to “Viewing Repository” to switch between multiple matching repositories.
• In the lower right-hand quadrant of the interface, you will see a number of tabs which show information about the selected search result:
  • Maven Information: This contains basic identifiers and a snippet of XML you can use to add this artifact as a project dependency.
  • Archive Browser (Nexus Professional): This gives you a way to explore the contents of an archive in the repository.  You can view the files and folders contained in a matching search result.
  • Artifact Information: This tab contains timestamps, file size, checksum values, and a list of repositories containing a given artifact.
  • Artifact Metadata (Nexus Professional): This tab shows all of the built-in and custom metadata associated with an artifact.

This new search interface will be a part of the Nexus 1.7.2 search interface available in both Nexus Open Source and Nexus Professional.

 

We’ve received a steady stream of requests to release “Maven by Example” and “Maven: The Complete Reference” in the ePub format. Today, we’re announcing the availability of Maven by Example as an ePub. Download the Maven by Example ePub today and start learning about Maven.

This release has been tested on Apple’s iBook on both an iPad and iPhone and in the Adobe Digital Editions reader. To read this book on an iPad or iPhone: install the latest version of iTunes, download the ePub from Sonatype, and import the file using “Add to Library…” from the File menu.

We would appreciate any feedback you might have about formatting and layout. If you have questions, comments, or problem reports about our ePub books, send an email to book@sonatype.com. If you are reporting a typo or a problem with our ePub books, please make sure to include the device and software you are using as well as the section number, as page counts can vary depending on text size and device layout.

Stay tuned for “Maven: The Complete Reference” in an ePub format.

 

Another step towards a final release of Maven 3.0 was made the other day when version 3.0-beta-2 was released. I’ve been using Maven 3 since its alpha days, and despite the alpha/beta moniker, I find it to be superior to any Maven 2.x version. If you are starting a new project, I strongly recommend using Maven 3.

That being said, in the complete development environment there are a few missing pieces when working with Maven 3. One of these is getting Sonar to work with Maven 3, which had not been possible until the latest version of Sonar was released.

This post will explain how to get Maven 3 and Sonar to work together.

To get started, the first thing you must do is install Sonar 2.2 and Maven 3.0-beta-1 or later. No extra configuration of Sonar is required for Maven 3, and the same configuration of properties in Maven also applies. Please note that the URL for the Sonar server — sonar.host.url — must not end with a slash character.

The difference when using Maven 3 lies in the Maven plugin. Currently, the plugin at Codehaus Mojo doesn’t work with Maven 3, so you need to use the one at the Sonar project instead.

Now, in your Maven project, execute the command below.  Until the Sonar Maven plugin at Codehaus Mojo is updated with Maven 3 support, this is the command to use:

mvn org.codehaus.sonar:sonar-maven3-plugin:2.2:sonar

Your project should now be analyzed and the result uploaded to Sonar.

One thing to be aware of is that you need to align the version of sonar-maven3-plugin with the version of Sonar that is installed. When you upgrade Sonar, you need to change the version on the Maven side as well.

To view the progress of Maven 3 support in the plugin at Codehaus Mojo, follow MOJO-1545.

 

In the previous posts in this series, we’ve talked about how a repository manger changes the development cycle. In this post, I’m going to talk about how a repository manager can be used as a way to interact with third parties. Specifically, I’m going to talk about vendors and partners.

While Maven Central contains a huge number of libraries, there are still some pretty important things missing. Anything covered by a non-open-source license, like the Oracle JDBC drivers or anything proprietary, isn’t going to be made available from the Maven Central repository. Very often there are going to be commercial libraries that you will need to download and install in a third party repository. While this works, it requires some manual work to upload the artifact.

Vendors and Repository Management

When your vendors embrace repository management, there’s no reason why you would have to manually install any third party JARs in your corporate repository. Instead, vendors who need to provide binaries to customers would provide them via authenticated public repositories. These vendor-run repositories, protected by authentication credentials, would allow customers to synchronize local, corporate repository managers with commercial software vendors. This would allow vendors to capitalize on a new, “always connected” model for software delivery.

Now, the situation for the customer is simple. Once they set up an authenticated proxy repository to connect to a vendor’s repository, the vendor can deliver new updates and software directly to a customer’s repository.

The situation for the vendor is equally straightforward.  Instead of having to send the customer a packaged set of binaries, they simply tell the customer to update a version number in a dependency. The build will then interrogate the customer’s repository and then automatically download the repository from the vendor.

Both the customer and the vendor have fewer moving parts to worry about, and the task of delivering software becomes very simple.

The vendor receives another benefit from this setup. They are now able to keep track of which customers had requested which artifacts. The vendor can control which artifacts or features of a product suite a customer has access to, and they could use the repository manager as a way to enforce licensing or provide metered, on-demand access to software components.

Partners and Repository Management

In today’s increasingly connected world, it is very common for one company to provide an API for partners. Sites like Twitter and Google provide APIs for the general public, and other companies provide partner-specific APIs for collaboration. If you provide an API for a partner company, you can expose libraries and interfaces using a public-facing, authenticated repository.

Suppose that you work at a financial institution that needs to partner with another financial institution. The two institutions have agreed to collaborate with one another using a simple set of REST services. While these REST services are very well documented, you also want to make sure that your partner is invoking these REST services according to a set of standards. To make it easier to consume these services, you’ve created a simple client library for your partner.

To make it even easier for your partners to use this library, you’ve published versions of this library to a public-facing, authenticated repository. Using the security features of a modern repository manager, you can create partner-specific library versions and limit access to just the intended recipients. You can also maintain detailed audits of when a particular API was delivered to a particular partner.

As more and more organizations start to adopt the best practice of running a repository manager, these same organizations will be creating ad-hoc, private relationships to emulate the ease of Maven Central for a corporate environment. Partners, vendors, and consortia of companies will create repositories to facilitate code sharing and cooperation.

 

In the last post of this series we talked about how a repository manager, when coupled with a continuous integration server, allows you to work on individual modules in complex multi-module projects. In the post before that, we talked about the repository as something that enables greater collaboration between workgroups. In this post, I explore how using a repository manager simplifies deployment.

The following diagram shows the process of pushing code to production and automating a deployment:

This post will first discuss production deployments by setting up a hypothetical example. Assume that your organization runs a fairly complex application that consists of a few back office systems, a CRM system, and a web application. When your company needs to deploy one of these systems to production, the result is a highly orchestrated symphony of activity. Servers need to be taken offline in a coordinated manner across a distributed network of machines. Downtime needs to be planned and communicated to the right individuals, and backups need to be created before any new code is pushed to a production system.

In other words, any time you need to push new code to production, it is a real challenge to make sure that everyone is coordinating in just the right way. A few days before a production deployment, the development team freezes a particular branch and tags a release. This release is then installed in a QA environment. The QA team tests the release candidate, and if all goes well, this release is then deployed to production.

Without a repository manager, your operations teams has to know how to checkout a release tag from SCM, they have to run the entire build to generate a few binary artifacts, and then they need to write some scripts to automate a deployment to production.

With a repository manager, you can publish a release candidate binary to a hosted repository. This release candidate binary can then be used by a series of deployment scripts (or a tool like ControlTier or Puppet). You can have a deployment script that will publish the deployment to a QA environment, and then you can have a deployment script that takes the same, certified binary and publishes it to production.

The main difference between these two approaches is that the second approach adds some isolation between the SCM system and the deployment script. If your production deployment depends directly on your build, you have to recompile and repackage your entire system every time you do a deployment. This is often a challenge because the people that run the build system for deployment are seldom the same engineers that develop your system. If your operations team has to check out source code from an SCM just to build a production network, you are introducing some risk into your process.

Instead, use the repository manager as a way to share binaries with operations. If you are deploying applications to QA and production, having to rebuild an entire system from SCM just to deploy a system to an environment is an unnecessary risk. Instead of certifying tags in an SCM, your QA testers should be certifying binaries. This way you can be certain that the systems you are deploying to production have been tested.

 

The August edition of the Sonatype newsletter is now available.  Read all about Aether, Eclipse Day and other events we’ll be attending, and our upcoming webinar series on Build Promotion with Nexus Professional.

The Sonatype newsletter helps you catch up on everything we’ve been up to, and read about upcoming projects, releases and presentations from the Sonatype team.

If you want to get updates from the Sonatype team the moment news breaks and cannot wait until the next newsletter, subscribe to this Twitter feed:

• http://twitter.com/SonatypeCM

If  you would like to get on our mailing list, you can subscribe here.

 

JAX asked me some questions about Aether so I’m providing the the English version of the answers here for the community. The German version should show up on the JAX site shortly.

Can you give us an introduction to Aether?

Aether is a library for interacting with artifact repositories. This involves the specification of local repository formats, remote repository formats, workspaces, transports, and artifact resolution. People are generally familiar with repositories whether they be local or remote. Workspaces are additional sources where artifacts can be resolved from. Workspaces can be used in IDEs to provide resolution of projects you are working on, in shells like the Maven Shell or Roo, or any other long-lived process where a developer needs to resolve against in-development projects. I think people are familiar with various transports but HTTP is by far the dominant transport used with artifact repositories, but Aether lets you define additional ones if you need to. Along with all the rules to resolve artifacts taking into consideration any transformations, relocations, and conflict resolution strategies you might need to employ. We also plan to allow Aether to define version schemes, but the first work was just started on this by Alin Dreghiciu.

It is very important to note that Aether has no dependencies on Maven. When I said Aether is a library for interacting with artifact repositories, I didn’t mean Maven artifact repositories. Aether is a general purpose library for interacting with artifact repositories. If you wanted to specify your dependency metadata in a properties files Aether will let you do that. If you want to store your artifacts in a database Aether will let you do that. But, of course, we needed Aether to work for Maven so we created an implementation of what we call an ArtifactDescriptorReader to process Maven POMs. That implementation lives in the Maven codebase and that’s how we make Aether work for Maven.

Why is interoperability at the repository level, so important for Maven projects?

It’s more a question of interoperability between tools that interact with Maven repositories. Maven projects interoperate because they all use Maven. What we want to ensure is that projects building with SBT, Leinigen, Gradle, Buildr, Ant, or a homegrown system interoperate reliably with Maven repositories. Maven repositories are the defacto standard storehouses of JVM components. If a project that builds with Gradle can’t interoperate with a project built with Maven on the same machine because the local repository format that Ivy produces is not compatible then we see that as highly problematic. We want to prevent these problems, and we know that people creating these build tools would prefer to use as neutral a library as possible. Of course, to interoperate with Maven repositories integrators will require the ArtifactDescriptorReader to process POMs from remote repositories but they can create their own format for describing their dependency metadata.

At the project announcement, it is stated that “if compatibility with Maven repositories is important for your project then it’s not going to get any better than Aether.” Why is this?

If a project is using the Aether implementations that Maven itself is using then they are guaranteed to have the same behavior Maven does for resolving artifacts. If SBT is using the exact same code as Maven to read and process POMs, and the same local repository implementation then the level of interoperability is pretty much guaranteed. All the various tools may want to express their dependencies in their own particular way, but it would be bad if they interpreted POMs differently, did conflict resolution differently, or deployed artifact metadata incorrectly. These types of problems just wreak havoc for the average user trying to use artifacts from Maven repositories.

What is Aether’s relationship to Mercury and SAT4J library?

Mercury was our previous attempt at a library like Aether. We learned many things from Mercury, but it was a failed attempt because we tried to do too much at once. We tried to completely separate the resolution code from Maven, and introduce SAT4J at the same time. This proved to be a fatal mistake, but we have learned from our mistakes. Since Mercury we’ve brought Pascal Rapicault, the lead of the Eclipse p2 project, to the Sonatype team. Pascal is very familiar with SAT4J, but more importantly Pascal has a very good working relationship with Daniel Le Berre, who is the author of the SAT4J library. In talking with Pascal about our goals for Aether we concluded that we should design Aether to eventually allow a collaboration with SAT4J but not use it in our first pass. Our primary goal was to create a general purpose library that could be used in any tool including Maven. It’s also going to take more research and experimentation to understand exactly how we will integrate SAT4J. It’s not a question of if we will integrate SAT4J, it’s a question of when.

What is the current status of Aether Ant Tasks?

The Aether Ant Tasks are functional. They currently do the basic resolving, installing, and deploying. But there are some clarifications that we want to make in the syntax before we make an official release. The Aether Ant Tasks can be found in their own branch at Github. For those interested in discussing what the syntax should look like for the Aether Ant Tasks we encourage you to subscribe to the Aether development list. One short term goal with the Aether Ant Tasks is to provide a complete Ant-based solution for submitting artifacts to Maven Central.

We also intend to demonstrate how to make submissions to Maven Central programmatically using Aether so integrators can participate as freely in the exchange of JVM artifacts as Maven projects do now.

 

In the previous post in this series I discussed three compelling ways in which a repository manager can benefit the development cycle. It proxies artifacts locally, it is optimized to store binary artifacts, and it facilitates a new level of collaboration and agility that isn’t possible when your SCM is only way for workgroups to collaborate. In this post, I’m going to talk about how a repository manager works in concert with a continuous integration server like Hudson or Bamboo.

First, the how, what, and when of a continuous integration server. Continuous integration (CI) servers are an established fact of of modern development infrastructure. It is a server which, for the most part, waits and watches. It keeps a vigilant eye on your source control system and jumps into action every time it sees a code change. When code changes, your CI system is usually configured to run the entire build, execute all of your unit and integration tests, and send out an email to every developer if it identifies a defect or a failed test.

It does this so that you will have an easier time identifying where a particular problem was introduced to the source code. If John checks in some bad code, the CI system runs the build immediately, and about 30 minutes later, everyone in the group receives an email with the subject header “John just broke the build”. It is a great way to identify errors, and it is also a great way to motivate developers to test locally before committing to a source control system as no one likes to be the reason for a build failure email.

Running a CI server is more than “just a good idea”. Once your system reaches a certain level of complexity you can’t scale a system without commiting to continuous integration and testing. If you don’t have continuous integration, you end up having to put all development on hold each time you want to perform a release. If you don’t build, test, and deploy your system on a regular basis – if it isn’t something that is well rehearsed, integration becomes a time consuming nightmare of manual testing and builds that often leads to inconsistent builds. This is especially true if your development effort spans multiple systems and multiple development workgroups. You run a CI system because building, testing, and deploying your system should be automatic: it should be as trivial as pressing a button.

The concept of a CI server is only slightly more established than a repository manager, and very often you will see that an organization has identified the need for a CI server before they’ve identified the need for a repository manager. If you are coding a complex system, there is a very good chance that you are already running a CI server. The most popular servers out there are Hudson, Bamboo, and CruiseControl. While the connection between CI servers and repository managers might not be immediately obvious, when used together they can introduce some new possibilities for the way you develop your systems.

Continuous Publishing

When you have a system to continuously build your code, you also have a system that can continuously publish SNAPSHOT artifacts to a repository manager to enable a more granular approach to development. What do I mean by “a more granular approach to development”? To answer that question, let’s take a look at a complex multi-module project using the example of the eCommerce group from the previous post in this series.

Assume you have a new programmer starting tomorrow. Instead of throwing him at the entire forty-thousand lines of code, you would like to be able to give that developer a small, easy to digest task. You want this developer to add support for PayPal’s Adaptive Payments API in your eCommerce system. That’s it. You don’t want them to be distracted by the overwhelming scope of the project, and you certainly can’t afford for them to take a three month voyage through your project’s code before they start contributing to the effort. Deadlines are tight, and you don’t have enough people on your team. It is important that new hires start programming as soon as they walk in the door.

Without a repository manager hooked up to a continuous integration server, if you try to checkout just the ecom-paypal project, the build is going to fail because it will try to download dependencies from a repository manager. In the case of the ecom-paypal project, assume that the dependency graph looks like this.

When you have a repository manager and a continuous integration server, you can configure your continuous integration server to publish SNAPSHOT artifacts (in-progress SNAPSHOT binaries) to your repository manager. This will allow you to just check out a single, isolated portion of a much larger multi-module project.

Without a repository manager, trying to build version 1.3-SNAPSHOT of ecom-paypal in isolation is going to generate errors because you are forced to checkout the entire codebase to build and install all of the dependencies in your local repository. With a repository manager, SNAPSHOT artifacts are being continuously published because Hudson is checking you SCM every few minutes and building the latest code. When you run the ecom-paypal module’s build in isolation, Maven is going to download the most recent SNAPSHOT.

Without a repository manager, your new developer is going to have to download the entire codebase and run a large time-consuming build. With a repository manager you can work on specific components of a larger multi-module project. This ability to divide and conquer your codebase comes in very handy when you need a consultant to take a look at a specific problem, or when you need to look at a coding problem in isolation.

When you continuously publish build artifacts to a repository manager, you move away from the single monolithic project build and toward a project layout and architecture that lends itself to modularization.

In tomorrow’s post: How a Repository Manager decouples deployments from source code, and what that means for developer operations.

 

In the first part of this series, I gave you a glimpse at the big picture of repository management, and I listed some common anti-patterns present in systems that haven’t yet installed a repository manager. In this post, I’m going to focused on the benefits a repository manager has on the development cycle. How does using a repository manager change the way your developers will approach development? What problems does it solve? And what possibilities does it introduce?

Caching artifacts locally

The first, and most obvious, benefit is that a repository manager will proxy and cache artifacts from a remote repository. If you use Maven or another tool that can download artifacts from Maven Central, you have probably had the experience of checking out a large build from source control and running it, only to have to wait for minutes while the build downloads dependencies from Maven Central.  An even bigger frustration than that, if your connection to the internet is tenuous, or if Central happens to be unavailable for whatever reason, you cannot build your code.

With a repository manager, downloading your dependencies takes much less time and your builds don’t rely on internet access. Unless this is the first time building a particular project, all of the artifacts your build uses are going to be cached at the repository manager. Instead of waiting for artifacts to be downloaded over the public internet, your build will download artifacts from a local server. Since you avoid the public internet, this process is sped up significantly. The build that once took 15 minutes to download dependencies will now take less than a minute to download everything it needs.

Get those JARs out of Subversion

The Oracle JDBC driver, a proprietary JAR from a vendor, is the kind of one-off, third party library that is not going to made available on a public repository. Without a repository, the most obvious solution is to just check these JARs into source control and store them right next to your code. When you run your build, branch your project, and release your code you are just passing around these binaries as a part of your project. While this approach does work, you are overloading the source control system to perform a task it wasn’t designed for.

While Subversion, and many other modern SCM tools, will gladly version binaries, you are storing a static, unchanging artifact in source control. Every time you checkout out source control, you are checking out binaries, and if you are unlucky enough to work somewhere that stores all libraries in source control, you will often be asked to check out a large repository of JARs. Your SCM is going to carefully keep track of changes and version files that will never change.

This is both inefficient and unnecessary. If you use a repository manager, you can upload third party libraries to a third party repository that stores artifacts on the repository manager. Then you can mix these third party libraries into a public repository group that will combine the contents of public repositories with your own repositories. In other words, you can declare a dependency on a custom, manually uploaded third party JAR as easily as you would declare a dependency on a library stored in Maven Central. Don’t store binaries in source control, use a repository manager instead.

Collaboration

Collaboration is best illustrated with a hypothetical organization. Imagine you work at an organization with three large development groups of about 30 developers. There is an eCommerce group, which is responsible for writing systems that interact with banks and service providers like PayPal. There is a CRM group, which has the unenviable job of merging two disparate CRM solutions from Peoplesoft and Oracle and wrapping those systems in an easy to use Java API. Then there is a third group, the web applications group, which has the primary responsibility for wrapping these two back office systems in a slick web interface. These relationships are captured in the following diagram:

Imagine that the eCommerce group and the CRM group have to assemble the entire external API into a single project: ecommerce-api and crm-api. These APIs contain all of the logic required to connect to a set of internal services hosted by each group. As the eCommerce group and CRM groups innovate and offer new services, they will cut releases of these new APIs and publish those releases to the repository manager.

When the web applications group is ready to start using a new version of the eCommerce or CRM API they can simply define a dependency on a version of this artifact. The build for the web applications group will then download the appropriate version of the eCommerce of CRM API from the corporate repository manager.

In this way, the repository manager is a central collaboration point for difference groups within the same company. Without the use a repository manager, you will often see disparate groups forced to check each others code out of SCM and build it from scratch just to generate client libraries. In the organization that has embraced repository management, these libraries are published as binaries for clients (in this case other workgroups) to consume from the repository manager.

When you collaborate using the SCM, it is very difficult to scale. Every workgroup is forced to use the same set of tools for build management and you will very often see the entire IT operation having to synchronize releases. When you share build artifacts using the repository manager, you decouple workgroups from one another and allow your internal teams to collaborate using a more adaptive, open-source model.

If the eCommerce group needs to innovate, they can do so without affecting the source code of the web applications group. They can publish new releases to the repository manager, and the web applications group can decide when and how they are going to consume these releases by changing the version of a dependency in a build.

When you collaborate using the SCM, all of your workgroups are joined at the hip by a common codebase. When you use a repository manager you allow different workgroups to innovate and create at their own pace.

In next week’s posts: Continuous Deployment and Integration

 

Whenever I speak to someone doing Java development, I always ask if they are using a repository manager. Repository managers are still an emerging technology, but I’ve noticed a consistent trend: more and more developers view a repository manager as an essential part of development infrastructure. This certainly wasn’t the case just two years ago, and I think that the big motivator behind this trend is that the quality and stability of Maven Central has improved remarkably because of the efforts of people like Brian Fox and others who are focused on making the service more stable.

Another reason why we’ve seen more adoption is that most developers understand the benefits of using a tool like Maven for automatic dependency management. In 2005, it was common to see projects store binary JARs alongside source code in projects. In 2010, you rely on the repository and the metadata it contains. If you use a library like Guice, you’ll add a dependency on the artifact and let your build tool take care of the details. To do otherwise would be to commit yourself to a manual work updating JARs and testing dependencies each time a new version of an external library is released.

Despite the increasing prevalence of repository managers, I still stumble upon workgroups and organizations that haven’t heard of repository management. When you ask if they are using a repository manager, they might think you are referring to Subversion or source control. This series of posts is a high-level overview of the main benefits of repository management. If you are trying to convince someone to start using a repository manager, the next few blog posts are for you.

Repository Management: The Big Picture

Compare the diagram shown above with the diagram shown below. In the next few posts, I am going to emphasize the specific benefits of using a repository manager. Specifically, I’m going to talk about:

• How a repository manager changes the development cycle
• How continuous integration is used to continuously publish internal build artifacts
• How a repository manager simplifies the process of building and deploying systems to production
• How a repository manager can act as a gateway between vendors and external partners

When you don’t use a Repository Manager

Before I get started on the benefits of repository management, I want to talk about the realities you face when you don’t use a repository manager. Here are some common anti-patterns when you don’t use a repository manager:

• All of your developers download artifacts directly from public repositories. A new developer starts on a Monday. That developer will spend an hour downloading a massive library of dependencies from Maven Central. Worse, if Maven Central happens to be down that day, they will be out of luck entirely.
• Proprietary or Vendors libraries are passed around, from developer to developer. If you don’t use a repository manager, how do you distribute the Oracle JDBC driver? Maybe you place it in a shared file system and tell people to download it and install it in ~/.m2/repository. More likely, developers just pass this JAR around as an email attachment with some ad-hoc instructions.
• JARs are checked into source control. If you don’t use a tool like Maven, which knows how to download artifacts from a remote repository, you might be following the very common pattern of checking binary dependencies and libraries into source control. I’ve seen many instances of companies creating ad-hoc JAR repositories and checking these repositories into source control, only to version and branch these static binary files with every release.
• The source control repository is used to store everything from source code to binary builds. Because there is no repository designed to store binaries, developers start to use tools like Subversion to keep track of binaries. As time passes, the Subversion repository becomes an ad-hoc file system for files that have no business in an SCM.
• The continuous integration server depends on public repositories. When you change your build or add a new dependency, your CI system downloads dependencies from the public repo. It depends on the availability of this public resource to run builds.
• Production deployments have to run the entire build, from start to finish, to generate binaries for deployment. When a build is tested and then ultimately pushed to production, the build and deployment scripts checkout source code, run the build, and deploy the resulting binaries to production systems.
• Sharing source code with external partners means granting them access to your SCM.  Since there is no established mechanism for publishing source or binary artifacts, the only way to share code with partners is to either send an archive of your source, or provide them with direct access to your SCM.

The general theme in all of these anti-patterns is that either your systems depend on public resources, or they all depend on the SCM system as a central collaboration point. In the next few posts, I’m going to detail how using a repository manager provides a solution for each of these issues. I’ll go into why each of these anti-patterns is a bad idea, and how you can use Maven, Nexus, and Hudson together to solve these problems and create a more efficient software development effort.

Stay tuned for the next post: Caching and Collaborating.

 

Introducing Aether

Aether (pronounced ē’thər, as in flying though the ether) aims to be the standard library for interacting with Maven repositories. Without question, the most important aspect of the Maven ecosystem is interoperability at the repository level. There are many emerging options in the build space for JVM-based languages, but the one thing they all have in common is their interaction with Maven repositories. It’s clear that Maven repositories play a critical role within JVM-based development infrastructures and Aether will provide the necessary interoperability, through a common set of tools and APIs, that’s critical for happy users in the ecosystem.

Aether & Maven 3.x

Benjamin Bentmann, Sonatype’s lead on Maven 3.x, has fully integrated Aether back into Maven 3.x. Though Aether is an extraction from Maven, Aether is not Maven specific. That said Maven 3.x and Maven repositories are our first priorities. The upshot is that if you embed Aether, you will be embedding the same library that Maven 3.x uses when it interacts with a Maven repository. If compatibility with Maven repositories is important for your project then it’s not going to get any better than Aether.

Aether & Mercury

Mercury was our first attempt at extracting a Repository API from Maven and that didn’t work. We bit off more then we could chew, and so our first attempt was a learning experience. We attempted to integrate the excellent SAT4J library but didn’t have enough experience to pull that off. In addition there may be features of SAT4J Maven requires that are not present yet. As such, we felt it better to attempt something less ambitious, yet functional, and invest in research for future improvements.

Aether, p2 & SAT4J

As part that investment, we will be collaborating in sponsored research with Daniel Le Berre, who is the author of SAT4J. Daniel is a tenured professor at University of Lens, and researcher at Centre de Recherche en Informatique de Lens and is affiliated with Centre National de la Recherche Scientifique. His work will not only help us with p2, but also with many things we want to accomplish in Maven. We see a lot of overlap in the work Sonatype is doing on Maven and p2 and ultimately we really see there being one artifact resolving framework.

Aether & Ant

Rest assured there will be excellent Aether Ant Tasks. Making sure that Aether works flawlessly inside Maven 3.x is our first priority, but we are committed to making a set of excellent Aether Ant Tasks. If you’re interested in Aether Ant Tasks you can follow the progress here.

Aether Resources

If you are interested in Aether here are the goods:

• Aether Sources @ Github
• Aether Issue Tracking @ Sonatype
• Aether Wiki @ Sonatype
• Mailing lists:
  • Aether Users @ sonatype.org
  • Aether Developers @ sonatype.org
  • Aether SCM @ sonatype.org

P.S. Thanks to Pierre-A Grégoire for the name Aether!