Plans for OpenJDK

Our team at Red Hat has been doing some planning now that OpenJDK has been released. First, a summary of OpenJDK’s status:

What Sun has released:

  • most of the 1.6-level JDK, including:
  • the Hotspot virtual machine with x86 and x86_64 JITs
  • the Java compiler
  • most of the class libraries

What Sun hasn’t released:

  • 4% of the class library code — some of the class library code is “encumbered” meaning that Sun can’t release it under the GPL. The encumbrances are in the sound system, font rasterizer, graphics rasterizer, SNMP and crypto areas.
  • plugin and javaws
  • a 1.5 version

The second list points the way to some of our immediate goals. Here is a list of some of the goals we decided on:

  1. Packaging OpenJDK as IcedTea

    We can’t call what we ship “OpenJDK” because that’s a Sun trademark. We’ve decided on the temporary name “IcedTea”, and we can say that it’s based on code from openjdk.java.net. We’ve posted an SRPM containing the free OpenJDK sources. But it still requires non-free stuff to build, which brings us to the next goal.

  2. Building with free software

    Currently OpenJDK requires openmotif, Sun’s proprietary JDK 1.6 and a bundle of proprietary “plugs” (to replace the encumbrances) to build. Our group’s first goal is to make OpenJDK buildable against a set of fully-free packages.

  3. Setting up icedtea.classpath.org

    With help from some GNU Classpath community members we’re setting up icedtea.classpath.org with a Wiki, mailing list and a Subversion repository, to host the work we’ll be doing on OpenJDK. This is a temporary site where we can experiment with the OpenJDK code while Sun works out its contribution processes. The site is not complete yet.

  4. Testing without the encumbrances

    Because we haven’t built OpenJDK without the binary plugs for the encumbrances yet, we don’t know exactly how it fails if they’re not present. We need to see if the result can be proposed for inclusion in Fedora.

  5. Replacing encumbered bits with parts from GNU Classpath

    Once we’ve tested running without the encumbrances we’ll know where to focus our replacement efforts.

  6. Testing OpenJDK interpreter on other architectures

    Since at this point we only have x86 and x86_64 JITs, platform coverage will be a problem. In the short term we should test the OpenJDK interpreter portability since it may be possible to use it as a stop-gap on the architectures for which we don’t have JITs.

  7. Working with Sun on contributor license agreements, use of trademarks, self-certification and governance.

    Eventually, we’d like to ship in Fedora a certified Java implementation that we can call OpenJDK and “Java compatible”. Sun intends to make that possible through a self-certification process. Eventually we should be able to contribute code directly to the OpenJDK project — Sun has worked out the governance model it will use and is implementing a Mercurial repository.

  8. Working with other Red Hat groups on OpenJDK

    We expect to work closely with other Red Hat groups on both system integration (kernel, tools, desktop, packaging) and on requests from internal “consumers” of the JDK (JBoss, Eclipse).

GCJ

We will continue to support GCJ where it is deployed, but any new work we do will be on the OpenJDK codebase. GCJ still has better architecture coverage than OpenJDK, supporting all 7 RHEL architectures where OpenJDK supports 2, but we’ve decided that the benefits of OpenJDK outweigh this advantage of GCJ, and that effort is better spent trying to get (or write, if need be) more JITs for OpenJDK.

Timelines

We don’t have definite timelines yet. We’re in an investigation stage right now, so precise timelines will have to wait until we’ve had more time to discuss the OpenJDK codebase and Sun’s timelines.

IcedTea

I packaged the new code drop from the OpenJDK project using the JPackage conventions. Because naming rules are still being worked out for derivatives of OpenJDK code, I’ve decided to call the result “java-1.7.0-icedtea”. This is just a preliminary development package; I hope Sun will approve renaming it “java-1.7.0-openjdk” or “java-1.6.0-openjdk” when it officially lands in Fedora.

http://people.redhat.com/fitzsim/java-1.7.0-icedtea-1.7.0.0-0.1.nonfree.src.rpm

The OpenJDK build still depends on some non-free software so I’m providing an SRPM containing only the GPL’d code. People interested in building the complete not-fully-free OpenJDK can fill the gaps with binary plugs.

I’m going to investigate eliminating the non-free requirements. I haven’t studied the code enough to know if it’s worth producing a fully free build before starting encumbrance replacement efforts — it depends how much functionality is lost by excluding the binary blobs. Another interesting experiment would be bootstrapping OpenJDK with a free runtime.

Java Modules in Fedora

Andrew Overholt recently pointed me to a blog entry on Java module systems. I can’t comment in detail on the JSRs or how they should interact because they’re all behind click-through licenses. My instinct tells me the ideal answers are: “Sun, drop JSR 277 and adopt OSGi; IBM, develop OSGi within the Java tree, on the same schedule as Java”, though this may be politically impossible.

Beyond politics, the post got me thinking about how a Java module system should integrate with Fedora. Here is a sketch of how I think it should work.

OSGi Modules in Fedora

OSGi modules distributed with Fedora will need to be wrapped by RPMs, so that they can be built, released and managed using existing tools and processes.

rpmbuild Integration

OSGi and RPM metadata provide similar information, so the adoption of OSGi will simplify the creation of Java RPMs. The packager should only be required to list the proper build dependencies in a spec file; Requires and Provides should be auto-calculated by rpmbuild. This is how C module (DSO) RPMs are currently produced.

The reverse operation should also be supported, for transforming non-modularized upstream projects into OSGi modules at rpmbuild time, using packager-provided metadata (the Name, Version, Requires and Provides fields). Then we could easily convert our currently-shipping Java packages to OSGi modules. At the same time, we should encourage upstream projects to properly modularize their jars, including any build dependency jars they bundle.

rpmbuild should automatically namespace packager-built modules with a Fedora release prefix, to distinguish them from the equivalent upstream module.

Module Loader Customizations

The Fedora OSGi loader should prioritize these Fedora-namespaced modules over non-Fedora modules with the same version.

yum Integration

Likewise, the OSGi updater should compare upstream modules to those available in Fedora, and where versions match, pull the Fedora module. There will need to be an OSGi-yum bridge so that the OSGi updater can request module metadata from yum, and instruct it which package to install. The required yum interfaces already exist to support update programs like pirut and pup. For dependencies not resolved in the yum repositories, the OSGi updater should fall back to its list of jar repositories.

Upstream Integration

The biggest improvement to the current Java packaging situation may well come from upstream projects. Once a module system becomes part of the Java standard, I hope to see upstream projects adopt it in a disciplined way. If upstream projects started bundling modules instead of just jars, packaging these projects would become simpler. The problem has been that it’s unclear which version of a particular jar is bundled with a project’s source — a module answers that question, and helps the packager set up BuildRequires. If this versioning convention becomes common, the BuildRequires generation step could even be automated by a “make new-sources-java” target in Makefile.common.

Projects also bundle dependencies in binary jars, for deployment on systems lacking native package management. With upstream module versioning, Fedora namespacing, and the Fedora module loader customizations, system-installed modules, where available, would be used in favour of the equivalent bundled modules. This should be the desired behaviour, since system administrators should trust Fedora Java packages more than the upstream binaires.

rpm Integration

Along these lines, system administrators may be interested in performing operations on the set of all system-installed OSGi packages. There is talk of adding tagging support to rpm and yum; once it is in place, we should apply a common tag to all packages that contain OSGi modules.

OSGi versus JSR 277

I used OSGi as the example module system because I’m familiar with it, but the above sketch would probably apply equally well to JSR 277. However, to complete the usefulness of a module system, the system Java environment must recognize it. If Fedora wanted to use OSGi system-wide (not just within Eclipse) then the system Java’s behaviour would have to deviate (irresponsibly) from the Java standard, to recognize and resolve OSGi modules. So while our packages can all be made “OSGi-ready” now, only Eclipse would see the benefits.

Since not deviating from the Java standard is more important than the benefits of modules, whichever module system is integrated into Java itself will become the Fedora-wide Java module system. Once a decision is made, we can start implementing the integration improvements I’ve sketched out.

FOSDEM 2007

Tom Marble is a wizard at getting the right people in a room together. In the Distro DevJam we had Java packagers from Debian, Ubuntu, Fedora, Red Hat, OpenSUSE, SUSE, Mandriva, Gentoo and FreeBSD, together with all the right people from Sun. This was the perfect group for discussing packaging issues surrounding OpenJDK (and a great audience for my talk).

My feeling is that there is now a general consensus among the distros about how to solve the technical issues of packaging OpenJDK. The remaining tricky issues are procedural:

1) creating a good OpenJDK contributor license agreement that works for free software developers
2) giving distros the ability to self-certify
3) making the JCP open to free software contributors (e.g., no click-throughs on specs)

We discussed 2 and 3 in the DevJam but it was too soon to talk about solutions. I hinted at 1 in my slides with “accepts most patches upstream”, but in retrospect I feel I should have emphasized the importance of a good CLA when seeking copyright assignment from external free software contributors. Anyway, given the discussion in the room, I feel that these problems will eventually be solved but that some patience will be required.

Sun’s showing at FOSDEM was very impressive. The presentations were helpful in that they provided a preview of Sun’s development processes. Because I’ve worked on GNU Classpath’s AWT, I was especially glad to meet Igor Nekrestyanov, who is working on the Sun Java2D team to replace the class library encumbrances.

Already as a result of our FOSDEM meetings we’re seeing opportunities for collaboration. External contributors are still in an awkward position until the class library code drop actually happens, but in the meantime there are opportunities to contribute to third-party libraries that Sun is using to replace the encumbrances. Along these lines, I learned at FOSDEM that the Java plugin will not be released with the initial code drop. Maybe gcjwebplugin can be adapted to act as a temporary stop-gap.

One non-Java FOSDEM observation: the Fedora booth was excellent, complete with Fedora flip-flop/boating-key-chain giveaways.

The New Free Java Project

Surprise

I was happily surprised on Monday when Sun announced that they would
release Java ME, SE and EE under the GPL. Actually, “happily
surprised” is an understatement: Sun also adopted the GNU Classpath
linking exception! I couldn’t have hoped for a better result.

Watching the announcement webcast was a surreal experience. It was as
if I were dreaming. Rich Green standing in front of the giant “GPLv2”
slide. The pre-recorded statement of praise from RMS. Jonathan
Schwartz pointedly asking Rich Green if OpenSolaris would be GPL’d.
The Java world was upside down. The FSF-versus-Sun standoff
evaporated before my eyes. Even on Friday as the GPL rumours
swirled, I really couldn’t have imagined this event. Rarely does a
long-standing conflict end so swiftly and constructively.

Lessons Learned

Over the past few months I was tempted to be cynical about the pending
announcement, but I really tried to keep an open mind. I learned this
mindset from Mark Wielaard and Dalibor Topic, two exemplar free
software advocates. They were very patient, polite, and helpful with
people from Sun who were contemplating the licensing decision. Mark
and Dalibor’s diplomacy almost certainly had a hand in Sun’s choice of
license.

The New Free Java Project

Given the excellence of the announcement itself, complete with FSF
endorsements, an initial code drop, open-subscription mailing lists,
and an exhaustively comprehensive FAQ, I’m confident that Sun
will run the free software Java project as competently as they’ve
stewarded the Java language. And in an abstract sense, the license
choice is a declaration from Sun, a contract with the free software
community, that they will continue the project honestly and openly.

My Personal Reaction

The dream-like aura surrounding Monday’s webcast was probably stronger
for me because I’ve been working on libgcj and GNU Classpath for the
last three years. For all that work to result in a complete victory
is a wonderful feeling. There is also a tinge of sadness though, that
the GNU Classpath project, along with all or most of the code I wrote
for it, will likely be retired in the coming year. I think GNU
Classpath has become a successful free software martyr. (Although
parts of the project may rise again in zombie form: could there be a
place for a GTK AWT peerset in the new Java world? ;-)) The GNU
Classpath community is great to work in, and I hope our ties remain
strong. I especially look forward to FOSDEM 2007 where I expect
everyone, even man-di, to be present for a huge party! 🙂

Thoughts on Fedora Core 6

gcjwebplugin

My main goal for Fedora Core 6 was the inclusion of gcjwebplugin, so I’m very pleased that it will in fact be included. Though it is not enabled by default, having it in the base distribution means that interested people can experiment with it and file bug reports, with the minimum of configuration. We’ve already received some bug reports as a result. This is exactly the level of adoption I had hoped for in this release. I hope others will try their favourite applets and file bug reports if they don’t work, since doing so helps direct our gcjwebplugin development efforts.

In the meantime, work continues on GNU Classpath’s security implementation, so we should be able to safely run untrusted applets in Fedora Core 7.

Looking over my Fedora Core 5 Wishlist I see that many of my FC-5 wishes have been realized in FC-6.

Merging
Most of the proposed merging work happened during the FC-6 cycle. The following projects were merged into GNU Classpath:

  • GNU Crypto
  • Jessie
  • cp-tools
  • gcjwebplugin

The only straggler is gjdoc. In a short period of time we’ve managed to merge all these separate but essential projects into one coherent whole in GNU Classpath. This is good for everyone: VM projects that bundle GNU Classpath (Kaffe, libgcj, IKVM.NET, CACAO), who now all have access to the same tools; distributions, who now have fewer packages to maintain; and free Java developers, who now have a single, central location in which they can fix bugs in the class library and tool chain. I’m especially happy about the current tools situation, because before the merging work, VM projects each wasted effort on their own implementations of some tools; now we can all focus on making the GNU Classpath tools the definitive best set.

Applications and Applets

  • MegaMek works on libgcj, and is packaged and awaiting approval for inclusion in Fedora Extras
  • Azureus works on libgcj and is in Fedora Extras
  • many applets work on gcjwebplugin

AWT and Swing

The constant stream of AWT and Swing fixes and improvements in GNU Classpath means that many more AWT and Swing applications and applets work on libgcj. One large improvement in Fedora Core 6 is that the Cairo Graphics2D backend is now the default. Though GNU Classpath’s Graphics2D implementation is still being actively developed, the snapshot in FC-6 can run much of Sun’s Java2D demo:

screenshot of Sun's Java2D demo running on FC-6 gij

Fedora Core 5

Installation and Setup

I installed Fedora Core 5 on my IBM ThinkPad T41 last night. I’ll go through a few minor annoyances before the good news:

– NetworkManager wasn’t the default and I couldn’t find a way to enable it through the GUI (not that I should have to explicitly enable it anyway). I ended up dropping to the terminal and running it manually. Bug 171348

– After I got wireless networking working I wanted to be able to click on an MP3 link in Firefox and have the MP3 play automatically. So I installed the GStreamer plugins. When I clicked on an MP3, HelixPlayer was the selected default. But HelixPlayer started up and threw up its usual “format not supported… download?” dialog. So I uninstalled it. The next default was Rhythmbox. It started up but didn’t play anything automatically, and I couldn’t figure out how to play the file manually. So I uninstalled it. The next default was Totem, which worked perfectly. Bug 154392

There are 36 open bugs for Rhythmbox in Fedora Bugzilla but none that describes what I was seeing. I’ll have to file a new one.

– I tried to install the Eclipse and Java Development groups but Pup failed with “no more mirrors to try”. Bug 184326

The servers were probably busy last night so I’ll try again later.

Various media types weren’t supported out-of-the-box, obviously, but I installed the “GStreamer Universe” from GStreamer’s excellent FC5 web page and was very happy with the results. Most media types worked perfectly in Totem.

Installing the GStreamer repo files could have been smoother. It would be nice if pirut were the default handler for .repo files. Bug 186112

Impressions

Apart from these minor annoyances I’m very happy with this release. The new installer worked well, bootup feels much faster, all my laptop’s hardware worked out-of-the-box, sleep-on-lid-close works, NetworkManager eliminates manual network management, a few simple steps allow GStreamer to provide good media support. I’m very pleased that release-by-release I require fewer and fewer hacks to make Fedora into a comfortable, complete environment. In most cases in the past, these hacks involved installing unintegrated proprietary software.

Fedora Core 6

There are still places in Fedora Core 5 where such hacks are required. Three that stand out are:

– playing web-page-embedded audio and video

– Flash

– Java applets

I’m hoping that the free equivalents (GStreamer, Gnash and gcjwebplugin) will eliminate the need for these hacks in Fedora Core 6.

Fedora Core 5 Wishlist

Here’s my categorized list of goals for Fedora Core 5.

java-gcj-compat

  • GNU Crypto fix for Eclipse extssh support

    Casey Marshall has already committed a Diffie-Hellman JCE provider to GNU Classpath so this is just a matter of testing.

  • import all JAWT fixes

    All the necessary patches are already in GNU Classpath and libgcj.

  • Jessie merged into GNU Classpath
  • GNU Crypto core algorithms merged into GNU Classpath
  • gjdoc merged into GNU Classpath enabling …
  • an --enable-javadocs option to libgcj
  • make gcjx java-gcj-compat’s javac commands

    Not sure if this one is doable in time; I don’t mean full gcjx, just the bytecode-compiler portion in a standalone executable.

  • include Tritonus or some other sound implementation

AWT

  • ImageMagick backend for ImageIO
  • Graphics/Imaging refactoring for GTK 2.8 and Cairo
  • MegaMek packages in Fedora Extras

    This just involves a few AWT bug fixes now.

  • fix all 1.5 japi issues in java.awt.* packages
  • fix all 1.5 japi issues in javax.imageio.* packages

Swing

  • RHDB tools in Fedora Extras: Administrator, Visual Explain and Control Center.

    This will require updating these tools to support PostgreSQL 8.0.

  • Limewire in Fedora Extras

    It would be absolutely awesome to have this Swing app working on the free stack but I’m not sure it’s doable in the FC5 time frame. Would anyone like to undertake it?

SWT

  • Azureus in Fedora Extras

    Again, not sure if this is doable before FC5 but this would be awesome to have.

gcjwebplugin

  • implement and test security features in GNU Classpath to allow running untrusted bytecode

    It’s very unlikely that this one will be finished before FC5 but I’d like to at least have started the implementation by then.

I’m hoping this list will inspire some volunteers, especially for the big apps and GNU Classpath’s security framework. Most of these items are very doable and will greatly improve the free JPackage stack on Fedora.

AWT Progress

gcjappletviewer and gcjwebplugin

More and more applets are working with gcjappletviewer and gcjwebplugin. Here’s a screenshot of gcjappletviewer HEAD + GNU Classpath HEAD:

gcjappletviewer running a Sudoku applet

and one of gcjwebplugin HEAD + GNU Classpath HEAD:

gcjwebplugin running an aviation data applet

I’m getting through my AWT bug list. My next big chunk of work will be completing the transition to GTK 2.8 and Cairo, removing our GdkGraphics class and making GdkGraphics2D the default.