IcedTea

I packaged the new code drop from the OpenJDK project using the JPackage conventions. Because naming rules are still being worked out for derivatives of OpenJDK code, I’ve decided to call the result “java-1.7.0-icedtea”. This is just a preliminary development package; I hope Sun will approve renaming it “java-1.7.0-openjdk” or “java-1.6.0-openjdk” when it officially lands in Fedora.

http://people.redhat.com/fitzsim/java-1.7.0-icedtea-1.7.0.0-0.1.nonfree.src.rpm

The OpenJDK build still depends on some non-free software so I’m providing an SRPM containing only the GPL’d code. People interested in building the complete not-fully-free OpenJDK can fill the gaps with binary plugs.

I’m going to investigate eliminating the non-free requirements. I haven’t studied the code enough to know if it’s worth producing a fully free build before starting encumbrance replacement efforts — it depends how much functionality is lost by excluding the binary blobs. Another interesting experiment would be bootstrapping OpenJDK with a free runtime.

Java Modules in Fedora

Andrew Overholt recently pointed me to a blog entry on Java module systems. I can’t comment in detail on the JSRs or how they should interact because they’re all behind click-through licenses. My instinct tells me the ideal answers are: “Sun, drop JSR 277 and adopt OSGi; IBM, develop OSGi within the Java tree, on the same schedule as Java”, though this may be politically impossible.

Beyond politics, the post got me thinking about how a Java module system should integrate with Fedora. Here is a sketch of how I think it should work.

OSGi Modules in Fedora

OSGi modules distributed with Fedora will need to be wrapped by RPMs, so that they can be built, released and managed using existing tools and processes.

rpmbuild Integration

OSGi and RPM metadata provide similar information, so the adoption of OSGi will simplify the creation of Java RPMs. The packager should only be required to list the proper build dependencies in a spec file; Requires and Provides should be auto-calculated by rpmbuild. This is how C module (DSO) RPMs are currently produced.

The reverse operation should also be supported, for transforming non-modularized upstream projects into OSGi modules at rpmbuild time, using packager-provided metadata (the Name, Version, Requires and Provides fields). Then we could easily convert our currently-shipping Java packages to OSGi modules. At the same time, we should encourage upstream projects to properly modularize their jars, including any build dependency jars they bundle.

rpmbuild should automatically namespace packager-built modules with a Fedora release prefix, to distinguish them from the equivalent upstream module.

Module Loader Customizations

The Fedora OSGi loader should prioritize these Fedora-namespaced modules over non-Fedora modules with the same version.

yum Integration

Likewise, the OSGi updater should compare upstream modules to those available in Fedora, and where versions match, pull the Fedora module. There will need to be an OSGi-yum bridge so that the OSGi updater can request module metadata from yum, and instruct it which package to install. The required yum interfaces already exist to support update programs like pirut and pup. For dependencies not resolved in the yum repositories, the OSGi updater should fall back to its list of jar repositories.

Upstream Integration

The biggest improvement to the current Java packaging situation may well come from upstream projects. Once a module system becomes part of the Java standard, I hope to see upstream projects adopt it in a disciplined way. If upstream projects started bundling modules instead of just jars, packaging these projects would become simpler. The problem has been that it’s unclear which version of a particular jar is bundled with a project’s source — a module answers that question, and helps the packager set up BuildRequires. If this versioning convention becomes common, the BuildRequires generation step could even be automated by a “make new-sources-java” target in Makefile.common.

Projects also bundle dependencies in binary jars, for deployment on systems lacking native package management. With upstream module versioning, Fedora namespacing, and the Fedora module loader customizations, system-installed modules, where available, would be used in favour of the equivalent bundled modules. This should be the desired behaviour, since system administrators should trust Fedora Java packages more than the upstream binaires.

rpm Integration

Along these lines, system administrators may be interested in performing operations on the set of all system-installed OSGi packages. There is talk of adding tagging support to rpm and yum; once it is in place, we should apply a common tag to all packages that contain OSGi modules.

OSGi versus JSR 277

I used OSGi as the example module system because I’m familiar with it, but the above sketch would probably apply equally well to JSR 277. However, to complete the usefulness of a module system, the system Java environment must recognize it. If Fedora wanted to use OSGi system-wide (not just within Eclipse) then the system Java’s behaviour would have to deviate (irresponsibly) from the Java standard, to recognize and resolve OSGi modules. So while our packages can all be made “OSGi-ready” now, only Eclipse would see the benefits.

Since not deviating from the Java standard is more important than the benefits of modules, whichever module system is integrated into Java itself will become the Fedora-wide Java module system. Once a decision is made, we can start implementing the integration improvements I’ve sketched out.

FOSDEM 2007

Tom Marble is a wizard at getting the right people in a room together. In the Distro DevJam we had Java packagers from Debian, Ubuntu, Fedora, Red Hat, OpenSUSE, SUSE, Mandriva, Gentoo and FreeBSD, together with all the right people from Sun. This was the perfect group for discussing packaging issues surrounding OpenJDK (and a great audience for my talk).

My feeling is that there is now a general consensus among the distros about how to solve the technical issues of packaging OpenJDK. The remaining tricky issues are procedural:

1) creating a good OpenJDK contributor license agreement that works for free software developers
2) giving distros the ability to self-certify
3) making the JCP open to free software contributors (e.g., no click-throughs on specs)

We discussed 2 and 3 in the DevJam but it was too soon to talk about solutions. I hinted at 1 in my slides with “accepts most patches upstream”, but in retrospect I feel I should have emphasized the importance of a good CLA when seeking copyright assignment from external free software contributors. Anyway, given the discussion in the room, I feel that these problems will eventually be solved but that some patience will be required.

Sun’s showing at FOSDEM was very impressive. The presentations were helpful in that they provided a preview of Sun’s development processes. Because I’ve worked on GNU Classpath’s AWT, I was especially glad to meet Igor Nekrestyanov, who is working on the Sun Java2D team to replace the class library encumbrances.

Already as a result of our FOSDEM meetings we’re seeing opportunities for collaboration. External contributors are still in an awkward position until the class library code drop actually happens, but in the meantime there are opportunities to contribute to third-party libraries that Sun is using to replace the encumbrances. Along these lines, I learned at FOSDEM that the Java plugin will not be released with the initial code drop. Maybe gcjwebplugin can be adapted to act as a temporary stop-gap.

One non-Java FOSDEM observation: the Fedora booth was excellent, complete with Fedora flip-flop/boating-key-chain giveaways.

The New Free Java Project

Surprise

I was happily surprised on Monday when Sun announced that they would
release Java ME, SE and EE under the GPL. Actually, “happily
surprised” is an understatement: Sun also adopted the GNU Classpath
linking exception! I couldn’t have hoped for a better result.

Watching the announcement webcast was a surreal experience. It was as
if I were dreaming. Rich Green standing in front of the giant “GPLv2”
slide. The pre-recorded statement of praise from RMS. Jonathan
Schwartz pointedly asking Rich Green if OpenSolaris would be GPL’d.
The Java world was upside down. The FSF-versus-Sun standoff
evaporated before my eyes. Even on Friday as the GPL rumours
swirled, I really couldn’t have imagined this event. Rarely does a
long-standing conflict end so swiftly and constructively.

Lessons Learned

Over the past few months I was tempted to be cynical about the pending
announcement, but I really tried to keep an open mind. I learned this
mindset from Mark Wielaard and Dalibor Topic, two exemplar free
software advocates. They were very patient, polite, and helpful with
people from Sun who were contemplating the licensing decision. Mark
and Dalibor’s diplomacy almost certainly had a hand in Sun’s choice of
license.

The New Free Java Project

Given the excellence of the announcement itself, complete with FSF
endorsements, an initial code drop, open-subscription mailing lists,
and an exhaustively comprehensive FAQ, I’m confident that Sun
will run the free software Java project as competently as they’ve
stewarded the Java language. And in an abstract sense, the license
choice is a declaration from Sun, a contract with the free software
community, that they will continue the project honestly and openly.

My Personal Reaction

The dream-like aura surrounding Monday’s webcast was probably stronger
for me because I’ve been working on libgcj and GNU Classpath for the
last three years. For all that work to result in a complete victory
is a wonderful feeling. There is also a tinge of sadness though, that
the GNU Classpath project, along with all or most of the code I wrote
for it, will likely be retired in the coming year. I think GNU
Classpath has become a successful free software martyr. (Although
parts of the project may rise again in zombie form: could there be a
place for a GTK AWT peerset in the new Java world? ;-)) The GNU
Classpath community is great to work in, and I hope our ties remain
strong. I especially look forward to FOSDEM 2007 where I expect
everyone, even man-di, to be present for a huge party! 🙂

Thoughts on Fedora Core 6

gcjwebplugin

My main goal for Fedora Core 6 was the inclusion of gcjwebplugin, so I’m very pleased that it will in fact be included. Though it is not enabled by default, having it in the base distribution means that interested people can experiment with it and file bug reports, with the minimum of configuration. We’ve already received some bug reports as a result. This is exactly the level of adoption I had hoped for in this release. I hope others will try their favourite applets and file bug reports if they don’t work, since doing so helps direct our gcjwebplugin development efforts.

In the meantime, work continues on GNU Classpath’s security implementation, so we should be able to safely run untrusted applets in Fedora Core 7.

Looking over my Fedora Core 5 Wishlist I see that many of my FC-5 wishes have been realized in FC-6.

Merging
Most of the proposed merging work happened during the FC-6 cycle. The following projects were merged into GNU Classpath:

  • GNU Crypto
  • Jessie
  • cp-tools
  • gcjwebplugin

The only straggler is gjdoc. In a short period of time we’ve managed to merge all these separate but essential projects into one coherent whole in GNU Classpath. This is good for everyone: VM projects that bundle GNU Classpath (Kaffe, libgcj, IKVM.NET, CACAO), who now all have access to the same tools; distributions, who now have fewer packages to maintain; and free Java developers, who now have a single, central location in which they can fix bugs in the class library and tool chain. I’m especially happy about the current tools situation, because before the merging work, VM projects each wasted effort on their own implementations of some tools; now we can all focus on making the GNU Classpath tools the definitive best set.

Applications and Applets

  • MegaMek works on libgcj, and is packaged and awaiting approval for inclusion in Fedora Extras
  • Azureus works on libgcj and is in Fedora Extras
  • many applets work on gcjwebplugin

AWT and Swing

The constant stream of AWT and Swing fixes and improvements in GNU Classpath means that many more AWT and Swing applications and applets work on libgcj. One large improvement in Fedora Core 6 is that the Cairo Graphics2D backend is now the default. Though GNU Classpath’s Graphics2D implementation is still being actively developed, the snapshot in FC-6 can run much of Sun’s Java2D demo:

screenshot of Sun's Java2D demo running on FC-6 gij

Fedora Core 5

Installation and Setup

I installed Fedora Core 5 on my IBM ThinkPad T41 last night. I’ll go through a few minor annoyances before the good news:

– NetworkManager wasn’t the default and I couldn’t find a way to enable it through the GUI (not that I should have to explicitly enable it anyway). I ended up dropping to the terminal and running it manually. Bug 171348

– After I got wireless networking working I wanted to be able to click on an MP3 link in Firefox and have the MP3 play automatically. So I installed the GStreamer plugins. When I clicked on an MP3, HelixPlayer was the selected default. But HelixPlayer started up and threw up its usual “format not supported… download?” dialog. So I uninstalled it. The next default was Rhythmbox. It started up but didn’t play anything automatically, and I couldn’t figure out how to play the file manually. So I uninstalled it. The next default was Totem, which worked perfectly. Bug 154392

There are 36 open bugs for Rhythmbox in Fedora Bugzilla but none that describes what I was seeing. I’ll have to file a new one.

– I tried to install the Eclipse and Java Development groups but Pup failed with “no more mirrors to try”. Bug 184326

The servers were probably busy last night so I’ll try again later.

Various media types weren’t supported out-of-the-box, obviously, but I installed the “GStreamer Universe” from GStreamer’s excellent FC5 web page and was very happy with the results. Most media types worked perfectly in Totem.

Installing the GStreamer repo files could have been smoother. It would be nice if pirut were the default handler for .repo files. Bug 186112

Impressions

Apart from these minor annoyances I’m very happy with this release. The new installer worked well, bootup feels much faster, all my laptop’s hardware worked out-of-the-box, sleep-on-lid-close works, NetworkManager eliminates manual network management, a few simple steps allow GStreamer to provide good media support. I’m very pleased that release-by-release I require fewer and fewer hacks to make Fedora into a comfortable, complete environment. In most cases in the past, these hacks involved installing unintegrated proprietary software.

Fedora Core 6

There are still places in Fedora Core 5 where such hacks are required. Three that stand out are:

– playing web-page-embedded audio and video

– Flash

– Java applets

I’m hoping that the free equivalents (GStreamer, Gnash and gcjwebplugin) will eliminate the need for these hacks in Fedora Core 6.

Fedora Core 5 Wishlist

Here’s my categorized list of goals for Fedora Core 5.

java-gcj-compat

  • GNU Crypto fix for Eclipse extssh support

    Casey Marshall has already committed a Diffie-Hellman JCE provider to GNU Classpath so this is just a matter of testing.

  • import all JAWT fixes

    All the necessary patches are already in GNU Classpath and libgcj.

  • Jessie merged into GNU Classpath
  • GNU Crypto core algorithms merged into GNU Classpath
  • gjdoc merged into GNU Classpath enabling …
  • an --enable-javadocs option to libgcj
  • make gcjx java-gcj-compat’s javac commands

    Not sure if this one is doable in time; I don’t mean full gcjx, just the bytecode-compiler portion in a standalone executable.

  • include Tritonus or some other sound implementation

AWT

  • ImageMagick backend for ImageIO
  • Graphics/Imaging refactoring for GTK 2.8 and Cairo
  • MegaMek packages in Fedora Extras

    This just involves a few AWT bug fixes now.

  • fix all 1.5 japi issues in java.awt.* packages
  • fix all 1.5 japi issues in javax.imageio.* packages

Swing

  • RHDB tools in Fedora Extras: Administrator, Visual Explain and Control Center.

    This will require updating these tools to support PostgreSQL 8.0.

  • Limewire in Fedora Extras

    It would be absolutely awesome to have this Swing app working on the free stack but I’m not sure it’s doable in the FC5 time frame. Would anyone like to undertake it?

SWT

  • Azureus in Fedora Extras

    Again, not sure if this is doable before FC5 but this would be awesome to have.

gcjwebplugin

  • implement and test security features in GNU Classpath to allow running untrusted bytecode

    It’s very unlikely that this one will be finished before FC5 but I’d like to at least have started the implementation by then.

I’m hoping this list will inspire some volunteers, especially for the big apps and GNU Classpath’s security framework. Most of these items are very doable and will greatly improve the free JPackage stack on Fedora.

AWT Progress

gcjappletviewer and gcjwebplugin

More and more applets are working with gcjappletviewer and gcjwebplugin. Here’s a screenshot of gcjappletviewer HEAD + GNU Classpath HEAD:

gcjappletviewer running a Sudoku applet

and one of gcjwebplugin HEAD + GNU Classpath HEAD:

gcjwebplugin running an aviation data applet

I’m getting through my AWT bug list. My next big chunk of work will be completing the transition to GTK 2.8 and Cairo, removing our GdkGraphics class and making GdkGraphics2D the default.

libjawt.so, java-gcjHEAD-compat, Big Merge

Here’s a screenshot of a JOGL demo running on Sun and libgcj:

JOGL Demo on Sun and libgcj

As you can see, it’s hard to tell which is which!

Big Merge

The screenshot represents several efforts coming together. First, the demo is running on natively-compiled GNU Classpath HEAD. Tom Tromey recently finished the Big Merge, making Classpath a sub-directory of gcc/libjava. To develop this demo, I replaced gcc/libjava/classpath with a checkout of Classpath HEAD and rebuilt libgcj. After patching a few places where divergences crept in I have a GCJ HEAD + Classpath HEAD development environment setup. Developing in this way is much more pleasant than the previous merge-every-patch approach; now I use the same source checkout for libgcj and JamVM+Classpath.

java-gcjHEAD-compat

The screenshot shows me running the java command in the exact same way in both windows, thanks to java-gcjHEAD-compat. It is an RPM spec file that allows you to create a first-class JPackage-style JVM alternative that points to a GCJ installation in a non-standard prefix. For example, I’ve installed GCJ HEAD in /home/fitzsim/install. java-gcjHEAD-compat points /usr/bin/java (through alternatives) to /home/fitzsim/install/bin/gij.

This is very convenient for developing libgcj as a JPackage JVM replacement since all JPackages assume the JVM is installed in /usr/lib/jvm. It’s also useful for building RPMs against libgcj HEAD, which I’ve done with JOGL in the screenshot. Now, JOGL is a heavy user of the AWT Native Interface which is implemented by a small library, libjawt.so. That brings me to my next topic.

libjawt.so Binary Compatibility

It has proved quite challenging to create a drop-in replacement for Sun’s libjawt.so. The jawt.h and jawt_md.h headers that Sun includes in the AWT Native Interface document are marked confidential and proprietary so I didn’t want to refer to them when creating an implementation for GNU Classpath. My initial libjawt.so attempt was not binary-compatible and so created problems for packages like JOGL that need to run on either libgcj or a proprietary JVM.

Using gdb and ptype I was able to determine the layout of JAWT types, so now libgcj’s libgcjawt.so is binary-compatible, except that its SONAME is versioned by libtool. But the SONAME of Sun’s libjawt.so is unversioned, simply “libjawt.so”, so even naming our library the same would not produce a binary-compatible result (e.g. “libjawt.so.6”) — and besides, doing so would cause confusion since libgcj’s “libjawt” is always present in the default library path, whereas Sun’s libjawt is not. So, what to do?

The last step in drop-in replaceability turned out to be a clever trick that Jakub Jelinek suggested:

libjawt.so:
	echo | $(GCJ_BIN_DIR)/gcc$(gcc_suffix) -shared -O2 -fpic -o libjawt.so -Wl,-soname,libjawt.so -xc - -lgcjawt

That’s a line from java-gcj-compat’s Makefile.am. It creates an empty library with the proper SONAME that links to the versioned libgcjawt.so. That’s it! java-gcj-compat’s libjawt.so gets installed in $JAVA_HOME/jre/lib/i386 just like Sun’s and apps needing libjawt.so to build or run don’t know the difference! I was relieved that this step was so simple — I was expecting linker script madness.

The proof that this all works is in the screenshot; I built Anthony Green‘s JOGL RPMs using java-gcjHEAD-compat and its libjawt.so and I’m running the JOGL demo on both Sun and java-gcjHEAD-compat. lsof shows the correct libjawt.so being loaded in each case:

$ /usr/sbin/lsof | grep libjawt.so
java      32435 fitzsim  mem       REG        3,2     2680     470785 /usr/lib/jvm/java-1.4.2-gcjHEAD-1.4.2.0/jre/lib/i386/libjawt.so
java      32460 fitzsim  mem       REG        3,2     3824     812009 /usr/java/jdk1.5.0_01/jre/lib/i386/libjawt.so

Now I want to see JOGL submitted to Fedora Extras and all this binary-compatibility work land in Fedora Core 5!