fitzsim's development log

Tom Marble is a wizard at getting the right people in a room together. In the Distro DevJam we had Java packagers from Debian, Ubuntu, Fedora, Red Hat, OpenSUSE, SUSE, Mandriva, Gentoo and FreeBSD, together with all the right people from Sun. This was the perfect group for discussing packaging issues surrounding OpenJDK (and a great audience for my talk).

My feeling is that there is now a general consensus among the distros about how to solve the technical issues of packaging OpenJDK. The remaining tricky issues are procedural:

1) creating a good OpenJDK contributor license agreement that works for free software developers
2) giving distros the ability to self-certify
3) making the JCP open to free software contributors (e.g., no click-throughs on specs)

We discussed 2 and 3 in the DevJam but it was too soon to talk about solutions. I hinted at 1 in my slides with “accepts most patches upstream”, but in retrospect I feel I should have emphasized the importance of a good CLA when seeking copyright assignment from external free software contributors. Anyway, given the discussion in the room, I feel that these problems will eventually be solved but that some patience will be required.

Sun’s showing at FOSDEM was very impressive. The presentations were helpful in that they provided a preview of Sun’s development processes. Because I’ve worked on GNU Classpath’s AWT, I was especially glad to meet Igor Nekrestyanov, who is working on the Sun Java2D team to replace the class library encumbrances.

Already as a result of our FOSDEM meetings we’re seeing opportunities for collaboration. External contributors are still in an awkward position until the class library code drop actually happens, but in the meantime there are opportunities to contribute to third-party libraries that Sun is using to replace the encumbrances. Along these lines, I learned at FOSDEM that the Java plugin will not be released with the initial code drop. Maybe gcjwebplugin can be adapted to act as a temporary stop-gap.

One non-Java FOSDEM observation: the Fedora booth was excellent, complete with Fedora flip-flop/boating-key-chain giveaways.

Surprise

I was happily surprised on Monday when Sun announced that they would
release Java ME, SE and EE under the GPL. Actually, “happily
surprised” is an understatement: Sun also adopted the GNU Classpath
linking exception! I couldn’t have hoped for a better result.

Watching the announcement webcast was a surreal experience. It was as
if I were dreaming. Rich Green standing in front of the giant “GPLv2”
slide. The pre-recorded statement of praise from RMS. Jonathan
Schwartz pointedly asking Rich Green if OpenSolaris would be GPL’d.
The Java world was upside down. The FSF-versus-Sun standoff
evaporated before my eyes. Even on Friday as the GPL rumours
swirled, I really couldn’t have imagined this event. Rarely does a
long-standing conflict end so swiftly and constructively.

Lessons Learned

Over the past few months I was tempted to be cynical about the pending
announcement, but I really tried to keep an open mind. I learned this
mindset from Mark Wielaard and Dalibor Topic, two exemplar free
software advocates. They were very patient, polite, and helpful with
people from Sun who were contemplating the licensing decision. Mark
and Dalibor’s diplomacy almost certainly had a hand in Sun’s choice of
license.

The New Free Java Project

Given the excellence of the announcement itself, complete with FSF
endorsements, an initial code drop, open-subscription mailing lists,
and an exhaustively comprehensive FAQ, I’m confident that Sun
will run the free software Java project as competently as they’ve
stewarded the Java language. And in an abstract sense, the license
choice is a declaration from Sun, a contract with the free software
community, that they will continue the project honestly and openly.

My Personal Reaction

The dream-like aura surrounding Monday’s webcast was probably stronger
for me because I’ve been working on libgcj and GNU Classpath for the
last three years. For all that work to result in a complete victory
is a wonderful feeling. There is also a tinge of sadness though, that
the GNU Classpath project, along with all or most of the code I wrote
for it, will likely be retired in the coming year. I think GNU
Classpath has become a successful free software martyr. (Although
parts of the project may rise again in zombie form: could there be a
place for a GTK AWT peerset in the new Java world? ;-)) The GNU
Classpath community is great to work in, and I hope our ties remain
strong. I especially look forward to FOSDEM 2007 where I expect
everyone, even man-di, to be present for a huge party! 🙂

gcjwebplugin

My main goal for Fedora Core 6 was the inclusion of gcjwebplugin, so I’m very pleased that it will in fact be included. Though it is not enabled by default, having it in the base distribution means that interested people can experiment with it and file bug reports, with the minimum of configuration. We’ve already received some bug reports as a result. This is exactly the level of adoption I had hoped for in this release. I hope others will try their favourite applets and file bug reports if they don’t work, since doing so helps direct our gcjwebplugin development efforts.

In the meantime, work continues on GNU Classpath’s security implementation, so we should be able to safely run untrusted applets in Fedora Core 7.

Looking over my Fedora Core 5 Wishlist I see that many of my FC-5 wishes have been realized in FC-6.

Merging
Most of the proposed merging work happened during the FC-6 cycle. The following projects were merged into GNU Classpath:

• GNU Crypto
• Jessie
• cp-tools
• gcjwebplugin

The only straggler is gjdoc. In a short period of time we’ve managed to merge all these separate but essential projects into one coherent whole in GNU Classpath. This is good for everyone: VM projects that bundle GNU Classpath (Kaffe, libgcj, IKVM.NET, CACAO), who now all have access to the same tools; distributions, who now have fewer packages to maintain; and free Java developers, who now have a single, central location in which they can fix bugs in the class library and tool chain. I’m especially happy about the current tools situation, because before the merging work, VM projects each wasted effort on their own implementations of some tools; now we can all focus on making the GNU Classpath tools the definitive best set.

Applications and Applets

• MegaMek works on libgcj, and is packaged and awaiting approval for inclusion in Fedora Extras
• Azureus works on libgcj and is in Fedora Extras
• many applets work on gcjwebplugin

AWT and Swing

The constant stream of AWT and Swing fixes and improvements in GNU Classpath means that many more AWT and Swing applications and applets work on libgcj. One large improvement in Fedora Core 6 is that the Cairo Graphics2D backend is now the default. Though GNU Classpath’s Graphics2D implementation is still being actively developed, the snapshot in FC-6 can run much of Sun’s Java2D demo:

Installation and Setup

I installed Fedora Core 5 on my IBM ThinkPad T41 last night. I’ll go through a few minor annoyances before the good news:

– NetworkManager wasn’t the default and I couldn’t find a way to enable it through the GUI (not that I should have to explicitly enable it anyway). I ended up dropping to the terminal and running it manually. Bug 171348

– After I got wireless networking working I wanted to be able to click on an MP3 link in Firefox and have the MP3 play automatically. So I installed the GStreamer plugins. When I clicked on an MP3, HelixPlayer was the selected default. But HelixPlayer started up and threw up its usual “format not supported… download?” dialog. So I uninstalled it. The next default was Rhythmbox. It started up but didn’t play anything automatically, and I couldn’t figure out how to play the file manually. So I uninstalled it. The next default was Totem, which worked perfectly. Bug 154392

There are 36 open bugs for Rhythmbox in Fedora Bugzilla but none that describes what I was seeing. I’ll have to file a new one.

– I tried to install the Eclipse and Java Development groups but Pup failed with “no more mirrors to try”. Bug 184326

The servers were probably busy last night so I’ll try again later.

Various media types weren’t supported out-of-the-box, obviously, but I installed the “GStreamer Universe” from GStreamer’s excellent FC5 web page and was very happy with the results. Most media types worked perfectly in Totem.

Installing the GStreamer repo files could have been smoother. It would be nice if pirut were the default handler for .repo files. Bug 186112

Impressions

Apart from these minor annoyances I’m very happy with this release. The new installer worked well, bootup feels much faster, all my laptop’s hardware worked out-of-the-box, sleep-on-lid-close works, NetworkManager eliminates manual network management, a few simple steps allow GStreamer to provide good media support. I’m very pleased that release-by-release I require fewer and fewer hacks to make Fedora into a comfortable, complete environment. In most cases in the past, these hacks involved installing unintegrated proprietary software.

Fedora Core 6

There are still places in Fedora Core 5 where such hacks are required. Three that stand out are:

– playing web-page-embedded audio and video

– Flash

– Java applets

I’m hoping that the free equivalents (GStreamer, Gnash and gcjwebplugin) will eliminate the need for these hacks in Fedora Core 6.

Here’s my categorized list of goals for Fedora Core 5.

java-gcj-compat

• GNU Crypto fix for Eclipse extssh support

  Casey Marshall has already committed a Diffie-Hellman JCE provider to GNU Classpath so this is just a matter of testing.

• import all JAWT fixes

  All the necessary patches are already in GNU Classpath and libgcj.

• Jessie merged into GNU Classpath
• GNU Crypto core algorithms merged into GNU Classpath
• gjdoc merged into GNU Classpath enabling …
• an --enable-javadocs option to libgcj
• make gcjx java-gcj-compat’s javac commands

  Not sure if this one is doable in time; I don’t mean full gcjx, just the bytecode-compiler portion in a standalone executable.

• include Tritonus or some other sound implementation

AWT

• ImageMagick backend for ImageIO
• Graphics/Imaging refactoring for GTK 2.8 and Cairo
• MegaMek packages in Fedora Extras

  This just involves a few AWT bug fixes now.

• fix all 1.5 japi issues in java.awt.* packages
• fix all 1.5 japi issues in javax.imageio.* packages

Swing

• RHDB tools in Fedora Extras: Administrator, Visual Explain and Control Center.

  This will require updating these tools to support PostgreSQL 8.0.

• Limewire in Fedora Extras

  It would be absolutely awesome to have this Swing app working on the free stack but I’m not sure it’s doable in the FC5 time frame. Would anyone like to undertake it?

SWT

• Azureus in Fedora Extras

  Again, not sure if this is doable before FC5 but this would be awesome to have.

gcjwebplugin

• implement and test security features in GNU Classpath to allow running untrusted bytecode

  It’s very unlikely that this one will be finished before FC5 but I’d like to at least have started the implementation by then.

I’m hoping this list will inspire some volunteers, especially for the big apps and GNU Classpath’s security framework. Most of these items are very doable and will greatly improve the free JPackage stack on Fedora.

gcjappletviewer and gcjwebplugin

More and more applets are working with gcjappletviewer and gcjwebplugin. Here’s a screenshot of gcjappletviewer HEAD + GNU Classpath HEAD:

and one of gcjwebplugin HEAD + GNU Classpath HEAD:

I’m getting through my AWT bug list. My next big chunk of work will be completing the transition to GTK 2.8 and Cairo, removing our GdkGraphics class and making GdkGraphics2D the default.

Nostalgia:

Caps Lock seems to be stuck on but Corel Office for Java basically works on GNU Classpath!

Here’s a screenshot of a JOGL demo running on Sun and libgcj:

As you can see, it’s hard to tell which is which!

Big Merge

The screenshot represents several efforts coming together. First, the demo is running on natively-compiled GNU Classpath HEAD. Tom Tromey recently finished the Big Merge, making Classpath a sub-directory of gcc/libjava. To develop this demo, I replaced gcc/libjava/classpath with a checkout of Classpath HEAD and rebuilt libgcj. After patching a few places where divergences crept in I have a GCJ HEAD + Classpath HEAD development environment setup. Developing in this way is much more pleasant than the previous merge-every-patch approach; now I use the same source checkout for libgcj and JamVM+Classpath.

java-gcjHEAD-compat

The screenshot shows me running the java command in the exact same way in both windows, thanks to java-gcjHEAD-compat. It is an RPM spec file that allows you to create a first-class JPackage-style JVM alternative that points to a GCJ installation in a non-standard prefix. For example, I’ve installed GCJ HEAD in /home/fitzsim/install. java-gcjHEAD-compat points /usr/bin/java (through alternatives) to /home/fitzsim/install/bin/gij.

This is very convenient for developing libgcj as a JPackage JVM replacement since all JPackages assume the JVM is installed in /usr/lib/jvm. It’s also useful for building RPMs against libgcj HEAD, which I’ve done with JOGL in the screenshot. Now, JOGL is a heavy user of the AWT Native Interface which is implemented by a small library, libjawt.so. That brings me to my next topic.

libjawt.so Binary Compatibility

It has proved quite challenging to create a drop-in replacement for Sun’s libjawt.so. The jawt.h and jawt_md.h headers that Sun includes in the AWT Native Interface document are marked confidential and proprietary so I didn’t want to refer to them when creating an implementation for GNU Classpath. My initial libjawt.so attempt was not binary-compatible and so created problems for packages like JOGL that need to run on either libgcj or a proprietary JVM.

Using gdb and ptype I was able to determine the layout of JAWT types, so now libgcj’s libgcjawt.so is binary-compatible, except that its SONAME is versioned by libtool. But the SONAME of Sun’s libjawt.so is unversioned, simply “libjawt.so”, so even naming our library the same would not produce a binary-compatible result (e.g. “libjawt.so.6”) — and besides, doing so would cause confusion since libgcj’s “libjawt” is always present in the default library path, whereas Sun’s libjawt is not. So, what to do?

The last step in drop-in replaceability turned out to be a clever trick that Jakub Jelinek suggested:

libjawt.so:
	echo | $(GCJ_BIN_DIR)/gcc$(gcc_suffix) -shared -O2 -fpic -o libjawt.so -Wl,-soname,libjawt.so -xc - -lgcjawt

That’s a line from java-gcj-compat’s Makefile.am. It creates an empty library with the proper SONAME that links to the versioned libgcjawt.so. That’s it! java-gcj-compat’s libjawt.so gets installed in $JAVA_HOME/jre/lib/i386 just like Sun’s and apps needing libjawt.so to build or run don’t know the difference! I was relieved that this step was so simple — I was expecting linker script madness.

The proof that this all works is in the screenshot; I built Anthony Green‘s JOGL RPMs using java-gcjHEAD-compat and its libjawt.so and I’m running the JOGL demo on both Sun and java-gcjHEAD-compat. lsof shows the correct libjawt.so being loaded in each case:

$ /usr/sbin/lsof | grep libjawt.so
java      32435 fitzsim  mem       REG        3,2     2680     470785 /usr/lib/jvm/java-1.4.2-gcjHEAD-1.4.2.0/jre/lib/i386/libjawt.so
java      32460 fitzsim  mem       REG        3,2     3824     812009 /usr/java/jdk1.5.0_01/jre/lib/i386/libjawt.so

Now I want to see JOGL submitted to Fedora Extras and all this binary-compatibility work land in Fedora Core 5!

Sven de Marothy completely rewrote the image-handling sections of the GTK peers. This was one of the most convoluted areas in the peers and one of the buggiest. Now it is very clear and well-organized. Check out this screenshot:

The top is JDK 1.4, centre is Classpath 0.14 and bottom is Classpath HEAD with Sven’s patch. This work is awesome because it means we’ll have a solid foundation when we make the transition to using BufferedImages exclusively. Thanks Sven!

I submitted my ecj option parsing patch to Eclipse.

The patch went into Rawhide’s eclipse-ecj last night, which meant I was able to remove the last option munging stuff from java-gcj-compat. Behold:


$ ls -l /usr/lib/jvm/java-gcj/bin
jar -> /usr/bin/fastjar
java -> /usr/bin/gij
javac -> /usr/bin/ecj
javadoc -> /usr/bin/gjdoc
javah -> /usr/bin/gjnih
rmic -> /usr/bin/grmic
rmiregistry -> /usr/bin/grmiregistry


No wrapper scripts, just direct symlinks. All these tools are now command-line compatible with their proprietary equivalents.

java-gcj-compat’s remaining holes are representative of the entire free stack’s J2SE deficiencies. Specifically:

1. dependencies on external crypto providers
2. no CORBA tools
3. no Java plugin

These are the areas that we need to focus on this year. 1 should be pretty easy; just a matter of merging GNU Crypto, Jessie and Bouncy Castle into GNU Classpath. 2 will be harder. Ideally we could get JacORB dual-licensed LGPL/GPL+exception, reimplement all the non-free OMG headers it uses and merge it into GNU Classpath. Quite a bit of effort. 3 will be harder still; we’ll need to finish libgcj’s security implementation (with test suites and everything) and fix many bugs in our AWT and Swing implementations.