On Monday, I found Conkeror through the EmacsWiki. It’s a Firefox extension that provides an Emacs-like interface to the browser. It includes all the essential Emacs features (minibuffer, incremental search, bookmark tab completion, buffer management) and an additional killer feature: numbered links. I’ve always wanted an efficient way to follow links and this is it. You can’t get much faster than <number>-<Return>!
Here are some testing strategies that I’d like to see implemented for IcedTea. I’ve asked one of our interns to investigate these points.
Reduce Mauve-on-IcedTea failures to 0
Provide reproducible way of running Mauve on IcedTea which should result in 0 failures
“make dist” target for Mauve
Once these are complete we can a) set up nightly regression runs on http://icedtea.classpath.org, b) insist on having no regressions in package releases by running Mauve as part of the RPM build.
Make IcedTea demos build standalone
The IcedTea demos include libraries built during the IcedTea build, but these libraries cannot be built using the provided sample Makefiles. Installing the java-1.7.0-icedtea-demo RPM should a) install the demo source code in /usr/share/java-1.7.0-icedtea/demo, b) install the demo code’s build requirements as dependencies, c) provide working sample Makefiles. That way, trying the demos can be reduced to something like:
yum install java-1.7.0-icedtea-demo
cp -r /usr/share/java-1.7.0-icedtea/demo ~
Ideally we’d test building and running the demos as part of the RPM build.
Run DaCapo benchmark reproducibly on IcedTea
Run graphical demos with java.awt.Robot and Xvfb
I’d like to include such graphical tests in the RPM build if the setup isn’t too fragile.
Longer term if Sun releases the jtreg harness (and some tests), we won’t hate it. 😉 More likely we’ll want to integrate it too into our RPM builds.
Here are some thoughts about how to make OpenJDK easily maintainable on Fedora. Achieving this goal will require changes to OpenJDK, updates to the Fedora implementation of the JPackage conventions and an rpm patch.
Here is a list of problems and a strawman solution for each. Each item needs careful investigation to determine the most appropriate solution.
Read OpenSSL certs directly
Now: We generate a cacerts file from certificates shipped with the Fedora OpenSSL package.
Future: OpenJDK should parse the OpenSSL certificates directly. That would give us one point of maintenance for the certificates and allow OpenJDK to take advantage of new certificates when OpenSSL is updated.
Replace font.properties with fontconfig
Now: fontconfig.Fedora.properties hard-codes Fedora fonts that OpenJDK uses.
Future: OpenJDK should use fontconfig to determine which fonts to use, to match desktop font preferences.
Read glibc timezone and locale data
Now: OpenJDK bundles its own set of timezone and locale data.
Future: OpenJDK should read glibc’s timezone and locale data. This provides a single point of maintenance for these data and eliminates the need to reinstall OpenJDK, or run a separate tool like tzupdater, whenever timezone or locale data updates are required.
Security policy and provider configuration under /etc/java
Now: OpenJDK security policies are installed and configured under /usr/lib/jvm/java-$version-$vendor/jre/lib/security.
Future: OpenJDK security policies and providers should be configured from /etc/java, since /etc is the standard location for administrator-edited configuration files. If configuration were under /etc/java/java-$version, the settings could be shared between same-versioned JDKs. There should also be a .d directory so that new security policies and providers can be easily installed by separate packages.
Endorsed directories: /usr/lib*/java-$version/endorsed, /usr/share/java-$version/endorsed
Now: OpenJDK loads endorsed jars from /usr/lib/jvm/java-$version-$vendor/jre/lib/ext.
Future: OpenJDK should load endorsed jars from locations like /usr/lib/java-$version/endorsed and /usr/share/java-$version/endorsed so that separate packages can easily install and uninstall additional endorsed jars.
Properties files in /etc/java
Now: OpenJDK loads .properties files from /usr/lib/jvm/java-$version-$vendor/jre/lib.
Future: OpenJDK should load .properties files from /etc/java, since /etc is the standard location for config files.
Split out bundled code, push Sun patches upstream
Now: OpenJDK bundles source code packages from many other projects.
Future: Using the new module system or some other mechanism, OpenJDK should use the system-installed versions of these packages. Any modifications to these projects that Sun has been maintaining in their tree should go upstream.
Default JNI load path, to support loading 32-bit and 64-bit JNI libraries from the same jar
Now: System.loadLibrary relies on the default library paths.
Future: System.loadLibrary should also look in standard paths where packages can install JNI libraries: /usr/lib/jni on 32-bit JVMs and /usr/lib64/jni on 64-bit JVMs.
Default /usr/share/java “CLASSPATH” directory
Now: JPackage provides a build-classpath script to build CLASSPATH settings from jars found in system-default locations such as /usr/share/java.
Future: OpenJDK’s module system should resolve dependencies from these system-default locations, obviating the build-classpath script. Related: Red Hat bug 170128.
OpenJDK Package Changes
Multiarch support in jpackage-utils
Now: the jpackage-utils package is built as a noarch package but contains references to /usr/lib.
Future: jpackage-utils should also support JDK installations in /usr/lib64. Doing so will raise questions about how JDK alternatives are handled, since they are based in /usr/lib. It may be best to change the alternatives tool itself to special-case /usr/lib and /usr/lib64.
Simplify package split to base, javadoc
Now: JDK packages are split up into base (JRE), devel, demo, src and javadoc subpackages.
Future: We could reduce this to just a base “java” package, which includes the JRE, development tools, demos and src.zip. javadoc will still need to remain a separate subpackage because of its installed size.
Simplify alternatives to just “java”
Now: There are many JDK primary alternatives: java, javac, jre_$vendor, $jre_$version, java_sdk_$vendor, java_sdk_$version, libjavaplugin.so
Future: With the simplified package split we could reduce this list to two primary alternatives: java and javadoc. But simplification may not be required if everyone just uses system-switch-java instead of dealing with alternatives directly.
Fix rpm $1 arch-specific bug
Now: Red Hat bug 127359 prevents correct upgrading of alternatives symlinks in multilib installations.
Future: The bug must be fixed.
I’m experimenting with GNOME’s accessibility support; specifically, how it interacts with Swing and AWT applications. I’m impressed with the framework’s implementation. I had trouble saving GOK‘s alternative pointer device setting, and there seems to be an issue with having the alternative pointer device control the core pointer, but otherwise the interface is stable and usable. As an experiment, I wrote this blog post using only a single-button mouse we had lying around the office 🙂 The process was slow but smooth. My other experiments were also successful: GOK works with Swing and AWT applications over the Java Access Bridge.
Our team at Red Hat has been doing some planning now that OpenJDK has been released. First, a summary of OpenJDK’s status:
What Sun has released:
- most of the 1.6-level JDK, including:
- the Hotspot virtual machine with x86 and x86_64 JITs
- the Java compiler
- most of the class libraries
What Sun hasn’t released:
- 4% of the class library code — some of the class library code is “encumbered” meaning that Sun can’t release it under the GPL. The encumbrances are in the sound system, font rasterizer, graphics rasterizer, SNMP and crypto areas.
- plugin and javaws
- a 1.5 version
The second list points the way to some of our immediate goals. Here is a list of some of the goals we decided on:
Packaging OpenJDK as IcedTea
We can’t call what we ship “OpenJDK” because that’s a Sun trademark. We’ve decided on the temporary name “IcedTea”, and we can say that it’s based on code from openjdk.java.net. We’ve posted an SRPM containing the free OpenJDK sources. But it still requires non-free stuff to build, which brings us to the next goal.
Building with free software
Currently OpenJDK requires openmotif, Sun’s proprietary JDK 1.6 and a bundle of proprietary “plugs” (to replace the encumbrances) to build. Our group’s first goal is to make OpenJDK buildable against a set of fully-free packages.
Setting up icedtea.classpath.org
With help from some GNU Classpath community members we’re setting up icedtea.classpath.org with a Wiki, mailing list and a Subversion repository, to host the work we’ll be doing on OpenJDK. This is a temporary site where we can experiment with the OpenJDK code while Sun works out its contribution processes. The site is not complete yet.
Testing without the encumbrances
Because we haven’t built OpenJDK without the binary plugs for the encumbrances yet, we don’t know exactly how it fails if they’re not present. We need to see if the result can be proposed for inclusion in Fedora.
Replacing encumbered bits with parts from GNU Classpath
Once we’ve tested running without the encumbrances we’ll know where to focus our replacement efforts.
Testing OpenJDK interpreter on other architectures
Since at this point we only have x86 and x86_64 JITs, platform coverage will be a problem. In the short term we should test the OpenJDK interpreter portability since it may be possible to use it as a stop-gap on the architectures for which we don’t have JITs.
Working with Sun on contributor license agreements, use of trademarks, self-certification and governance.
Eventually, we’d like to ship in Fedora a certified Java implementation that we can call OpenJDK and “Java compatible”. Sun intends to make that possible through a self-certification process. Eventually we should be able to contribute code directly to the OpenJDK project — Sun has worked out the governance model it will use and is implementing a Mercurial repository.
Working with other Red Hat groups on OpenJDK
We expect to work closely with other Red Hat groups on both system integration (kernel, tools, desktop, packaging) and on requests from internal “consumers” of the JDK (JBoss, Eclipse).
We will continue to support GCJ where it is deployed, but any new work we do will be on the OpenJDK codebase. GCJ still has better architecture coverage than OpenJDK, supporting all 7 RHEL architectures where OpenJDK supports 2, but we’ve decided that the benefits of OpenJDK outweigh this advantage of GCJ, and that effort is better spent trying to get (or write, if need be) more JITs for OpenJDK.
We don’t have definite timelines yet. We’re in an investigation stage right now, so precise timelines will have to wait until we’ve had more time to discuss the OpenJDK codebase and Sun’s timelines.
I packaged the new code drop from the OpenJDK project using the JPackage conventions. Because naming rules are still being worked out for derivatives of OpenJDK code, I’ve decided to call the result “java-1.7.0-icedtea”. This is just a preliminary development package; I hope Sun will approve renaming it “java-1.7.0-openjdk” or “java-1.6.0-openjdk” when it officially lands in Fedora.
The OpenJDK build still depends on some non-free software so I’m providing an SRPM containing only the GPL’d code. People interested in building the complete not-fully-free OpenJDK can fill the gaps with binary plugs.
I’m going to investigate eliminating the non-free requirements. I haven’t studied the code enough to know if it’s worth producing a fully free build before starting encumbrance replacement efforts — it depends how much functionality is lost by excluding the binary blobs. Another interesting experiment would be bootstrapping OpenJDK with a free runtime.
Andrew Overholt recently pointed me to a blog entry on Java module systems. I can’t comment in detail on the JSRs or how they should interact because they’re all behind click-through licenses. My instinct tells me the ideal answers are: “Sun, drop JSR 277 and adopt OSGi; IBM, develop OSGi within the Java tree, on the same schedule as Java”, though this may be politically impossible.
Beyond politics, the post got me thinking about how a Java module system should integrate with Fedora. Here is a sketch of how I think it should work.
OSGi Modules in Fedora
OSGi modules distributed with Fedora will need to be wrapped by RPMs, so that they can be built, released and managed using existing tools and processes.
OSGi and RPM metadata provide similar information, so the adoption of OSGi will simplify the creation of Java RPMs. The packager should only be required to list the proper build dependencies in a spec file; Requires and Provides should be auto-calculated by rpmbuild. This is how C module (DSO) RPMs are currently produced.
The reverse operation should also be supported, for transforming non-modularized upstream projects into OSGi modules at rpmbuild time, using packager-provided metadata (the Name, Version, Requires and Provides fields). Then we could easily convert our currently-shipping Java packages to OSGi modules. At the same time, we should encourage upstream projects to properly modularize their jars, including any build dependency jars they bundle.
rpmbuild should automatically namespace packager-built modules with a Fedora release prefix, to distinguish them from the equivalent upstream module.
Module Loader Customizations
The Fedora OSGi loader should prioritize these Fedora-namespaced modules over non-Fedora modules with the same version.
Likewise, the OSGi updater should compare upstream modules to those available in Fedora, and where versions match, pull the Fedora module. There will need to be an OSGi-yum bridge so that the OSGi updater can request module metadata from yum, and instruct it which package to install. The required yum interfaces already exist to support update programs like pirut and pup. For dependencies not resolved in the yum repositories, the OSGi updater should fall back to its list of jar repositories.
The biggest improvement to the current Java packaging situation may well come from upstream projects. Once a module system becomes part of the Java standard, I hope to see upstream projects adopt it in a disciplined way. If upstream projects started bundling modules instead of just jars, packaging these projects would become simpler. The problem has been that it’s unclear which version of a particular jar is bundled with a project’s source — a module answers that question, and helps the packager set up BuildRequires. If this versioning convention becomes common, the BuildRequires generation step could even be automated by a “make new-sources-java” target in Makefile.common.
Projects also bundle dependencies in binary jars, for deployment on systems lacking native package management. With upstream module versioning, Fedora namespacing, and the Fedora module loader customizations, system-installed modules, where available, would be used in favour of the equivalent bundled modules. This should be the desired behaviour, since system administrators should trust Fedora Java packages more than the upstream binaires.
Along these lines, system administrators may be interested in performing operations on the set of all system-installed OSGi packages. There is talk of adding tagging support to rpm and yum; once it is in place, we should apply a common tag to all packages that contain OSGi modules.
OSGi versus JSR 277
I used OSGi as the example module system because I’m familiar with it, but the above sketch would probably apply equally well to JSR 277. However, to complete the usefulness of a module system, the system Java environment must recognize it. If Fedora wanted to use OSGi system-wide (not just within Eclipse) then the system Java’s behaviour would have to deviate (irresponsibly) from the Java standard, to recognize and resolve OSGi modules. So while our packages can all be made “OSGi-ready” now, only Eclipse would see the benefits.
Since not deviating from the Java standard is more important than the benefits of modules, whichever module system is integrated into Java itself will become the Fedora-wide Java module system. Once a decision is made, we can start implementing the integration improvements I’ve sketched out.
Tom Marble is a wizard at getting the right people in a room together. In the Distro DevJam we had Java packagers from Debian, Ubuntu, Fedora, Red Hat, OpenSUSE, SUSE, Mandriva, Gentoo and FreeBSD, together with all the right people from Sun. This was the perfect group for discussing packaging issues surrounding OpenJDK (and a great audience for my talk).
My feeling is that there is now a general consensus among the distros about how to solve the technical issues of packaging OpenJDK. The remaining tricky issues are procedural:
1) creating a good OpenJDK contributor license agreement that works for free software developers
2) giving distros the ability to self-certify
3) making the JCP open to free software contributors (e.g., no click-throughs on specs)
We discussed 2 and 3 in the DevJam but it was too soon to talk about solutions. I hinted at 1 in my slides with “accepts most patches upstream”, but in retrospect I feel I should have emphasized the importance of a good CLA when seeking copyright assignment from external free software contributors. Anyway, given the discussion in the room, I feel that these problems will eventually be solved but that some patience will be required.
Sun’s showing at FOSDEM was very impressive. The presentations were helpful in that they provided a preview of Sun’s development processes. Because I’ve worked on GNU Classpath’s AWT, I was especially glad to meet Igor Nekrestyanov, who is working on the Sun Java2D team to replace the class library encumbrances.
Already as a result of our FOSDEM meetings we’re seeing opportunities for collaboration. External contributors are still in an awkward position until the class library code drop actually happens, but in the meantime there are opportunities to contribute to third-party libraries that Sun is using to replace the encumbrances. Along these lines, I learned at FOSDEM that the Java plugin will not be released with the initial code drop. Maybe gcjwebplugin can be adapted to act as a temporary stop-gap.
One non-Java FOSDEM observation: the Fedora booth was excellent, complete with Fedora flip-flop/boating-key-chain giveaways.
I was happily surprised on Monday when Sun announced that they would
release Java ME, SE and EE under the GPL. Actually, “happily
surprised” is an understatement: Sun also adopted the GNU Classpath
linking exception! I couldn’t have hoped for a better result.
Watching the announcement webcast was a surreal experience. It was as
if I were dreaming. Rich Green standing in front of the giant “GPLv2”
slide. The pre-recorded statement of praise from RMS. Jonathan
Schwartz pointedly asking Rich Green if OpenSolaris would be GPL’d.
The Java world was upside down. The FSF-versus-Sun standoff
evaporated before my eyes. Even on Friday as the GPL rumours
swirled, I really couldn’t have imagined this event. Rarely does a
long-standing conflict end so swiftly and constructively.
Over the past few months I was tempted to be cynical about the pending
announcement, but I really tried to keep an open mind. I learned this
mindset from Mark Wielaard and Dalibor Topic, two exemplar free
software advocates. They were very patient, polite, and helpful with
people from Sun who were contemplating the licensing decision. Mark
and Dalibor’s diplomacy almost certainly had a hand in Sun’s choice of
The New Free Java Project
Given the excellence of the announcement itself, complete with FSF
endorsements, an initial code drop, open-subscription mailing lists,
and an exhaustively comprehensive FAQ, I’m confident that Sun
will run the free software Java project as competently as they’ve
stewarded the Java language. And in an abstract sense, the license
choice is a declaration from Sun, a contract with the free software
community, that they will continue the project honestly and openly.
My Personal Reaction
The dream-like aura surrounding Monday’s webcast was probably stronger
for me because I’ve been working on libgcj and GNU Classpath for the
last three years. For all that work to result in a complete victory
is a wonderful feeling. There is also a tinge of sadness though, that
the GNU Classpath project, along with all or most of the code I wrote
for it, will likely be retired in the coming year. I think GNU
Classpath has become a successful free software martyr. (Although
parts of the project may rise again in zombie form: could there be a
place for a GTK AWT peerset in the new Java world? ;-)) The GNU
Classpath community is great to work in, and I hope our ties remain
strong. I especially look forward to FOSDEM 2007 where I expect
everyone, even man-di, to be present for a huge party! 🙂