Lisp on Talos II

I’ve been following the status of Lisp on ppc64le lately.

I’m running ppc64le Debian sid. Just after I had set up my system, I did some experimentation with what Debian packages had to offer. ECL was the only Lisp that worked, so I started using it for various projects. (I’ve since learned on #sbcl that CLISP built from source is also a good option.)

Ideally I wanted to be able to use SBCL, so I wondered how far into an SBCL bootstrap I could get with ECL as the host compiler. A few months ago, I found the answer was not very far.

Since then though, the SBCL maintainers have been hard at work on two fronts: making SBCL bootstrap against ECL again (on any architecture) and porting SBCL to PPC64.

Recently with a few minor local changes, I was able to bootstrap SBCL natively on my Talos II starting with ECL, then build SBCL again with the bootstrapped SBCL. I had to build ECL from the tip of its development branch, but it has a nice build system and, being based on C, doesn’t require any bootstrapping steps.

So the SBCL ppc64 port is really shaping up; hopefully the next release will advertise ppc64le support. For now, plenty of packages work for me already, like SILME and Quicklisp.

I’d also like to try the same bootstrap procedure on ppc64be ABI version 1 (Debian sid) and ppc64be ABI version 2 (another distro, probably Adélie). I’m working on setting up an environment with qemu-system-ppc64.

Anyway, I’m happy that I can now use SBCL on my Talos II.

An excellent machine

I finished building my Talos II system, and I decided to post my thoughts on it here.

This is an excellent machine, the best workstation I’ve ever used. It’s very fast — it compiles Emacs and the kernel in 2 minutes, and just spins up the fans a little bit while doing so (under normal operation it’s quiet). And it’s in a completely different league in terms of openness than any other computer I’ve owned (and any other high performance computer on the market today).

The system came with a nice user’s manual, and full schematics! I’ve already referred to the schematics to set up the pinmux for a rarely-used serial port so that I could use a TTL serial cable I had lying around (I submitted the device tree patch to Raptor).

It’s really two computers in one: the “baseboard management controller” (BMC), a low power ARMv6, with a full distro on it, and the main POWER9 CPUs. The BMC boots up as soon as you plug in the power supply, before you even press the power button. (It would be nice if there were an ncurses top-like viewer for fan speeds and temperatures that I could leave running on the BMC serial console, but I haven’t found such a thing yet.)

It has serial ports everywhere! One right into the main CPU, and two into the BMC. This is great for low level development, e.g., breaking into bootcode at an early stage.

I left the machine running overnight after first booting it. My neighbourhood had a power glitch, and in the morning I discovered the main CPU was off, and power cycling it via the BMC wasn’t working. Before unplugging and plugging back in, I asked on #talos-workstation, and it turns out I had hit a bug in the first-release firmware. With special commands I was able to power cycle the main CPU just via BMC software (no unplug required). Wanting to know the details, I asked if there was a data sheet for the chip I was interacting with. The amazing thing, from an openness perspective, is that one of the Raptor engineers instead referred me directly to the Verilog source code of the FPGA handling the power sequencing. No searching for datasheets, no black box testing, just straight source code (which is recompilable using an open source FPGA toolchain, BTW.)

It’s so refreshing to not have to do reverse engineering and speculation on opaque things when something fails. On this machine, everything is there, you just look up the source code.

I’ve always disliked the inflexibility and opacity of BIOS/EFI in the x86 world. IBM has done an amazing job here with OpenPOWER. All the early boot code is open, no locked management engines or anything like that. And they’ve adopted Petitboot as the bootloader. It runs on a Linux kernel, so I was able to bootstrap via deboostrap over NFS by building everything within the bootloader environment. Running a compiler in a boot environment is surreal. Even with free options on x86 like libreboot or coreboot, and GRUB, the boot environment is extremely limited. With Petitboot at times I wondered if I even needed to boot into a “desktop” kernel (at least for serial-only activities.)

Now I’m setting up my development environment and I’m learning about the PPC64 ELFv2 ABI, with a view toward figuring out how to bootstrap SBCL. I feel lucky that I got a POWER9 machine early while there are still some rough edges to figure out.

Talos II available for pre-order

The Talos II is now available for pre-order. It is the more affordable, more power-efficient successor to the Talos I machine I wrote about in a previous post.

This is a very affordable machine for how powerful it will be, and there are minimal mainboard + CPU + RAM bundles (e.g., this one), around which one can build a workstation with more readily-available parts. I’ve placed an order for one of the bundles, and will buy the chassis and GPU separately (mainly to avoid high cross-border shipping fees for the full workstation).

The Talos II is an important machine for Free Software, and will likely be RYF-certified by the FSF. Pre-orders end September 15th!

Talos Secure Workstation

This is a very important machine that really deserves to get built. Anyone who cares about Free Software should consider funding this project at some level, and spreading the word to their friends. If this project succeeds, it will bootstrap a market for new, owner-controlled performant desktop machines. If it fails, no such computers will exist. The project page and updates explain the current (rather depressing) state of general purpose computing better than I could, so take a look.

Excorporate 0.7.0 released in GNU ELPA

I finished getting Excorporate and all its dependencies into GNU ELPA. Excorporate lets Emacs retrieve calendar items from an Exchange server.

Excorporate in GNU ELPA

I had to rewrite the default UI to use Org Mode, because Calfw isn’t entirely copyright-assigned to the FSF yet. The Calfw UI is still there for reference, but as a text file so that GNU ELPA’s build and publishing steps ignore it. Both UI handlers use the same updated APIs from the main excorporate.el library.

Excorporate Org handler

I made sure Excorporate and all its dependencies use only features available since GNU Emacs 24.1. This is pretty good coverage; Emacs 24.1 introduced the packaging system, so if an Emacs version supports packages, it supports Excorporate.

Other than DNS lookups, Excorporate is completely asynchronous, so it won’t block the Emacs main loop. And it is pure Emacs Lisp so it runs on any operating system that Emacs does.

In addition to Org Mode support, release 0.7.0 collects all the suggestions users have made on this blog and adds Exchange 2007 support.

To install: M-x package-install RET excorporate

To get the source code:

git clone git://git.savannah.gnu.org/emacs/elpa.git

To report bugs: M-x report-emacs-bug

Crossgrading from Fedora 22 i686 to x86_64

I’ve always upgraded my Fedora laptop incrementally using the recommended tools (yum, preupgrade, fedup). For this reason, my initial decision to install i686 Fedora (back when 32-bit compatibility was important) has carried through, and I’ve always used i686 Fedora, even though the CPU supports x86_64.

Recently I wanted to try out Mezzano in a virtual machine, but it is x86_64-only. Obviously, virtualization tools don’t support running an x86_64 guest on an i686 host, so I shelved the experiment.

Then I found a use for Docker (testing Emacs HTTP authentication schemes), and wanted to try it out. But I ran into this surprising limitation: Docker is 64-bit only.

At this point I had two motivators to move to x86_64, but no desire to wipe my system. I discovered an interesting new term, “crossgrading”, when I found these instructions: Cross-grading from i686 to x86_64: it is possible (but unsupported). They were written for Fedora 14, but I decided to try them out on Fedora 22 (after backing up all my data).

Long story short, careful application of dnf download and rpm --force (and an rpm database rebuild) resulted in a still-working x86_64 Fedora 22 system, with no need to wipe the drive. I’m glad Roberto Ragusa took the time to write up those instructions, and I wanted to document that they roughly still apply, eight Fedora versions later.

EUDC Improvements

I use Emacs Unified Directory Client (EUDC) for completing email addresses from LDAP and BBDB databases. It’s nice to be able to complete names from LDAP when composing emails, obviously, but it’s also nice in Org mode to M-x eudc-expand-inline someone’s name into my notes.

When I first configured the EUDC LDAP backend for my environment — RHEL 6 ldapsearch, LDAP-over-SSL server — setup was very involved. There were lots of poor defaults, strange extra configuration files, function call requirements, and ldapsearch incompatibilities. EmacsWiki instructions were very long just to get sane “Givenname Surname <email@address>” completion in GNUS.

I filed a bug report with configuration simplifications, bug fixes and EUDC Info manual updates, and somehow I ended up as the EUDC maintainer. I’ve committed the improvements to the Emacs master branch; they’ll be released in Emacs 25.

If you’ve tried EUDC in the past and been turned off by its arcane configuration, you might want to re-read the “LDAP Configuration” section of the Info manual, and try again. If you still can’t get it working, file a bug report at bug-gnu-emacs and I’ll try to respond to it within a few days. Mention EUDC or LDAP in the subject of the report. Likewise, if you do get it working with hacks then let me know via a bug report.

Excorporate 0.6.0: Exchange integration for Emacs

Here is Excorporate version 0.6.0.

New in this release:

– Support for overriding settings autodiscovery by manually configuring an EWS URL

– A check to prevent soap-client conflicts with other packages

– Handle user@sub.domain.com/autodiscover.domain.com autodiscovery case

– Support for Exchange 2013 (via the new API addition, exco-server-version)

Thanks to everyone who tested autodiscovery on the 0.5.4 release and provided feedback. I’ve listed most of you in the Acknowledgements section.

Excorporate has been accepted for inclusion in GNU ELPA but I’m still waiting on copyright assignment for one dependency before releasing it there. This release still takes the form of a tarball. First download it, then install it with M-x package-install-file.

Excorporate: EWS support for Emacs

I use Exchange at work for calendaring. I also use terminal-mode emacsclient when I’m logged in from another machine. In that scenario I can’t easily open a web browser to use Outlook Web Access. It annoyed me that I couldn’t check my schedule from within a terminal Emacs session. Thus, I did the only sensible thing and implemented full Exchange Web Services API support for Emacs.

Excorporate terminal screenshot

The result is a library called Excorporate that has proof-of-concept Calendar integration to show today’s meetings. It’s all written in Elisp, making it cross-platform. It’s also cross-Emacs; I tested it on Emacs 24.1 through Emacs 24.4, on GNU/Linux and MS-Windows. It works on all Emacs versions that support packages.

The tricky parts were:

  • extending soap-client to support all the XML Schema (XSD) datatypes required by the EWS WSDL file,
  • asynchronous URL retrieval without blocking the Emacs main loop,
  • autodiscovery of Exchange server settings from an email address, and,
  • implementing NTLMv2 authentication in Elisp.

I’m trying to get it into GNU ELPA so that other modes can always count on its core APIs being available from a default Emacs installation. I haven’t put it on GitHub yet, to keep it hidden from the ruthless efficiency of MELPA. I was impressed and appreciative when MELPA bundled slime-volleyball shortly after I released it. But I’d prefer to keep Excorporate in GNU ELPA only for now.

Anyway, I’m publishing this mode in the hope that it’s useful to other Emacs-and-Exchange users.

Emacs and TWiki

At work I have cause to edit a TWiki that my team uses for internal documentation. I wanted to use Emacs for this task, so that I wouldn’t need to interact with a web browser text area widget. I couldn’t find anything that did that, so I took the closest thing I could find, erin.el, a TWiki markup mode, and added connectivity to it via Emacs’s url package.

The result is a fork of erin.el that supports these new operations:

Log in: M-x erin-log-in
Edit a topic: M-x erin-edit-topic
Commit edits: C-c C-c
Cancel edits: C-c C-k
Log out: M-x erin-log-out

without ever leaving Emacs. Now with EWW to see the resulting pages, there’s no need to leave Emacs at all.

This being a fork, it doesn’t qualify for MELPA, and I can’t get in touch with the original author, so it will stay in limbo as a raw repo, without ever being packaged.

A while after I did this development, emacs-twiki-mode sprang up. It looks like it has some nice advantages, like orgtbl editing. If it used Emacs’s built-in URL handling instead of an external bash script I would probably switch to it. For now my erin.el fork works well enough for me. I do wish there were one monster Emacs mode that would handle all Wiki server implementations, including connectivity; a sort of Gnus for Wikis. Oh well.