uLisp on the SMART Response XE

The Lisp Badge mini computer has turned out to be quite useful and fun for little hardware hacking projects. Its designer, David Johnson-Davies, suggested that the SMART Response XE would make a good off-the-shelf uLisp computer, eliminating the need to build one from scratch.

I ordered a few SMART Response XEs from an auction site to see what was possible. I found Larry Bank‘s excellent Arduino library and fdufnews‘s schematics, which provided a great starting point. With guidance from David, I completed an initial uLisp port:

uLisp 4.1 running on the SMART Response XE

To load the code, I use an ISP programmer and a special PCB with POGO pins:

ISP POGO programming of the SMART Response XE

On Debian, I run:

make ispload

to load LispBadge.ino without a bootloader.

The SMART Response XE uses the ATmega128RFA1 microcontroller, which has a ZigBee IEEE 802.15.4 transceiver. David and I are discussing adding uLisp functions to make use of this capability.

Mezzano on Librebooted ThinkPads

I decided to try running Mezzano on real hardware. I figured my Librebooted ThinkPads would be good targets, since, thanks to Coreboot and the Linux kernel, I have reference source code for all the hardware.

On boot, these machines load Libreboot from SPI flash; included in this Libreboot image is GRUB, as a Coreboot payload.

Mezzano, on the other hand, uses the KBoot bootloader. I considered chainloading KBoot from GRUB, but I wondered if I could have GRUB load the Mezzano image directly, primarily to save a video mode switch.

I didn’t want to have to reflash the Libreboot payload on each modification (writing to SPI flash is slow and annoying to recover from if something goes wrong), so I tried building a GRUB module “out-of-tree” and loading it in the existing GRUB. Eventually I got this working, at which point I could load the module from a USB drive, allowing fast development iteration. (I realize out-of-tree modules are non-ideal so if there’s interest I may try to contribute this work to GRUB.)

The resulting GRUB module, mezzano.mod, is largely the KBoot Mezzano loader code, ported to use GRUB facilities for memory allocation, disk access, etc. It’s feature-complete, so I released it to Sourcehut. (I’ve only tested it on Libreboot GRUB, not GRUB loaded by other firmware implementations.)

Here’s a demo of loading Mezzano on two similar ThinkPads:

For ease of use, mezzano.mod supports directly loading the mezzano.image file generated by MBuild — instead of requiring that mezzano.image be dd‘d to a disk. It does so by skipping the KBoot partitions to find the Mezzano disk image. The T500 in the video is booted this way. Alternatively, mezzano.mod can load the Mezzano disk image from a device, as is done for the W500 in the video. Both methods look for the Mezzano image magic — first at byte 0 and, failing that, just after the KBoot partitions.

I added the set-i8042-bits argument because Coreboot does not set these legacy bits, yet Mezzano’s PS/2 keyboard and mouse drivers expect them; at this point Mezzano does not have a full ACPI device tree implementation.

Excorporate 1.0.0

I released Excorporate 1.0.0 recently and declared the API stable. I was careful not to break API compatibility throughout Excorporate’s development so the API version stays the same at “0”.

The project is now in a state where it does everything I want it to do, API-wise. The UI is still missing features like meeting creation, but I just call the required Elisp functions when I need to, referring to the “API Usage” section of the Info manual.

I think there’s a lot of potential to create nice user interface features with Excorporate’s API — like a scheduler that shows people’s availability with ASCII-art bars, usable on a TTY. The included Org, diary and calfw front-ends show real-world usage of the API. I hope people send patches for new user interface features and keybindings, and contribute new authentication methods. I’ll continue watching for bug reports.

Quickly Start a Common Lisp Script

So you want to write a utility script, and you want to write it in Common Lisp. I created a template Common Lisp script called start.lisp. It’s meant to be renamed and hacked up but it provides a starting point for a new Common Lisp script, with some utility libraries included.

Here’s a “one-liner” that you can paste into a text editor, verify visually, then copy-n-paste from the editor into a terminal. This will get you up and running on major distros:

sudo apt install sbcl || \
sudo dnf install sbcl || \
sudo yum install sbcl && \
git clone --recursive https://git.sr.ht/~fitzsim/cl-starter-script && \
./cl-starter-script/start.lisp --help

It uses some shell tricks I found on the EmacsWiki, and a UIOP1 feature ((uiop:argv0) with __CL_ARGV0) I discovered via the impressive cl-launch project. It’s too bad cl-launch isn’t more widely packaged, since it seems like a good idea.

I may add more utility features to this template repository, but I’ll also try to keep it simple and self-contained. Self-contained meaning after the initial git clone there’s no need to go back to the Internet for more libraries.

This is only meant for utility scripts (and for me as a learning exercise for ASDF and Common Lisp packages). It is available in my Sourcehut, and mirrored to Microsoft GitHub.

For bigger projects, check out Quickproject, and for installing newer Common Lisp implementations than your operating system provides, consider Roswell.

Thanks to Didier Verna for help with ASDF and for quickly incorporating into CLON some new features I requested.

1. “Utilities for Implementation- and OS- Portability”

Hosting Jitsi on ppc64le

I recently tried self-hosting Jitsi on Debian on the Talos II.

I had to apply some small workarounds for ppc64le, so I thought I’d post them here.

The cause of the first issue, no audio or video in a call, was reported in the Jitsi Videobridge log, /var/log/jitsi/jvb.log:

Exception in thread "Smack-Single Threaded Executor 0 (0)" java.lang.UnsatisfiedLinkError: /tmp/nativeutils5300293642203108/libjnisctp.so: /tmp/nativeutils5300293642203108/libjnisctp.so: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a Power PC 64 LE-bit platform)

To fix this, you have to build libjnisctp.so for ppc64le and substitute it for the x86-64 version of the library. Unless this is fixed upstream, you’ll have to re-do the steps starting from wrapper=[...] if you upgrade the jitsi-videobridge2 package.

git clone https://github.com/sctplab/usrsctp.git
git clone https://github.com/jitsi/jitsi-sctp
cd jitsi-sctp/
cp -R ../usrsctp usrsctp/
mvn package -DbuildSctp -DbuildNativeWrapper -DskipTests
wrapper="$(dpkg -L jitsi-videobridge2|grep jniwrapper-native)"
sudo cp "${wrapper}" "${wrapper}.bak"
cp "${wrapper}" ./tohack.jar
mkdir hacks
cd hacks
jar xf ../tohack.jar
cp ../jniwrapper/native/target/libjnisctp-linux-ppc64le.so \
jar cf hacked.jar *
sudo cp hacked.jar "${wrapper}"

I also had to help along the installation of luajwtjitsi, a dependency of Prosody, which at first errored out with:

sudo luarocks install luaossl
Installing https://luarocks.org/luaossl-20200709-0.src.rock

Error: Could not find library file for CRYPTO
  No file libcrypto.a in /usr/lib
  No file libcrypto.a in /usr/lib/x86_64-linux-gnu
  No file libcrypto.so in /usr/lib
  No file libcrypto.so in /usr/lib/x86_64-linux-gnu
  No file matching libcrypto.so.* in /usr/lib
  No file matching libcrypto.so.* in /usr/lib/x86_64-linux-gnu
You may have to install CRYPTO in your system and/or pass CRYPTO_DIR or CRYPTO_LIBDIR to the luarocks command.
Example: luarocks install luaossl CRYPTO_DIR=/usr/local

luarocks needs a hint as to the ppc64le library locations for luaossl, a dependency of luajwtjitsi.

sudo luarocks install luaossl \
	CRYPTO_LIBDIR=/usr/lib/powerpc64le-linux-gnu \
sudo luarocks install luajwtjitsi

I’m impressed with Jitsi; its self-hosting documentation is straightforward, and once it is installed, the video bridge works smoothly. I didn’t do detailed comparisons, but video call quality seems as good as any of the centrally-run services I’ve used.

Pocket Lisp Computer

I recently built three Lisp Badge computers with some help from my kids. I bought a hot air soldering station and learned TQFP soldering. The kids did some through-hole and SMT soldering and really enjoyed it!

The hardware assembly and debugging process was really fun, other than worrying several times that I had put too much heat into a component, or set the wrong programmable fuse. During that phase I received some advice from the board’s designer, which really helped.

I’ve learned from the hardware people at work to always order extra parts, and I did, including an extra PCB. I was half expecting to damage stuff while learning, so I was really happy that we ended up with all three boards fully working, after locating and fixing some cold solder joints.

It was challenging as DIY projects go, since the Lisp Badge is not available as a kit. But ever since I saw the Technoblogy post about it, I knew I had to attempt building one, and it was worth it. Other than the display, compatible parts were all available from Digi-Key, and I got the PCBs from OSH Park.

The result is a really neat little computer. Here is a picture showing “big text” support that I added:

Three Lisp Badge computers displaying (lisp-badge) in large text split across the screens.

I also added support for building standalone on Debian (Arduino-Makefile), made the audio buzzer work, and wrote an example of how to play a tune on the buzzer. I published the changes to my sourcehut.

It’s fun to try writing small programs on the badge itself, within the constraints of its minimal uLisp REPL.

Lisp on Talos II

I’ve been following the status of Lisp on ppc64le lately.

I’m running ppc64le Debian sid. Just after I had set up my system, I did some experimentation with what Debian packages had to offer. ECL was the only Lisp that worked, so I started using it for various projects. (I’ve since learned on #sbcl that CLISP built from source is also a good option.)

Ideally I wanted to be able to use SBCL, so I wondered how far into an SBCL bootstrap I could get with ECL as the host compiler. A few months ago, I found the answer was not very far.

Since then though, the SBCL maintainers have been hard at work on two fronts: making SBCL bootstrap against ECL again (on any architecture) and porting SBCL to PPC64.

Recently with a few minor local changes, I was able to bootstrap SBCL natively on my Talos II starting with ECL, then build SBCL again with the bootstrapped SBCL. I had to build ECL from the tip of its development branch, but it has a nice build system and, being based on C, doesn’t require any bootstrapping steps.

So the SBCL ppc64 port is really shaping up; hopefully the next release will advertise ppc64le support. For now, plenty of packages work for me already, like SILME and Quicklisp.

I’d also like to try the same bootstrap procedure on ppc64be ABI version 1 (Debian sid) and ppc64be ABI version 2 (another distro, probably Adélie). I’m working on setting up an environment with qemu-system-ppc64.

Anyway, I’m happy that I can now use SBCL on my Talos II.

An excellent machine

I finished building my Talos II system, and I decided to post my thoughts on it here.

This is an excellent machine, the best workstation I’ve ever used. It’s very fast — it compiles Emacs and the kernel in 2 minutes, and just spins up the fans a little bit while doing so (under normal operation it’s quiet). And it’s in a completely different league in terms of openness than any other computer I’ve owned (and any other high performance computer on the market today).

The system came with a nice user’s manual, and full schematics! I’ve already referred to the schematics to set up the pinmux for a rarely-used serial port so that I could use a TTL serial cable I had lying around (I submitted the device tree patch to Raptor).

It’s really two computers in one: the “baseboard management controller” (BMC), a low power ARMv6, with a full distro on it, and the main POWER9 CPUs. The BMC boots up as soon as you plug in the power supply, before you even press the power button. (It would be nice if there were an ncurses top-like viewer for fan speeds and temperatures that I could leave running on the BMC serial console, but I haven’t found such a thing yet.)

It has serial ports everywhere! One right into the main CPU, and two into the BMC. This is great for low level development, e.g., breaking into bootcode at an early stage.

I left the machine running overnight after first booting it. My neighbourhood had a power glitch, and in the morning I discovered the main CPU was off, and power cycling it via the BMC wasn’t working. Before unplugging and plugging back in, I asked on #talos-workstation, and it turns out I had hit a bug in the first-release firmware. With special commands I was able to power cycle the main CPU just via BMC software (no unplug required). Wanting to know the details, I asked if there was a data sheet for the chip I was interacting with. The amazing thing, from an openness perspective, is that one of the Raptor engineers instead referred me directly to the Verilog source code of the FPGA handling the power sequencing. No searching for datasheets, no black box testing, just straight source code (which is recompilable using an open source FPGA toolchain, BTW.)

It’s so refreshing to not have to do reverse engineering and speculation on opaque things when something fails. On this machine, everything is there, you just look up the source code.

I’ve always disliked the inflexibility and opacity of BIOS/EFI in the x86 world. IBM has done an amazing job here with OpenPOWER. All the early boot code is open, no locked management engines or anything like that. And they’ve adopted Petitboot as the bootloader. It runs on a Linux kernel, so I was able to bootstrap via deboostrap over NFS by building everything within the bootloader environment. Running a compiler in a boot environment is surreal. Even with free options on x86 like libreboot or coreboot, and GRUB, the boot environment is extremely limited. With Petitboot at times I wondered if I even needed to boot into a “desktop” kernel (at least for serial-only activities.)

Now I’m setting up my development environment and I’m learning about the PPC64 ELFv2 ABI, with a view toward figuring out how to bootstrap SBCL. I feel lucky that I got a POWER9 machine early while there are still some rough edges to figure out.

Talos II available for pre-order

The Talos II is now available for pre-order. It is the more affordable, more power-efficient successor to the Talos I machine I wrote about in a previous post.

This is a very affordable machine for how powerful it will be, and there are minimal mainboard + CPU + RAM bundles (e.g., this one), around which one can build a workstation with more readily-available parts. I’ve placed an order for one of the bundles, and will buy the chassis and GPU separately (mainly to avoid high cross-border shipping fees for the full workstation).

The Talos II is an important machine for Free Software, and will likely be RYF-certified by the FSF. Pre-orders end September 15th!

Talos Secure Workstation

This is a very important machine that really deserves to get built. Anyone who cares about Free Software should consider funding this project at some level, and spreading the word to their friends. If this project succeeds, it will bootstrap a market for new, owner-controlled performant desktop machines. If it fails, no such computers will exist. The project page and updates explain the current (rather depressing) state of general purpose computing better than I could, so take a look.