Matthew Garrett

Simple conference organisation suggestion

Sat, 11 Jul 2009 05:32:17 GMT

If you find the comments discussed here unacceptable, don't invite RMS to keynote at your conference without an explicit apology and expression of understanding from him beforehand. I'm seriously at the end of my patience with people being unwilling to call others on behaviour they perceive as unacceptable. Either make it obvious or accept that people will treat your failure to do so as implicit support.

Edited to clarify that I think the apology should be from RMS, not from people who invite him to keynote

Chrome OS

Fri, 10 Jul 2009 15:47:11 GMT

It turns out that I was entertainingly wrong a while ago, though I persist in claiming that this is an utterly ridiculous idea and I should be forgiven for thinking that Google were sane. The whole thing really still doesn't make sense to me. Worthwhile support of hardware is difficult. I'm going to take it as a given that Google aren't going to claim to support arbitrary hardware. People who would never otherwise try Linux will install it, and the state of many Linux drivers is sufficiently poor that it'd do a great deal to damage their brand. The logical assumption is that it'll be available pre-installed on devices where Google have worked closely with the hardware vendors.

Which concerns me somewhat. History isn't filled with compelling examples of this. Xandros's low-level support for the Eee mostly seemed to consist of a pile of shell scripts made of cheese and failure. The bizarro-Linux on the hilariously dodgy MIPS-based netbook I have is about as functional as my wisdom teeth. The best example of a Linux vendor working with OEMs is probably Canonical, and their enthusiasm for merging hardware support code in their OEM-specific distributions has led to things like touchpad gesture support based on using a known security hole or drivers that reimplement one that's already mainline.

What I'm trying to say here is that pretty much every desktop Linux product based on cooperation between OEMs and an existing Linux vendor has been built on top of a tower of shit. That's partly because it's a hard problem, but it's also because most OEMs produce dreadful Linux code and the Linux vendors don't have the resources to rewrite it in a clean way in the timescale permitted between hardware being finalised and shipping product. I haven't seen Google recruiting a larger than normal number of people with Linux distribution experience lately, so I suspect that the situation may be the same there. This is probably fine if the number of products is relatively small - there's an opportunity to QA them sufficiently to ensure that the rough edges underneath don't accidentally take someone's hand off. But otherwise there is a genuine risk that poor-quality devices will appear with Chrome OS and people will blame Google for the poor user experience.

So it seems like a risk for Google. Either there'll be a small number of devices and the same vague level of discontent that surrounds the fact that the number of shipping Android devices doesn't seem to have reached expectations, or a large number of potentially crappy devices. What's the payoff? It's pretty clear that this is going to be based heavily around Google's web apps, possibly with disconnected operation. That gets Google a lot of lockin. It also neatly sidesteps the entire disaster that has been providing application add-ons and updates for netbook Linux distributions. But it seems that Google could have achieved that by partnering with a distribution that already has experience in this field rather than going it alone.

I remain unconvinced that this is a sensible decision for Google. But then, I've already demonstrated that I don't have the faintest idea what's going on. Which probably puts me in the same field as most of the analysts I was bitching about before. Zounds. The irony.

What does the desktop want from the kernel?

Mon, 06 Jul 2009 18:07:00 GMT

I'll be running a session on Wednesday at GCDS to find out what desktop developers would like to see from the kernel. There's a lot of interest in making things easier for file indexers, but if anyone has other problems that could be made easier with some level of kernel support then please turn up. No precise time or location yet, but probably around 3PM at the university. More details forthcoming.

Mon, 06 Jul 2009 17:18:10 GMT

One of the strengths of the open source community is that so much happens in the open. It's generally easy to find out what's happening in a project and directly interact with the developers. Code is out in the public. People frown upon closed discussion and implementation. But there's also a cost. Personality conflicts get hidden in the corporate world. We air them in public. And while in some ways that's arguably an advantage, it also results in things like this.

Now, to be fair, I've never been an especially big fan of Sam's work. His journalism generally leans towards lazy sensationalism rather than any attempt to actually understand the issues. He's not especially well versed on the basis of free software (see his assertion that the difference between the cathedral and bazaar development models is about project leadership rather than source code availablity and how people participate). He's managed to mischaracterise my opinions in the past, which means trusting his characterisation of anyone else is somewhat difficult. But that's not the point. Sam's article isn't about facts or analysis. It's about crucifying someone for expressing an opinion that Sam disagreed with.

I don't know Anirudh. From his website, he sounds like a pretty typical young hacker. He's contributing to a project that interests him. He's self taught. He holds strong opinions. And, last week, he published a rant on a topic he cared about - specifically his feeling that the patent concerns about Mono shouldn't prevent people from developing in it, and that RMS's statements about not using Mono harm the free software community more than they help it. I don't agree with all of his arguments. I don't think it was a hugely well structured rant. And, shockingly, telling RMS to fuck off isn't going to achieve a great deal.

But the point wasn't to change the world. People like voicing their opinions. I've done so several times on several occasions on a wide range of topics. People have disagreed with me. People have voiced concerns about the way I've expressed myself. People have flat out told me to stop being a cock. But nobody has attempted to tell me that I haven't earned the right to express that opinion. Nobody has expended four pages to tear me apart for daring to criticise another member of the community. Nobody has dared to call me a coward for deciding that I'd gone too far and modifying or retracting something I've written.

Yet that is precisely what Sam Varghese has done here. And let's be clear here - the use of the word "fuck" is a red herring. Sam is publically humiliating someone because he has his own agenda. He's picking on someone smaller than him because he can. He's explicitly stating that anyone arguing in favour of Mono can expect to be thoroughly abused in front of a large audience. In Sam's world I don't get to criticise the shockingly distasteful sexist remarks RMS made during his GCDS keynote because I didn't write emacs. I don't get to point out that ESR's understanding of racial genetics and natural selection are fundamentally flawed because I don't maintain the jargon file. I don't get to say that I think Ted Tso was wrong about ext4's semantic changes because I've never written a filesystem. And if I do any of these things, I can expect Sam to pop up and do his best to destroy my reputation.

Of course, he won't. Because in Sam's world, it's not actually about whether anyone has achieved great things. It's about whether the targets of his vitriol will be able to stand up for themselves or not. And while the truth is that nobody in the community is above criticism and nobody needs to earn their right to disagree, I doubt that Sam is ever going to show the courage that Anirudh did and publish a public retraction of his astonishing attack.

Shame on you, Sam Varghese.

Thu, 25 Jun 2009 21:38:42 GMT

I've had hard drives die before, but this one seems to be doing it in an especially pathological way. It started with the kernel throwing irq timeout errors and then stepped up into actual read errors, culminating in a corrupt journal, a read-only block device and a forced fsck. It's been behaving a little better since then, though occasional io stalls (without any kernel error) suggest that it's having to repeatedly retry some sectors. SMART says it's all fine, so obviously I'm backing it all up now before a new disk arrives tomorrow and I can sort out access to the data centre. Email might be a bit spotty for me until then.

Turns out that running a 2.5" PATA drive for approximately 4 straight years may not have been the best of ideas. Who'd have guessed?

Wed, 24 Jun 2009 14:07:17 GMT

I spent last week in the US, during which I discovered that my preconceptions of Albany as a post-industrial wasteland with no redeeming features were incorrect (it's actually a post-industrial wasteland with some decent bars), made minor contributions (mostly in the form of pesto) to a prize winning chalk picture and cycled on the wrong side of the road for the first time in a ridiculous number of years.

But more relevantly, I spent most of the week trapped in Westford. One of the things I've been looking into lately is USB autosuspend, a kernel feature which allows devices to be powered down when they're not in use. This is especially interesting for USB, since it's a poll-based protocol. If the end device is powered up then bus traffic will be generated, even if everything's idle. USB autosuspend allows this to be avoided and thus saves a worthwhile amount of power.

However, drivers need to support USB autosuspend before it's useful. Further, devices need to be able to cope with it. Some older kernel releases had autosuspend enabled by default, leading to all kinds of fun as people discovered that their scanners and printers had stopped working. This was fixed by leaving autosuspend disabled by default for most hardware. The first component that I'm working on is support for drivers to indicate whether or not a given piece of hardware supports autosuspend, allowing it to be enabled by default for that kernel. Handling this at a per-driver level means we shouldn't end up with obnoxious white or blacklists. The second is adding support for autosuspend to more drivers. Right now I'm concentrating on hardware that's common in laptops. There's support for autosuspend in the qcserial and uvc drivers, and some experimental (and unmerged) patches for bluetooth and some other modem drivers. Palm have just released their kernel code, which includes autosuspend support for the cdc-acm driver. I'm working on cleaning these up and getting them into a mergable state, and part of what I was doing in Westford was testing that various pieces of hardware work correctly. The good news is that they seem to, and with a bit of luck we'll try shipping support for a lot of this in F12. If that doesn't end up causing real world problems then it ought to be safe to merge it all to mainline.

The final component of this is handling autosuspend on devices with userspace drivers. Our only real choice there is for packages to ship udev fragments that enable it for working hardware. I've added support to the fprint package in rawhide, which means that fingerprint readers should now be powered down unless they're opened. Nobody seems to have complained yet, so I'm cautiously optimistic that this can be sent upstream without any problems.

Tue, 23 Jun 2009 20:41:44 GMT

It's now been over 6 months since Poulsbo hardware with Intel's GMA500 graphics core started shipping in volume. And we're still utterly lacking in any sort of worthwhile driver. It's an impressive turnaround from the recent days when the straightforward recommendation for mobile Linux hardware was "anything that has lots of Intel stuff in lspci", and while the Poulsbo situation in itself doesn't change that hugely it's potentially symptomatic of a worrying trend within parts of Intel.

The first thing to realise here is that, like most large companies, Intel consists of a large number of business units with different priorities. Their open-source technology center has historically had responsibility for providing Linux support for hardware, but this obviously depends on other business units cooperating with them. And there's strong evidence that many of those business units don't get it.

There's been signs of this for some time. Back before the days of the Intel X.org driver gaining native modesetting support, some people ran the Intel embedded graphics driver. This was (is?) a closed X driver that was able to provide native modesetting on platforms that could only otherwise be run at incorrect resolutions. One business unit was shipping a driver that was more functional than the official Intel Linux driver. To the best of my knowledge, none of that code was ever used in the rewritten Intel driver that now provides the same features.

Poulsbo is another example of this. Intel wanted a low-power mobile graphics chipset and chose to buy in a 3D core from an external vendor. IP issues prevent them from releasing any significant information about that 3D core, so the driver remains closed source. The implication is pretty clear - whichever section of Intel was responsible for the design of Poulsbo presumably had "Linux support" as a necessary feature, but didn't think "Open driver" was a required part of that. There's not a lot any other body inside Intel can do once IP-limiting contracts are signed and the hardware's shipping, but it ends up tarnishing the good reputation that other parts of Intel have built up anyway.

And while Poulsbo is the most obvious example of this to date, it's not the only one. Intel recently decided to make the EFI development kit discussion lists private. Various drivers for Moorestown (the followup platform to Poulsbo) have been submitted to the Linux kernel, and while they have the advantage of being GPLed they have the disadvantage of being barely above the level of typical vendor code. Objections that chunks of them simply don't integrate into Linux correctly has done little to get these problems fixed - I still have no real idea how the runtime interface to power management on the SD driver is supposed to be used, but I suspect the answer is probably "badly".

This all makes sense if you assume that there are large groups of people in Intel who don't talk to each other. But to the casual observer it just looks schizophrenic. Explaining to an irate user that the Intel who shipped a closed Linux graphics driver is only barely the same Intel who contribute so much to architectural improvements in the Linux graphics stack doesn't make their hardware work. And while all of this confusion is going on, Intel's competitors are catching up. Atheros are now making significant contributions to the state of Linux wireless. AMD are releasing graphics chipset documentation faster than Intel, and radeon support is improving rapidly.

Is the future going to be one where we can no longer simply say that Intel hardware will Just Work? Is their work on Moblin (easily the most compelling Linux UI for netbooks) going to be wasted on the broader Linux community because it'll mostly end up running on hardware that's not supported by the mainline Linux kernel? Does Intel have a real commitment to open source, or is that being lost in the face of short-term requirements?

Intel need to demonstrate that they have a company-wide understanding of what Linux support actually means or risk losing much of what they've earned over the past few years. I'm desperately hoping that Poulsbo and what we've seen so far of Moorestown are the exception, not the future norm.

Palm Pre

Wed, 10 Jun 2009 14:41:10 GMT

I'm in the UK and even if I weren't I wouldn't be using a CDMA phone, but the Palm Pre is undoubtedly an interesting device and so I've spent a short while looking at the root filesystem that can be extracted from the downloadable ROM image. There's some interesting things there. Bear in mind that this could be some QA internal image, so chunks of what I talk about may be wrong - on the other hand it seems to have been produced in May, so it's probably pretty close to what's shipping.

Hardware

As others have figured out by now, the Pre is codenamed Castle and there's some references to another device called Pixie. This is presumably the Eos device that's expected to end up with AT&T. The other interesting thing is that the image contains modem firmware for both CDMA and UMTS. They're shipped as tarballs - extracting them gives you something that looks awfully like the firmware for the Qualcomm Gobi dual-CDMA/HSDPA chipset used in a bunch of modern laptops. The core firmware is an ELF object for 32-bit ARM. It's got references to Qualcomm in it and it's also got an embedded RSA signature which presumably cuts back on the potential for interesting hacking. The

/usr/bin/PmModemUpdater -e < /usr/lib/modem/castleumtsfw.tar

command, executed as root, should flash the UMTS firmware to the modem (castlecdma_evt1_fw.tar is the CDMA firmware). However it should be noted that the MSM6801A baseband used on the Pre seems to be a CDMA-only chip and there's no obvious place on the board for a SIM socket. My assumption is that the GSM models will have a very similar baseband with different radios. Unlocking the device will presumably be similarly difficult to the iphone - someone will need to find a flaw in the Qualcomm firmware that allows the network lock to be skipped. The PmModemFactory command will let you unlock the phone, but you'll need the appropriate code to do so and can (I guess) permanently lock it if you enter the wrong code too often.

The internal flash appears to present as mmc for some reason.

Software

A lot of the software on the Pre is GPLed, and Palm are therefore obliged to provide copies of the appropriate source code to anyone who receives the software (note that receiving the device is not required - if the software is downloadable then the source offer must be made available to anyone who receives the software). The source code is not currently downloadable - instead Palm have included a written offer to supply the source code on request. This satisfies the GPL, but I can't be bothered doing that for the moment so all of this is purely based on examining the binaries.

The full list of applications extracted from the open source information documented included is here.

General

The Pre appears to be an OpenEmbedded-based system. It uses ipkg for package management and the majority of the basic userspace apps are all from Busybox.

Kernel

The Pre's running 2.6.24, and there's various references to Windriver. There's an internal git repository codenamed rockhopper, which is a little odd - rockhopper's the codename for a MIPS-based NEC evaluation board. It's probably coincidental, given that there's only so many species of penguin available. The version string includes "joplin" - there's evidence of this in mailing list posts dating back to last year, so it's not clear whether "joplin" is the entire WebOS kernel project or a codename that covers both the castle and pixie platforms.

The hardware-specific drivers look well-integrated, for the most part using the standard kernel interfaces (backlight, led and so on) - this is a marked contrast to Android, which tends to go its own way in this respect. There's a driver for the OMAP DSP, the exmap code for analysing application memory usage, oprofile for detailed profiling analysis, some crypto code, a driver for an SDIO-attached Marvell wifi chipset and support for being a USB gadget (ie, something that appears as a USB device when plugged into a USB host). Side note - depending on whether the system is a castle or a pixie, the g_composite gadget driver is given a different ID to let the host OS differentiate between the two. castle will appear with a USB id of 0x8002, and pixie 0x8012.

There's nothing else very interesting looking about the kernel. The source code should reveal more.

Booting

The Pre uses upstart as its init application. In contrast to mainstream Linux distributions that still mostly use upstart in sysvinit emulation mode, the Pre appears to be almost entirely based on native upstart events. One of the things they've used this for is to automatically stop various services when the update service is started. Beyond that there's nothing exceptional here, but looking at etc/event.d does give insight into what's running on the device.

Userland

In contast to Android, the Pre uses the standard GNU C library and associated userland. Filesystem layout is pretty typical Linux, with / containing the low level binaries and /usr containing the rest of the OS. The only slightly weird thing is that various utilities exist in both their busybox and full forms, and I'm not entirely clear on why it's shipping things like fsck.ext4 though I guess you could use an ext4 sd card or something.

Audio is handled through pulseaudio, while media is handled by gstreamer. There's a large collection of codecs for the DSP including WMA and h.264, but no obvious ogg support despite libogg and libvorbis being mentioned in the list of open source software shipped. Future plans? Pulled partway through development? Unclear.

ipod emulation

The Pre pretends to be an ipod by switching into a different USB profile (and thus giving the right USB information) and setting up a filesystem that looks like an ipod. It provides a SysInfoExtended file that looks like an ipod. The only terribly interesting things about this are the list of formats (AIFF, MP3, WAV, AAC, AppleLossless, Audible, H.264, MPEG4 and H.264LC), the fact that it supports album art, the fact that it doesn't support Apple's DRM and that the firewire GUID looks like it's probably the same on all Pres. Could Apple break this support in a future version of itunes? Yes, but none of this support is hardcoded in the Pre's hardware - Palm could provide an update that matches. I don't see any real benefit to Apple in breaking the support, and if it could be shown that they were doing it deliberately then there could be potential legal issues.

The code actually looks generic enough that it could probably be made to emulate other USB mass-storage based media players if you wanted to. I have no idea why you'd want to.

misc

There's a PalmOS ROM image for use in the emulator. usr/share/accountservices contains some files with interesting links in - one of them gives a full dump of the PHP setup on a Palm server, which gives some indications of their internal network setup and other things that are probably harmless but companies usually get paranoid over. The flasher application is called Trenchcoat. There's setup code for traffic shaping support - I haven't checked whether this is for application QoS on the device or whether it's intended for use with tethering.

Binutils is shipped. The Pre has an ARM assembler installed by default. The mind faintly boggles. It looks like there's tethering support via the Bluetooth PAN profile (see usr/bin/PmConnectionManager) as well as via USB. IPC is dbus-based rather than using something custom like Android's binder.

Overall

I'm impressed. There's a few rough edges and some obvious short-term hacks, but overall the Pre has the appearance of being a well-engineered distribution. It's recognisably Linux in a way the Android isn't. Since it seems to be possible to gain root by entering the developer mode, I suspect that modifying the firmware image isn't especially difficult. It'll be interesting to see what happens when GSM ones appear.

There's some more information at the pre dev wiki.

Antler customer service

Tue, 09 Jun 2009 15:24:03 GMT

In what makes a surprising change from my normal bitching about customer service, I called Antler last week after the rubber coating on the wheels of my luggage disintegrated. I've had that bag for almost 5 years and travelled something like 300,000 miles in that time, so it wasn't an especially surprising result - but the bag had a 7 year warranty covering defects in materials and manufacture, so they picked it up last week and dropped off a new one today. I'm impressed.

Thinkpad mixers

Mon, 04 May 2009 00:53:57 GMT

Older (up to the *60 series) Thinkpads have a slightly odd mixer setup. There's a mixer stage that sits between the software controllable mixer and the speakers. This is controlled by the firmware in response to the volume keys being hit and can't be prevented by the OS. This is unfortunate, since users are quite enthusiastic about seeing an on-screen display when they hit various keys and if we do the obvious thing and map the volume keys to KEY_VOLUMEUP and KEY_VOLUMEDOWN then they get the OSD but also get their software-controlled mixer changed at the same time as their firmware-controlled mixer. This causes boundless confusion.

Lorne Applebaum recently sent a patch that added an ALSA control to represent the Thinkpad mixer. This means that we can now read and control the Thinkpad-specific hardware without userspace needing any special knowledge. I've gone a little further and sent a followup patch that sends ALSA notifications on button presses. Lennart suggested that it would also be helpful to export dB values and wrote a nice app that measures them - that means that Pulseaudio will be able to set the mixer volume correctly by default.

The only missing part of the puzzle now is getting the OSD back. We still can't map the keys to KEY_VOLUMEUP and KEY_VOLUMEDOWN because that'll still result in two mixers being changed at once. We can't add new keys to indicate that it's a notification event rather than a command, since X can't deal with keycodes greater than 255 (actually 243, since it adds 8 to every kernel keycode for historical hilarity) and we've run out of space. We can't hook off the ALSA notifications because that'll result in the OSD popping up in response to someone dragging a slider in a volume control application. The easiest thing might be to add another ALSA notification that's only triggered if the notification was generated in the kernel and then have userspace listen for that.

Newer Thinkpads appear to have done away with this mixer setup entirely and just send standard volume events. This is a Good Thing.

In other news, I'm off on holiday to Australia for a couple of weeks so may be a bit slow in replying to things.

Extended Qualcomm Gobi features

Sun, 19 Apr 2009 00:32:55 GMT

And while I'm on the subject of bitching about hardware support, I had some time to poke at the Qualcomm Gobi hardware a bit more last week. With the aid of Dan Williams of Network Manager fame we identified[1] that the hardware actually does rather more than is obvious from the qcserial driver. There's two extra ports - one is for diagnostics, and the other claims to be an NMEA port. Poking support for these into qcserial is easy enough, but they don't seem especially chatty. The more interesting factor is that the fourth interface on the endpoint is a network device. This isn't unusual in modern hardware - PPP isn't a terribly helpful layering for data transfer, so most recent high speed data cards also provide support for some kind of network device. If you're lucky it's a spec-compliant cdc device. If you've got a gobi then it claims to be a vendor defined class and type and the .inf file for the driver claims that it's a proprietary cdc device. Testing this got cut short by us not actually having an account on Verizon[2], so we don't have any data dumps to work out what the framing looks like.

Interestingly, doing USB dumps revealed that Windows seemed to do everything via the network device. We didn't see any traffic from the other ports, even when attempting to turn on the GPS. So it may also have some magic out of band configuration that doesn't simply use AT commands.

If anyone has this stuff working under Windows, it'd be interesting to try getting some dumps to try to work out what's going on here.

[1] By the simple process of opening the device manager under Windows. Ahem.

[2] The test machine was a Vaio P, which came with a gobi but no SIM slot. Thanks, Sony. Thony.

Intel's commitment to open source driver support

Fri, 17 Apr 2009 00:14:18 GMT

Update: It turns out that the original PSB code is still on the moblin site - I just failed to notice that there's two pages of git repositories. I fail. Last update is from March 2008, though.

Edited again: An up-to-date version of the drm code is available as part of the Moblin kernel SRPM - repo.moblin.com isn't obviously linked off the Moblin website, but I should have thought to find and check their source repository.

By and large, working with Intel is a pleasure. In fact, a while ago I said so in a fairly obvious way. So it's a shame that the support for Poulsbo is still an absolute fucking mess with absolutely no obvious end in sight. Intel's Moblin group (the people who seem to be closest to having responsibility for providing any public Linux support for that hardware, though it's really not clear if they've got any more say in this than I do) have a kernel team that don't want or need a git tree despite having someone who's working on forward porting the driver to modern kernels - probably a job that requires more than one person given what a godawful fucking mess the last public release was, but it'd at least be nice to see what the current state is. Especially since you don't even seem to be able to get the last public release - moblin's git repositories have been rearranged and I can't find the psb-kmod one on their gitweb any more.

The most terribly depressing thing about this is that development is clearly still happening internally. Ubuntu is still being sent tarball releases by Intel, not that the code's getting any better judging by the type of diff involved. This is made even more entertaining by the bug in question being private, leaving no indication whatsoever about what bugs this code is meant to fix or why.

Closed development, random private tarball drops to partners without any public releases or changelogs, no indication of any upstream release, kernel developers entirely unrelated to the development of the driver trying to get code merged so people can actually use the hardware they bought? I'd expect this of some random far-east vendor with no experience in working with Linux, but not a company that's consistently one of the top ten contributors to the kernel and has frequently touted their level of support. There's no meaningful support for Poulsbo. There's just a thin cardboard cutout that's been carefully placed in front of a hole filled with users' hopes and money.

Seriously Intel. Sort it the fuck out.

HP EliteBook 2530p

Sat, 04 Apr 2009 18:05:15 GMT

HP apparently decided my 2510p was cursed and replaced it with a 2530p which arrived earlier this week. A bit of playing later and it all seems pretty happy. Backlight control works fine, though you'll need hal-info git to get the hotkeys working out of the box. Sound needs alsa from 2.6.30 or for snd_hda_intel to be loaded with the "model=laptop" parameter. Suspend seems to work out of the box. Hotkeys and rfkill work with hp-wmi. The webcam is supported by the uvcvideo driver. The wwan modem needs qcserial from 2.6.30 and a tool for loading the firmware. It's happy to boot from EFI which probably saves a second or so of boot time. Wireless is iwl5100 and supported by the drivers from Intel. Video is GM45 and worked fine without any tweaking. I've no idea if the modem is supported or not, and I haven't tested the fingerprint reader yet. Grub was slightly unhappy about the idea of an EFI system partition that wasn't declared in a GPT, but that was a two line diff.

I've backported code to rawhide where necessary, so F11 should be released with full support for the hardware. It's quite a nice machine, and seems more solid than the 2510p - I'll see how it's going in a year or so.

Qualcomm Gobi firmware loading

Sat, 04 Apr 2009 16:26:31 GMT

Shortly after I posted this entry on the qualcomm modem driver, Alexander Shumakovitch posted a shell script to do the loading. I've turned this into a C app and added checksumming support and udev integration, and now things seem to be working happily. There aren't many device IDs in the qcserial driver upstream right now, but I've just sent a patch to add a bunch. If you have a device this works for and you had to add the ID manually, let me know and I'll update things. You can get it here.

The curious tale of the driver that did nothing

Thu, 02 Apr 2009 18:43:23 GMT

Several vendors are now shipping Qualcomm's Gobi chipset, a cunning dual CDMA-GSM wireless broadband device. There's a driver for it in the Linux kernel called qcserial which claims to support it.

Do not be fooled. This driver is a vile lie.

The hardware comes up in a dumb state and requires firmware to be loaded before it'll do anything. The only way to obtain this firmware is from a Windows driver. The only way to load this firmware is under Windows. This isn't helpful, especially given that it drops the firmware whenever you use rfkill or suspend or power down the machine. In fact, the only way you can use this driver is to boot Windows, let it load the firmware, reboot into Linux, get online and then never turn off or suspend your computer or the radio.

So, don't be like me - swearing viciously and trying to generate useful USB packet dumps in an attempt to get the hardware working. Known bad parts are the HP un2400 and the Dell wireless 5600 - sony also have a Gobi part that's used in the P-series machines, and Acer have one as well. I'll update this if I ever get anywhere with a firmware uploader, but until then remember that the presence of a driver in the kernel doesn't mean that you can actually do anything with the hardware. Fyalcomm.

Sat, 28 Mar 2009 03:39:41 GMT

"Open"Solaris. Especially ironic given that there's a 2-clause BSD implementation out there already.

Reducing disk use

Fri, 27 Mar 2009 00:21:51 GMT

UNIX filesystems generally store three pieces of timing information about files - ctime (when the file was changed in any way), mtime (when the file contents, as opposed to its metadata, was last changed) and atime (when the file was last accessed by any process). This is a usefully flexible system, but the semantics of atime can be troublesome. atime must be updated every time a file is read, causing a read operation to instead become a read/write operation. This results in a surprising amount of io being generated in normal filesystem use, slowing the more relevant io and causing disks to spin up due to atime updates being required even if the file was read out of cache. It also results in a lot of unnecessary activity on flash media which may reduce their lifetime.

One option is to disable atime updates entirely. The problem with this approach is that certain applications depend on atime. This is especially common in mail clients which compare atime to mtime in order to determine whether a mailbox has been read since it was last modified. So, unfortunately, disabling atime entirely is impractical as a default. Back in 2006, Valerie Aurora submitted a patch that worked around this issue. The new relatime option meant that atime would only be updated if it would otherwise be older than ctime or mtime. Mail clients became happy and the world rejoiced.

Unfortunately, it turned out that there was one other common case of atime being used. Applications like tmpwatch monitor files in /tmp and delete them if they appear unused. In this case, "unused" means "has an atime older than a certain date". Since merely reading files doesn't update the ctime or mtime, relatime wouldn't cause the atime on these files to be updated and tmpwatch would happily delete them - even if users were reading them on a daily basis.

Ingo Molnar submitted a patch to add a further heuristic to the relatime behaviour. With it, the atime of a file will be updated if it's older than mtime, older than ctime or (and this is the important one) more than 24 hours in the past. This deals with the tmpwatch case nicely, while still providing a significant reduction in the quantity of atime updates.

Fedora shipped this patch for several releases, and Ubuntu have used it by default since 8.04. Unfortunately there were some concerns over certain aspects of its behaviour (in respect to its interface as opposed to the relatime functionality itself) and it never got merged. I pushed a trimmed down version that purely implements the change to the relatime behaviour, and earlier today Linus merged it and a further patch that makes relatime the default behaviour on Linux.

Most users won't notice this change in behaviour at all, other than as a small improvement in io performance and a reduction in the number of drive spinups. For users that do have issues, a new strictatime mount option has been added - using this will require an updated mount command, but it's a trivial patch. I'd be surprised if there are any real world use cases that are negatively affected by this, especially since it's been default behaviour in several distributions for a while, but there's always the potential that someone will be tripped up by it. We'll see.

Tue, 24 Mar 2009 23:59:00 GMT

Today is Ada Lovelace day, a project to celebrate the women involved in technology and make it easier to give examples of female role models in the industry. I've had the opportunity to meet many women working on Linux over the past few years, and if there's one thing that's tended to overshadow the achievements and dedication to their work it's the sheer amount of effort they've had to go to in order to gain equal recognition. And that made me realise that in many ways, the woman who's had the greatest impact on my career is Hanna Wallach[1]. About ten years ago she spent a ridiculous amount of effort teaching me that it didn't matter how much I professed to be entirely free of sexism if I then proceeded to do things that implicitly excluded women from being involved in computing communities. I've gone to some amount of effort to repay that (contains profanity and a man with a raccoon tail covering his crotch), but I'm very aware of how different my life might have been if Hanna hadn't gone to the trouble of ensuring that I knew not to be a dick.

I think the Linux community has become more welcoming since then, and Hanna and people like her have been instrumental in helping that happen. However, we still have people being driven away by the behaviour of others or arguments that technical contributions are more important than social behaviour, and while that's true we need role models for social change just as much as we need role models for technical achievement. Thanks, Hanna[2]. The Linux world's a better place because of you.

[1] And while she's a role model of mine for social reasons, she's also got a PhD in machine learning, was British computer science student of the year in 2001 and came top of her MSc class at Edinburgh. So there.

[2] "Thanna"

ext4 and spinups

Wed, 18 Mar 2009 18:59:53 GMT

As a followup to my discussion of ext4, I did some trivial testing today. This consisted of generating a file, writing to it, closing it, opening another file, writing to it, closing it and then renaming the second file over the first. Then repeating about 10,000 times. fsync() wasn't called at any point. I used the /proc/sys/vm/block_dump knob to get an idea of when stuff was actually hitting disk. This was using the current rawhide kernel, which has the various workarounds for avoiding data corruption merged. The good news is that I was unable to get into an inconsistent state - the files always contained either the original or the new data, and the absolute worst case was having a zero length temporary file. The bad news is that while ext3 only touched disk when it reached the commit interval, ext4 touched disk for every single rename() call.

There appears to be a plan for dealing with this, but the current state of affairs is that ext4 will (under an obviously incredibly synthetic and unrealistic set of conditions) reduce the amount of time a drive can spend spun down. The next step is to attempt to measure this in a real world setup and see whether it's compensated for by the other new features.

Important typographical update

Wed, 18 Mar 2009 02:27:06 GMT

⸘Unicode 5.1 added the inverted interrobang‽

ext4, application expectations and power management

Sat, 14 Mar 2009 21:04:01 GMT

Edited to add: Sorry, it turns out that ext4 does now have a heuristic to deal with this case. However, it seems that btrfs doesn't. My point remains, though - designing a filesystem in such a way that a useful behaviour is impossible is unhelpful for the vast majority of application writers, even if it does improve performance in some other use cases.

Further edit: It's mentioned in the comments that btrfs will force a flush of data when a rename is performed as of 2.6.30. That prevents the data loss, but it means that we're still stuck with disk access when we'd be happy with that being batched up for the future. It feels like we're optimising for the wrong case here.

Original text:

There's been a certain amount of discussion about behavioural differences between ext3 and ext4[1], most notably due to ext4's increased window of opportunity for files to end up empty due to both a longer commit window and delayed allocation of blocks in order to obtain a more pleasing on-disk layout. The applications that failed hardest were doing open("foo", O_TRUNC), write(), close() and then being surprised when they got zero length files back after a crash. That's fine. That was always stupid. Asking the filesystem to truncate a file and then writing to it is an invitation to failure - there's clearly no way for it to intuit the correct answer here. In the end this has been avoided by avoiding delayed allocation when writing to a file that's just been truncated, so everything's fine.

However, there's another case that also breaks. A common way of saving files is to open("foo.tmp"), write(), close() and then rename("foo.tmp", "foo"). The mindset here is that a crash will either result in foo.tmp being zero length, foo still being the original file or foo being your new data. The important aspect of this is that the desired behaviour of this code is that foo will contain either the original data or the new data. You may suffer data loss, but you won't suffer complete data loss - the application state will be consistent.

When used with its (default) data=ordered journal option, ext3 provided these semantics. ext4 doesn't. Instead, if you want to ensure that your data doesn't get trampled, it's necessary to fsync() before closing in order to make sure it hits disk. Otherwise the rename can occur before the data is written, and you're back to a zero length file. ext4 doesn't make guarantees about whether data will be flushed before metadata is written.

Now, POSIX says this is fine, so any application that expected this behaviour is already broken by definition. But this is rules lawyering. POSIX says that many things that are not useful are fine, but doesn't exist for the pleasure of sadistic OS implementors. POSIX exists to allow application writers to write useful applications. If you interpret POSIX in such a way that gains you some benefit but shafts a large number of application writers then people are going to be reluctant to use your code. You're no longer a general purpose filesystem - you're a filesystem that's only suitable for people who write code with the expectation that their OS developers are actively trying to fuck them over. I'm sure Oracle deals with this case fine, but I also suspect that most people who work on writing Oracle on a daily basis have very, very unfulfilling lives.

But anyway. We can go and fix every single piece of software that saves files to make sure that it fsync()s, and we can avoid this problem. We can probably even do it fairly quickly, thanks to us having the source code to all of it. A lot of this code lives in libraries and can be fixed up without needing to touch every application. It's not the end of the world.

So why do I still think it's a bad idea?

It's simple. open(),write(),close(),rename() and open(),write(),fsync(),close(),rename(), are not semantically equivalent. One is "give me either the original data or the new data"[2]. The other is "always give me the new data". This is an important distinction. fsync() means that we've sent the data to the disk[3]. And, in general, that means that we've had to spin the disk up.

So, on the one hand, we're trying to use things like relatime to batch data to reduce the amount of time a disk has to be spun up. And on the other hand, we're moving to filesystems that require us to generate more io in order to guarantee that our data hits disk, which is a guarantee we often don't want anyway! Users will be fine with losing their most recent changes to preferences if a machine crashes. They will not be fine with losing the entirity of their preferences. Arguing that applications need to use fsync() and are otherwise broken is ignoring the important difference between these use cases. It's no longer going to be possible to spin down a disk when any software is running at all, since otherwise it's probably going to write something and then have to fsync it out of sheer paranoia that something bad will happen. And then probably fsync the directory as well, because what if someone writes an even more pathological filesystem. And the disks sit there spinning gently and chitter away as they write tiny files[4] and never spin down and the polar bears all drown in the bitter tears of application developers who are forced to drink so much to forget that they all die of acute liver failure by the age of 35 and where are we then oh yes we're screwed.

So. I said we could fix up applications fairly easily. But to do that, we need an interface that lets us do the right thing. The behaviour application writers want is one which ext4 doesn't appear to provide. Can that be fixed, please?

[1] xfs behaves like ext4 in this respect, so the obvious argument is that all our applications have been broken for years and so why are you complaining now. To which the obvious response is "Approximately anyone who ever used xfs expected their data to vanish if their machine crashed so nobody used it by default and seriously who gives a shit". xfs is a wonderful filesystem for all sorts of things, but it's lousy for desktop use for precisely this reason.

[2] Yes, ok, we've just established that it actually isn't that in the same way that GMT isn't UTC and battery refers to a collection of individual cells and so you don't usually put multiple batteries in your bike lights, but the point is that this is, for all practical intents and purposes, an unimportant distinction and not one people should have to care about in their daily lives.

[3] The disk is free to sit there bored for arbitrary periods of time before it does anything, but that's fine, because the OS is behaving correctly. Sigh.

[4] Dear filesystem writers - application developers like writing lots of tiny files, because it makes a large number of things significantly easier. This is fine because sheer filesystem performance is not high on the list of priorities of a typical application developer. The answer is not "Oh, you should all use sqlite". If the only effective way to use your filesystem is to use a database instead, then that indicates that you have not written a filesystem that is useful to typical application developers who enjoy storing things in files rather than binary blobs that end up with an entirely different set of pathological behaviours. If I wanted all my data to be in oracle then I wouldn't need a fucking filesystem in the first place, would I?

Fri, 06 Mar 2009 04:46:26 GMT

After a bit of back and forth with Peter, we came up with a straightforward way of dealing with the fact that the Wacom driver needs a logical input device per input type[1], but the X server only generates an input device per hal device. The simplest solution turned out to be a hal callout that generates additional hal devices on demand, which also means we can add information to the fdi files to only add the appropriate device types. Ought to land in rawhide in the near future, at which point tablets should be basically working out of the box. Except that xsetwacom gets device name -> type mapping by attempting to parse xorg.conf. Pass the suicide.

Today's other accomplishment was spending long enough looking at Toshiba ACPI dumps to figure out how to enable hotkey reporting without needing to poll. Of course, I then found that the FreeBSD driver has done the same thing since 2004. Never mind. Patch has been posted to lkml and I've shoved it into rawhide, so that'll improve things for most Toshiba users. There's a few machines that have an entirely different BIOS and we don't know how the hotkeys there work at all, so life continues to be miserable for those of you that own them. Sorry.

[1] Stylus, cursor, eraser and so on

Misc

Mon, 23 Feb 2009 18:43:34 GMT

Minor Fedora updates - I fixed up the FDI file in the wacom package, so tablet PCs should have a working stylus out of the box in rawhide. The eraser won't work right now - the driver needs some reworking to bind multiple X devices to a single logical input device. I've also added support for brightness control via smartdimmer to nouveau, which should increase the number of machines that have working brightness control. I don't think this has landed in the rawhide kernel yet, but should do soon. There's the potential for some conflict with the mbp_nvidia_bl driver. We may end up dropping that.

HP updates - The button bar on my 2510 got replaced last week. I now have working volume buttons again. However, the machine now reboots whenever the machine is suspended and I close the lid. Diagnosed to either a faulty switch assembly or system board, which will require an engineer visit. An engineer dropped round today to fix the touchpad. Despite the case notes clearly stating that the problem was with the cable assembly, he was sent a replacement top cover unit. Without any cables. So he's coming back at some point. HP's customer support system apparently does not allow these cases to be merged. Which means I now have two visits to look forward to.

Android - I'm gradually working my way through the code, replacing various custom interfaces with standard ones. const char * const LCD_BACKLIGHT = "/sys/class/leds/lcd-backlight/brightness"; is an interesting standout so far.

Thu, 19 Feb 2009 01:29:01 GMT

In other news, my HP 2510p's screen was replaced last month after the hinge snapped. A chunk of plastic off the hinge cover snapped off two days ago. I'm somewhat puzzled by this, since I can't see any plausible way force could be applied to it - it's as if it came away slightly and then got crushed when I tried to close the lid. On top of the motherboard having been replaced 4 times now (twice due to faulty power connectors, one due to the fan being replaced and the motherboard being swapped at the same time, once because the machine started refusing to boot at LCA) and it still being slightly tempremental when booting, I'm not overly impressed - especially when I've only had it 18 months. Nobody else I know with one seems to have had the same level of difficulty, though dreadful thermal issues (especially when using the dock) seem to be common.

The X200s looks awfully shiny, but the 1440x900 screen option doesn't appear to be available in the UK. Oddly, despite having a SIM slot, it also doesn't seem to come with an HSDPA option.

Thu, 19 Feb 2009 01:18:20 GMT

Aside from the inherent humour in Opensolaris's attempt to migrate to a 15 year old shell, today brings the thrilling news that I'll be moving to Boston to join the engineering team in Westford, MA. I look forward to the Applebee's. Some of the more entertaining aspects of US immigration mean that it'll probably be in July at the earliest (365 days with the company, plus time spent in the US since starting), which means that I have plenty of time to properly investigate my local pubs to console myself over having to spend the rest of my life drinking American beer.