Kernel Traffic

Ioctls Alexander Larsson

John McCutchan posted a patch and said:

I have been working on inotify a dnotify replacement.

inotify is a char device that has two ioctls: INOTIFY_WATCH and INOTIFY_IGNORE Which expect an inode number and an inode device number. I know that on some file systems the inode number and inode device number are not guaranteed to be unique. This driver is only meant for file systems that have unique inode numbers.

The two biggest complaints people have about dnotify are

1) dnotify delivers events using signals.

2) dnotify needs a file to be kept open on the device, causing problems during unmount.

As for 1) -- since inotify is a char device the events are delivered when the user calls read (). This easily integrates in to an applications select() loop.

I have a solution for 2) but I haven't implemented it yet, as I don't have the expertise. What I want to do is distinguish between some one holding a 'real' reference on an inode, and a reference that is used only for watching. This way in the unmount code we can make sure that no one is holding a real reference to an inode on the device, then while invalidating the inodes, clear the inodes watch list and send UNMOUNT events to the inotify device. Changes would have to be made so that inodes are kept in the inode cache when a watching reference is held.

I would like some pointers on how to implement 2). I am not sure the code path that takes place on unmount, or the best place to change the inode cache semantics.

I would also appreciate comments on the design in general and code review (this is my first kernel project).

In addition to John's two objections to dnotify, Chris Wedgwood added a third, dnotify cannot easily watch changes for a directory hierarchy. But John said this didn't seem to be a concern for dnotify users like Alexander Larsson, maintainer of nautilus and gnome-vfs. But Davide Libenzi said that A dnotify replacement that does not have the ability to watch for a hierarchy is pretty much useless. John said he did plan on adding the feature eventually, it just didn't have such a high priority. At one point he explained, Inotify will support watching a hierarchy. The reason it was not implemented yet is because the one app that I really care about is nautilus and the maintainer of it says he doesn't care. He added, The big feature that inotify is trying to provide is not having to keep a file open (So that unmounting is not affected). I asked for some guidance from people more familiar with the kernel so that I can implement this feature, it requires changes made to the inode cache, and how unmounting is done. However, at this point Alexander Viro came down hard on the whole idea, saying:

First of all, on quite a few filesystems inumbers are stable only when object is pinned down. What's more, if it's not pinned down you've got nothing even remotely resembling a reliable way to tell if two events had happened to the same object - inumbers can be reused.

Besides, your "doesn't pin down" is racy as hell - not to mention the way you manage the lists, pretty much every function is an exploitable hole. Hell, just take a look at your "find inode" stuff - you grab superblock, find an inode by inumber (great idea, that - especially since on a bunch of filesystems it will get you BUG() or equivalent) then drop refernce to superblock (at which point it can be destroyed by umount()) _and_ do iput() (which will do lovely, lovely things if that umount did happen). Moreover, you return a pointer to inode, even though there's nothing to hold it alive anymore. And dereference that pointer later on, not caring if it had been freed/reused/whatever.

Overall: hopeless crap. And that's a direct result of your main feature - it's really broken by design.

John objected that the feature he was trying to implement is the only reasonable solution to a problem that needs fixing. You can't simply say that a file manager needing to be notified when directories change is broken. How would you solve this problem? But the discussion petered out quickly after this.

Disk Arrays: RAID Disks: IDE Hot-Plugging PCI Serial ATA

Jeff Garzik posted a status report for Linux support of Serial ATA (SATA). He said:

Hardware support

Intel ICH5 and ICH5-R

Summary: No TCQ. Looks like a PATA controller, but with a few added, non-standard SATA port controls. Hardware does not support hotplug. "Coldplug" support is potentially feasible.

libata driver status: Production, but see issue #2, #3.

drivers/ide driver status: Production, but see issue #1, #2.

Issue #1: Depending on BIOS settings, IDE driver may lock up computer when probing drives.

Issue #2: Excessive interrupts are seen in some configurations.

Issue #3: "Enhanced mode" or "SATA-only mode" may need to be set in BIOS.

Intel ICH6 ("AHCI")

Summary: Per-device queues, full SATA control including hotplug and PM.

libata driver status: "looks like ICH5" support available in ata_piix. Preliminary driver with full AHCI support now exists, and is being integrated into libata mainline.

Promise TX2/TX4/SX4

Summary: Per-host queues on all controllers. Full SATA control including hotplug and PM on all but one controller.

libata TX2/TX4 driver status: Production, but see issue #5.

libata SX4 driver status: Production, but see issue #6.

Issue #5: Some boards appear to have PATA as well as SATA ports. PATA is not currently supported, and no plans have yet been made to rectify this. Ideally drivers/ide would drive PATA, but if they are the same PCI device, that would not be feasible.

Issue #6: The SX4 hardware is not fully utilized by the Linux kernel driver. The SX4 hardware includes an on-board DIMM and hardware XOR offload. Using the on-board DIMM as cache, and issuing each RAID transaction once (instead of once for each disk), will result in increased performance, but the driver doesn't do that yet. SX4 hardware is very "RAID friendly", particularly RAID1/5. Users may wish to use the Promise driver to fully utilize the hardware.

Silicon Image 3112/3114

Summary: No TCQ. Looks like a PATA controller, but with full SATA control including hotplug and PM.

libata driver status: Beta.

drivers/ide driver status: Production, but see issue #4.

Issue #4: Need to have the most recent fixes posted to lkml, for stable operation and full performance (where possible).

Silicon Image 3124

Soon, hopefully. Silicon Image has made documentation and sample hardware available to me (jgarzik) for development. Some code exists internally.

Broadcom/ServerWorks/Apple

Summary: Huge per-device queues, full SATA control including hotplug and PM for the "Frodo4" and "Frodo8" boards. Apple K2 SATA, which also uses this chipset, has all the feature of Frodo4/8 save the host DMA queueing feature ("QDMA").

libata driver status: Beta, but no QDMA support yet.

VIA

Summary: No TCQ. Looks like a PATA controller, but with full SATA control including hotplug and PM.

libata driver status: Beta.

SiS

libata driver status: Beta

Vitesse

libata driver status: Beta

Marvell

libata driver status: in progress

Software support

Basic Serial ATA support

The "ATA host state machine", the core of the entire driver, is considered production-stable.

The error handling is _very_ simple, but at this stage that is an advantage. Error handling code anywhere is inevitably both complex and sorely under-tested. libata error handling is intentionally simple. Positives: Easy to review and verify correctness. Never data corruption. Negatives: if an error occurs, libata will simply send the error back the block layer. There are limited retries by the block layer, depending on the type of error, but there is never a bus reset.

Or in other words: "it's better to stop talking to the disk than compound existing problems with further problems."

As Serial ATA matures, and host- and device-side errata become apparent, the error handling will be slowly refined. I am planning to work with a few (kind!) disk vendors, to obtain special drives/firmwares that allow me to inject faults, and otherwise exercise error handling code.

Queueing support

Even though some SATA host controllers on the market already support command queueing (a.k.a. "TCQ"), libata does not yet support it.

However, libata was designed from the ground-up to support queueing, so I need only change a few lines of code, and write two functions, to enable this behavior.

Queueing will be enabled in libata soon, but to do so requires a long stretch of testing on a large variety of controllers and drives. This is very time-intensive, and is the largest part of this task.

Tangent: Host-based queueing and Native Command Queueing

Queueing is the process of sending multiple commands to a single device, without waiting for prior commands to finish. This increases performance and reduces latency. There are three types of queueing in the ATA world:

1) "legacy TCQ" -- some PATA devices support this. Just ignore it, it's going away.

2) "host-based TCQ" -- the host controller supports a queue of drive commands, whether or not the drive supports it.

3) "Native Command Queueing" -- both host and drive cooperate in the queueing and execution of drive commands. This should provide the highest performance and lowest latency of all three options.

#1 is support by drivers/ide _only_. libata will not support this.
#2 will soon be supported by libata.
#3 will be supported by libata when hardware is available from drive manufacturers.

Hotplug support

All SATA is hotplug.

libata does not support hotplug... yet.

Power Management support

Over and above the power management specified in the ATA/ATAPI specification, one can aggressively control the power consumption of SATA hosts, the SATA bus, and the SATA device.

SMART support

Soon. Requires the capability to directly submit ATA commands from userspace to the low-level device, which must be added with care.

Backward Compatibility FS: ext3 Chris Wright

Andy Lutomirski said:

This reintroduces useful capabilities.

In this model, the inheritable mask is a limit on what capabilities the task or any of its children may have. Permitted and effective masks have their old meanings.

Init gets all capabilities (even CAP_SETPCAP) since it seems absurd to do otherwise.

Part 1 shouldn't change user-visible behavior; it just moves the vfs capability logic into fs/exec.c (where it IMHO belongs) and the setuid logic into apply_creds (where it IMHO belongs). I made no effort to make apply_creds pretty since it all gets deleted in the next patch anyway.

Part 2 redoes the capability logic.

This code is paranoid about setuid. A later patch will allow that to be overridden by a sysctl so that securelevel can be done usefully.

I left cap_bset in, even though I can't see a use for it anymore.

I put some user tools at http://www.stanford.edu/~luto/cap/; I've used themn to exercise this code somewhat.

It applies to 2.6.6-mm1. It does _not_ apply to 2.6.6 vanilla, but the fix should be trivial (it conflicts with something in fs/exec.c). It will not apply to earlier versions, as it depends on the compute_creds race fix.

Please let me know what you think and if you see any holes in this code or possibilities of exposing / introducing bugs in other programs (think Sendmail bug).

This should also eliminate the Oracle magic-group mess :)

Andrew- is this sufficiently non-scary for -mm?

Where this is going:
There is probably some dead code now in sys_capset. I'll check on that. Also, I have an ext3 xattr caps patch lying around somewhere. I can try to get it working again.

Chris Wright was very happy to see this, and said he'd test it out. Andrew Morton also replied to Andy's question about whether the patch was too scary to be included in the main kernel tree. Andrew said, Scares the shit out of me, frankly ;). He clarified some of his objections, saying:

What if there are existing applications which are deliberately or inadvertently relying upon the current behaviour? That seems unlikely, but the consequences are gruesome.

If I'm right in this concern, the fixed behaviour should be opt-in. That could be via a new prctl() thingy but I think it would be better to do it via a kernel boot parameter. Because long-term we should have the fixed semantics and we should not be making people change userspace for some transient 2.6-only kernel behaviour.

In the course of a very brief discussion, it became clear that Andrew was willing to consider Andy's patch, so long as enough compatibility was maintained so that anything that would break given Andy's changes, would have been broken in one way or another already. So the possibility remained for Andy's changes to walk a tightrope of backward compatibility, in which stuff that worked in the old system would have to continue to work; while anything pathological in the old system, could break in other ways as well under Andy's new system.

Disks: IDE I2C Sound: i810 Jean Delvare Linus Torvalds Andrew Morton Greg KH

Jason Gaston said, This patch adds DID support for ICH6 and 6300ESB to i2c-i801.c(SMBus). In order to add this support I needed to patch pci_ids.h with the SMBus DID's. To keep things orginized I renumbered the ICH6 and ESB entries in pci_ids.h. I then patched the piix IDE and i810 audio drivers to reflect the updated #define's. I also removed an error from irq.c; there was a reference to a 6300ESB DID that does not exist. This patch is against the 2.6.6 kernel. Greg KH applied the patch to his tree, in queue to Andrew Morton and Linus Torvalds; and Jean Delvare asked Jason to produce a patch against the 2.4 kernels as well; and Jason said he'd give this a whirl.

Kernel Release Announcement POSIX Virtual Memory

Andrew Morton announced Linux 2.6.6-mm2, saying:

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.6/2.6.6-mm2/

Lots of VM changes - fixes from Andrea and generally moving things closer to the -aa tree.
The x86_64 gcc-3.3.3 shipped with SuSE 9.1 miscompiles the post-2.6.6 CPU scheduler changes, resulting in lockups after several minutes of heavy load. Hence this kernel refuses to build on gcc-3.3.x. Please use gcc-3.4.0 if you're on x86_64.
Rediscovered and hopefully fixed the page double-freeing bug which was identified in August 2002 (!). I decided it wasn't real, but it is.
arch updates, rlimits for rt-signals and posix message queues, tons of other stuff.

Capabilities POSIX

Andy Lutomirski said:

This implements working capabilities.

Changes from the previous version:

It's now just one patch (the split doesn't make sense anymore)
Cleaned up cap_bprm_apply_creds
Restrictions on setuid are somewhat stricter now
Rediffed against 2.6.6-mm2 (didn't change anything)
bprm is initialized correctly in fs/exec.c
printk to remind users that the patch is enabled
LSM builds correctly. Untested.

The whole thing is now a module parameter -- specify commoncap.newcaps=1 if capabilities is built-in or newcaps=1 in modprobe or insmod if not.

Known issues:

setxuid emulation is wrong (keepcaps doesn't work for setre(s)uid). I haven't fixed this because I don't what to change the semantics of PR_SET_KEEPCAPS. I'll probably add PR_SET_REALLY_KEEPCAPS to really keep capabilities.
When newcaps=0 (the default), linuxrc gets all capabilities. This is different from the old behavior. Init and programs started from linuxrc still match the old behavior. I couldn't think of any remotely clean way around that without breaking linuxrc for newcaps=1
I haven't tried it, but I imagine that unloading commoncap and then reloading it in a different mode may do unexpected things. So read the code before you try that. I see no reason to fix this one because the whole thing should go away eventually.

When newcaps=1, this extends the LSM interface: bprm->secflags & BINPRM_SEC_NO_ELEVATE means that privileges should not be elevated. So stacking modules (e.g. selinux) should set this before calling to the lower module and test it again before deciding whether to elevate privileges. That way, either all privs are elevated or none are.

I also added a flag (BINPRM_SEC_SECUREEXEC) with the obvious meaning. Otherwise cap_bprm_secureexec would have been a mess.

Chris Wright had some objections. He said:

I think it still needs more work. Default behavoiur is changed, like Inheritble is full rather than clear, setpcap is enabled, etc. Also, why do you change from Posix the way exec() updates capabilities? Sure, there is no filesystem bits present, so this changes the calculation, but I'm not convinced it's as secure this way. At least with newcaps=0.

I believe we can get something functional with fewer changes, hence easier to understand the ramifications. In a nutshell, I'm still not comfortable with this.

Also, it breaks my tests which try to drop privs and keep caps across execve() which is really the only issue we're trying to solve ATM.

Valdis Kletnieks replied:

The last time the "capabilities" thread reared its head a while ago, Andy made a posting that pretty conclusively showed that the Posix way was totally b0rken if you ever intended to support filesystem bits. So if you wanted to ever have a snowball's chance of supporting something like:

chcap cap_net_raw+ep /bin/ping

so you could get rid of the set-UID bit on 'ping', you had to toss the Posix propogation rules out the window. So we need to do either:

Toss the Posix out the window
Toss all the filesystems capabilities support out the window.

(I'm assuming that a suggestion that we make the choice a Kconfig option will be met with the sound of many kernel hackers either retching in disgust or screaming in horror ;)

Chris replied that the desired behavior still wasn't clear, and it's very uncomfortable to change mainline in subtle ways that could break security during stable series. Olaf Dietsche added that he'd done some work on supporting the POSIX way himself, and gave a URL. He said, This supports filesystem capabilities with the current (POSIX?) implementation. So, whatever Andy has shown, it has at least one counter evidence. but Valdis went to the page and noticed that it said at the top, "This implementation is likely *not* POSIX compatible." He asked what the deal was, and Olaf replied, This refers to the tools I provide. I should emphasize this on the page, thank you. My patch doesn't change the rules, how the capability bits are mingled.

A long technical debate ensued close by, with no real resolution.

FS: procfs FS: sysfs Jeff Garzik Andrew Morton

Robert Picco said, The driver supports the High Precision Event Timer. The driver has adopted a similar API to the Real Time Clock driver. It can support any number of HPET devices and the maximum number of timers per HPET device. For further information look at the documentation in the patch. Thanks to Venki at Intel for testing the driver on X86 hardware with HPET. Andrew Morton was unhappy with the lack of code comments or other documentation; and he and Jeff Garzik had other technical comments to make, including Jeff's preference for SysFS over Robert's choice of ProcFS for his driver. Jeff also pointed to a place to find HPET documentation (Chapter 5.18).

Robert, some days later, posted an updated patch which he hoped addressed some of Andrew's and Jeff's concerns; and Jeff was very pleased, and gave his OK for inclusion. There was a bit more technical discussion, but not much, and the thread petered out.

USB Version Control Olaf Hering

Greg KH said:

Here are USB patches for 2.6.62. There are a bunch of different things here:

interface access fixups
bugfixes
a few new drivers
lots more bugfixes

All of these (with the exception of a few minor patches from today) have been in the -mm tree for quite some time.

Please pull from: bk://kernel.bkbits.net/gregkh/linux/usb-2.6

Patches will be posted to linux-usb-devel as a follow-up thread for those who want to see them.

Olaf Hering had some problem compiling the code on top of the current official tree; and after some intervention by Linus Torvalds (Replace all "led" with "cytherm". The code was crap, and would never have compiled with debugging on anyway.) Greg said he'd post a new patch with clean fixes for this.

Disks: IDE PCI

Bartlomiej Zolnierkiewicz posted some guidelines for writing IDE drivers:

general rules:

code outside drivers/ide directory shouldn't need to include <linux/ide.h> if it does something is wrong
do not believe in popular myth that driver code can be of less quality than core kernel code
don't copy without thinking ugly and bogus code (there is still lot of such in IDE)
IDE is going into full host driver model so write host driver for your interface
host drivers should request/release IO resource themelves and set hwif->mmio to 2
remember about ordering issues (you break device ordering and some machines won't boot)
use linux-ide@vger.kernel.org mailing list

new architecture:

add only what is really necessary to <asm/ide.h> and put interface specific code into drivers/ide
ide_init_hwif_ports() is obsolete and dying, define IDE_ARCH_NO_OBSOLETE_INIT in <asm/ide.h>
define ide_default_irq(), ide_init_default_irq() and ide_default_io_base() to (0)
ide_init_default_hwifs() is gone

architecture specific drivers:

do not abuse ide_default_irq()/ide_default_io_base() from <asm/ide.h>
your driver should be in drivers/ide/<your_arch>/<your_driver_name>
see drivers in drivers/ide/arm and drivers/ide/legacy for examples (especially m68k and arm specific drivers)
do not use ide_setup_ports()

PCI drivers:

no need to split your driver on .c and .h files
/proc/ide/ interfaces are obsolete

Jeff Garzik offered some feedback. In particular, he felt that the handling of the ide_init_hwif_ports() situation was backwards from what it should be. Since it was an obsolete function, he reasoned, it would be better to make it available only to drivers that defined IDE_ARCH_OBSOLETE_INIT (as opposed to defining IDE_ARCH_NO_OBSOLETE_INIT for new drivers, as Bartlomiej had it). This way new drivers wouldn't have to have an essentially garbage definition just because certain old drivers did things a different way. Bartlomiej thought about this and said he'd change it to be as Jeff suggested.

Another of Jeff's comments related to the need to define ide_default_irq(), ide_init_default_irq() and ide_default_io_base() to (0) for new architectures. He suggested providing generic definitions, so that folks coding for new architectures wouldn't need to worry about this unless it was actually relevant to them. At one point in the discussion, he explained, Your document appears to imply that each new arch should define the above three symbols. My suggestion is to devise a method by which new arches don't have to care about those symbols at all, unless required to do so by the underlying hardware. Bartlomiej said he'd look into how this could be done.

Virtual Memory Bruce Perens

Mel Gorman said:

A long time ago, I told a number of people that I was asked to write a book based on the Linux VM documentation I had up on http://www.csn.ul.ie/~mel/projects/vm . I am happy to announce that this book is finished and now available in online stores (http://www.phptr.com/title/0131453483). Yes, this is a plug! It is published under the OPL as per the criteria of the Bruce Perens Open Book Series (http://www.perens.com/books) meaning it will be available for free download after 90 days. In other words, I intended for it to be easily available like the online documentation was but the option of having your own shiny printed copy is there :)

The information on the web site will remain as it is for people that want to read it but the book has a lot more information in it including TLB management, shared memory filesystem, a lot more code commentary and extra material and clarifications throughout the whole book. Each chapter also has information on the 2.6 kernel as was known around about 2.6.0-test4 which will help anyone trying to figure out the more recent code for themselves.

It has been fun and I hope people enjoy the final result.

Marc-Christian Petersen voiced the feelings of many, when he replied, I can't say how much I appreciate this and your effort. Thanks alot!

Disks: IDE Version Control

Randy Dunlap posted a patch to remove the entire PC9800 sub-architecture from the Linux kernel. By way of justification, he said that it was incomplete, hackish (at least in IDE), maintainers don't reply to emails and haven't touched it in awhile. Can't even config it to try to build it without other patches to the kernel tree. Andrew Morton added, the hardware is obsolete, isn't it? Does anyone know when they were last manufactured, and how popular they are? Norman Diamond said, they aren't popular any more, but the last ones were still respectably powerful and can run stuff like Windows 2000 Server. James Bottomley also replied to Andrew:

Hey, just being obsolete is no grounds for eliminating a subarchitecture...

However, I would have to say that being unmaintained is. Because of the penchant of x86 people to go "it compiles on my PC, ship it", the x86 subarchitectures are about the fastest bitrotting pieces of the kernel there are.

Since mach-pc9800 cannot currently be compiled and there's no evidence that it actually was, I'd remove it unless someone steps up quickly to maintain it (and get it to the point where it's actually compileable).

Andrew replied:

Well it's a question of whether we're likely to see increasing demand for it in the future. If so then it would be prudent to put some effort into fixing it up rather than removing it.

Seems that's not the case. I don't see a huge rush on this but if after this discussion nobody steps up to take care of the code over the next few weeks, it's best to remove it.

Jeff Garzik was sad to see this happening, and at one point he said, The PC9800 people spent a good long while working with Alan and others to get what little bits got merged into the kernel. But he acknowledged, I suppose disappearing and not maintaining the code is the overriding factor here...

Close by, James Bottomley also remarked, regarding the preferable situation of actually finding a maintainer for the code, I think the best way of making someone sit up and take notice is simply to remove it. After all, given that we have the kernel under source control it's not like it's going to be hard to put it back if someone actually does notice and screams...

FS: sysfs Virtual Memory Adrian Bunk Amit S. Kale

Andrew Morton announce Linux 2.6.6-mm3, saying:

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.6/2.6.6-mm3/

A few VM changes, getting things synced up better with Andrea's work.
A new kgdb stub, for ia64 (what happened to the grand unified kgdb project?)
The backing-store-for-sysfs patches need redoing, and have been temporarily dropped

As discussed in , Adrian Bunk posted several patches to remove portions of the PC9800 subarchitecture.

Regarding Andrew's question about KGDB, Tom Rini replied:

No one asked the ia64 folks who did that work "Hey, have you looked at the grand unified kgdb project on kgdb.sf.net ?" would be my guess.

Having said that, if you're willing to go with a slightly late initalizing (I saw part of the early_param work get dropped again I think, so I'm gonna guess you don't wanna deal with that again yet) KGDB for i386 and PPC32, I can whip something up vs 2.6.6 in a day or so.

Robert Picco replied, I did the ia64 port and started with Andrew's 2.6.4-mm2 i386 sources. I'm assuming the long term strategy is to move to a unified kgdb being done on sourceforge? If so, I'll take a look at this. Tom replied, My long term strategy is to get everyone using the version on sourceforge that splits out the common portions of the stub from the arch-specific portions. If you could go ahead and get ia64 working on this as well I'd appreciate it. Right now it's still vs 2.6.5, but I'm going to try and fix that today or tomorrow to be vs 2.6.6. An hour and a half later he posted, saying he'd done the update to 2.6.6; Amit S. Kale was pleased to hear it.

FS: JFS PCI Power Management: ACPI Sound: i810

Marcelo Tosatti announced Linux 2.4.27-pre3, saying, It contains network driver fixes, nForce2 hang PCI workaround, i810 audio fixes, JFS update, ACPI, sh64/ia64 arch updates, amongst others.

Backward Compatibility FS: UMSDOS FS: ext2 Small Systems Adrian Bunk Jan-Benedict

Adrian Bunk asked if there were any users or developers of UMSDOS, or if it should be removed from the 2.6 kernel. Jan-Benedict Glaw said it would be nice to keep it around for historical reasons, as one of the old-time ways of showing Linux to DOS users; but he acknowledged, However, one can achieve the same (with a lot more work) by placing a loop-mountable ext2 FS and start it from an initrd. Much more complicated, not that flexible (loop-mounted files don't typically grow:). Mark Beyer pointed out that some embedded systems still used it, so keeping it around for backward compatibility might be good; but Adrian replied that it was actually already broken, and that unless someone would actually maintain it, this would be a significant factor in deciding whether to drop it.

Compression FS: SquashFS

Phillip Lougher said:

I'm pleased to announce the first version of Squashfs 2.0. A lot of changes to the Squashfs filesystem have been made under the bonnet (hood), to improve compression. Squashfs 2.0 has added the concept of fragment blocks and has increased the block size to 64K. This achieves a 5 - 20% compression saving, and allows Squashfs to achieve better compression than Cloop while retaining the I/O efficiency of a compressed filesystem. In addition, the maximum number of UIDs and GIDs has been increased to 256. This allows Squashfs to better support live CDs.

For a description of fragment blocks and the other changes, please go to the project page http://squashfs.sourceforge.net.

Disk Arrays: RAID Disks: IDE Disks: SCSI FS: ReiserFS FS: ext3 Forward Port Kernel Release Announcement SMP Serial ATA Christoph Hellwig

Andrew Morton announced Linux 2.6.6-mm5, saying:

http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.6/2.6.6-mm5/
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.6/2.6.6-mm5/

Some rearchitecting of the VFS's symlink walking code. Reduces stack usage and apparently permits us to increase the maximum hop count from 5 to 8, although the patch doesn't actually do that.
Implementation of request barriers for IDE and SCSI. The idea here is that a filesystem can tag an IO request as a barrier and the disk will not reorder writes across the barrier. It provides additional integrity guarantees for the journalling filesystems. The feature is enabled for reiserfs and ext3.

On reiserfs do `mount /dev/hda /wherever -o barrier=flush' or `barrier=none'.

On ext3 do `mount ... -o barrier=1' or `barrier=0'.

ext3 also supports `mount -o remount,barrier=N'. I didn't check whether reiserfs supports switching at remount time and nobody tells me these things.

(Yes, we should give these mount options the same name).

Although this feature has been around for a while it is new code, and the usual cautions apply. If it munches all your files please tell Jens and he'll type them in again for you.
The pagecache radix-tree spinlocks have gone back to rwlocks again. It helps big SMP significantly and doesn't seem to make much difference to small SMP (1-2% at most IIRC). It does need some more measuring.
Added a new SATA RAID driver from 3ware. From a quick peek it seem to need a little work yet.

Regarding the state of the SATA code, Jeff Garzik said:

It's not too bad... but it looks more like a 2.2 driver forward ported to 2.4, than a 2.6.x driver. Needs some luvin' from the 2.6 scsi api crew.

Overall, it appears to be a message-based firmware engine like drivers/block/carmel.c, that hides the SATA details in the firmware.

Christoph Hellwig also pointed out that the driver submissions should always go to the linux-scsi mailing list, instead of just linux-kernel. Andrew and Adam Radford (who'd submitted the patch) confirmed that this had in fact been done, but the patch had been so big that the linux-scsi list software had rejected it.

Assembly SMP Andrew Morton Denis Vlasenko

Christoph Hellwig said, These days gcc uses i486+ only instruction by default in libstdc++ so most modern distros wouldn't work on i386 cpus anymore. To make it work again Debian merged Willy Tarreau's patch to trap those and emulate them on real i386 cpus. The patch is extremely non-invasive and would certainly be usefull for mainline. Any reason not to include it? Several folks including Andrew Morton looked over the patch and came back with criticisms, including potential OOPSes that could be caused by the code; and these were discussed briefly. Elsewhere, Willy Tarreau (the author of the patch) also replied to Christoph, with some of his own reservations:

I have mixed feelings about this because :

I don't know which version they based their port on. The version I published 2 years ago included CMOV emulation, and Denis Vlasenko found several bugs in it which I then fixed. Since the Debian port doesn't include CMOV, I wonder whether it includes those bugs or not (I'll have to diff the patches).
The code is ugly in some areas, and someone will kill me if this goes into mainline.
There are people (like Alan) who think that this should not go into mainline because this is a distro problem and nothing else. He says that only i386 packages should be installed on an i386 machine. He's perfectly right about this. I found it interesting for people like me who boot kernels on random machines, try to recover hard disk contents or other things using lots of dirty tools, and sometimes get hit by the "illegal instruction" trap. It also allowed me to run a pppd compiled with i586 glibc on my i386 firewall, but obviously this is just the easy way and not the right way to go.
I couldn't emulate locks, so this will break on SMP systems, and so will it if you need to access some memory share with an external microcontroller or something like that.
I've always wondered if this feature would not be exploitable to access unauthorized information. Eg: code an invalid opcode which would get emulated and references a memory area outside the user space. I put some verify_area() where I thought appropriate, but I might have left some caveats... Morten Welinder once insisted on the fact that each byte should be read once and only once so as to ensure that the user doesn't change the instruction while it is being emulated. I think it's already the case. He also said that I didn't take care of the segment selectors (such as SS) which some programs use perfectly legally (eg Wine). I don't know how to do that.
Denis Vlasenko suggested that we print some messages on the console when a program triggers the code, so as not to let the user think that his machine is slow as hell. But for this we would need not to flood the console (eg: once for each prog) but we don't want either to store anything in the task structure about the message having been displayed, so for now there's nothing. In fact, the only message which is displayed (in the most recent version) is about a warning about a LOCK prefix on an SMP kernel. But I didn't find it right here, so I suspect this is based on ancient code.
why not include the CMOV emulation while we're at it ? There are so many people using VIA EDEN chips who think it's i686 compatible that they may get hit too. IIRC, the chip only executes CMOV on registers, but very slowly (a few tens of cycles), while register to memory accesses generate a trap.

Other than that, I'm happy that someone found it useful, and happy too that someone did the 2.6 port :-)

Jeff Garzik disagreed with the opinion Willy attributed to Alan Cox, that this patch was a problem distributions had to deal with, and shouldn't ever be prat of the kernel. Jeff said, I want to add "old Alpha" emulation code, so that older Alphas can run binaries built on the newer alphas. Alan (in a now-rare linux-kernel post) replied:

Well it always depends on the platform. cmov emulation isnt useful because i686 gcc generates too many for it to be of value. For alpha it depends on the commonness of the emulated instructions and the emulation cost.

Either way it is a user space problem in almost all situations. See http://www-sop.inria.fr/geometrica/team/Sylvain.Pion/progs/mmx-emu/

There were other responses to Willy's post, and a technical discussion ensued, with Alan taking a lead role; but not much conclusive came out of it.

Kernel Release Announcement Virtual Memory Alan Cox Hugh Dickins Andrea Arcangeli Andrew Morton

Linus Torvalds announced Linux 2.6.7-rc1, saying, This is stuff all over the map, but most interesting (or at least most "core") is probably the merging of the NUMA scheduler and the anonvma rmap code. The latter gets rid of the expensive pte chains, and instead allows reverse page mapping by keeping track of which vma (and offset) each page is associated with. Special kudos to Andrea Arcangeli and Hugh Dickins. Horst von Brand replied, Not wanting to start a flamewar, but this sort of massive changes in a _stable_ series has got me quite confused... either 2.6.0 was premature, or the "just stabilize 2.6, new stuff only into 2.7 (when it opens)" got lost somewhere. He added later, It looks like Andrew Morton is playing the role Alan Cox had during part of 2.5: Checking/filtering/testing "not quite ready" stuff, when he was supposed to be like Marcelo Tossati: Keeper of the stable series, don't let anything too risky get even near it.

Jeff Garzik said that the VM Subsystem would always be a work in progress, but that for the stable series, development should quiet down in order to actually become stable. He added later, You've got all the major Linux vendors preparing (or releasing) 2.6.x-based product and IMO the 2.6 kernel is still a moving target, with non-trivial behavior (and sometimes API) changes every couple of kernel versions.

At one point, Linus argued that the changes that had been going into the 2.6 kernel had not been complete rewrites, but more "implementation details", and were thus OK.

There was some small discussion, but only enough to clarify the opposing positions.

Version Control Paul Mackerras Arjan van de Ven

It's unusual for Linus Torvalds to actually begin a discussion himself, unless it is a new kernel announcement; this time, he started off with:

Hola!

This is a request for discussion..

Some of you may have heard of this crazy company called SCO (aka "Smoking Crack Organization") who seem to have a hard time believing that open source works better than their five engineers do. They've apparently made a couple of outlandish claims about where our source code comes from, including claiming to own code that was clearly written by me over a decade ago.

People have been pretty good (understatement of the year) at debunking those claims, but the fact is that part of that debunking involved searching kernel mailing list archives from 1992 etc. Not much fun.

For example, in the case of "ctype.h", what made it so clear that it was original work was the horrible bugs it contained originally, and since we obviously don't do bugs any more (right?), we should probably plan on having other ways to document the origin of the code.

So, to avoid these kinds of issues ten years from now, I'm suggesting that we put in more of a process to explicitly document not only where a patch comes from (which we do actually already document pretty well in the changelogs), but the path it came through.

Why the full path, and not just originator?

These days, most of the patches in the kernel don't actually get sent directly to me. That not just wouldn't scale, but the fact is, there's a lot of subsystems I have no clue about, and thus no way of judging how good the patch is. So I end up seeing mostly the maintainers of the subsystem, and when a bug happens, what I want to see is the maintainer name, not a random developer who I don't even know if he is active any more. So at least for me, the _chain_ is actually mostly more important than the actual originator.

There is also another issue, namely the fact that when I (or anybody else, for that matter) get an emailed patch, the only thing I can see directly is the sender information, and that's the part I trust. When Andrew sends me a patch, I trust it because it comes from him - even if the original author may be somebody I don't know. So the _path_ the patch came in through actually documents that chain of trust - we all tend to know the "next hop", but we do _not_ necessarily have direct knowledge of the full chain.

So what I'm suggesting is that we start "signing off" on patches, to show the path it has come through, and to document that chain of trust. It also allows middle parties to edit the patch without somehow "losing" their names - quite often the patch that reaches the final kernel is not exactly the same as the original one, as it has gone through a few layers of people.

The plan is to make this very light-weight, and to fit in with how we already pass patches around - just add the sign-off to the end of the explanation part of the patch. That sign-off would be just a single line at the end (possibly after _other_ peoples sign-offs), saying:

Signed-off-by: Random J Developer <random@developer.org>

To keep the rules as simple as possible, and yet making it clear what it means to sign off on the patch, I've been discussing a "Developer's Certificate of Origin" with a random collection of other kernel developers (mainly subsystem maintainers). This would basically be what a developer (or a maintainer that passes through a patch) signs up for when he signs off, so that the downstream (upstream?) developers know that it's all ok:

Developer's Certificate of Origin 1.0

By making a contribution to this project, I certify that:

a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or

b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or

c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it.

This basically allows people to sign off on other peoples patches, as long as they see that the previous entry in the chain has been signed off on. And at the same time it makes the "personal trust" explicit to people who don't necessarily understand how these things work.

The above also allows for companies that have "release criteria" to have the company "release person" sign off on a patch, so that a company can easily incorporate their own internal release procedures and see that all the patches have gone through the right channel. At the same time it is meant to _not_ cause anybody to have to change how they work (ie there is no "extra paperwork" at any point).

Comments, improvements, ideas? And yes, I know about digital signatures etc, and that is _not_ what this is about. This is not about proving authorship - it's about documenting the process. This does not replace or preclude things like PGP-signed emails, this is _documenting_ how we work, so that we can show people who don't understand the open source process.

Arjan van de Ven suggested having developers register a public key with Linus, such that whenever they submitted a patch that had been signed with that key, there would be an implicit attestation similar to the three items listed above. Linus replied:

One reason that I'd prefer not to is simply the question of "who maintains the certificates?"

I certainly don't want to maintain any stateful paperwork with lots of people. This is why I personally would prefer it all to be totally state-less.

Also, there is a _fundamental_ problem with signing a patch in a global setting: the patches _do_ get modified as they move through the system (maybe just bug-fixes, maybe addign a missing piece, maybe removing a controversial part). So the signature ends up being valid only on your part of the communication, and then after that it needs something else.

And what I do _not_ want to see is a system where if somebody makes a trivial change, it then has to go back to you to be re-signed. That just would be horrible.

With those (pretty basic) caveats in mind, I don't see any fundamental problem in a PGP key approach, if it's a "local" thing between developers. In fact, I think PGP-signed patches are something we may want to look at from a "trust the email" standpoint, but I think it should be a _local_ trust. And part of that "local" trust might be a private agreement between ddevelopers that "it's ok to add the sign-off line for Arjan when the patch has come with that PGP signature" when the patch is passed on.

So to me, the sign-off procedure is really about documenting the path, and if a PGP key is there in certain parts of the path, then that would be a good thing, but I think it's a separate thing from what I'm looking for.

Davide Libenzi also said to Linus, Andrew already puts the "From:" thing in the patch comment, so this should be simply a matter of replacing "From:" with "Signed-off-by:", preserving it in logs, and documenting the thing in the patch submission doc. No? Linus replied:

Yes and no.

Right now it is _Andrew_ that does the From: line from you. In the sign-off procedure, it would be _you_ who add the "Signed-off-by:" line for yourself.

(And then Andrew would sign off on the fact that you signed off).

Not a big difference, I agree.

La Monte H.P. Yarroll replied, Andrew's From comment is already a little lossy, e.g. most LKSCTP patches show up as from Sridhar or DaveM even though there's a whole subproject of developers working behind Sridhar. I think the proposed process will increase the amount of explicit credit being recognized--a very good thing IMHO, since this is the core currency of our gift culture.

Elsewhere, in the course of discussion, Linus remarked, I don't expect this process to start taking effect for a while. Not only do we need to come to some level of agreement about it, but we need to give people the time to learn about it _without_ rejecting patches in the meantime. There is no real "flag-day" (and it's certainly not today), although I'm hoping that by the time I start up 2.7.x we'd have this in place.

Elsewhere, Ben Collins also replied to Linus' original post, asking, Say the patch comes to me from some patch collection maintainer, who got it from the original author. So the original person never put a Signed-off-by, and neither did the person who sent me the patch, should I still add the eplicit Signed-off-by's to the patch, and add myself, before sending it to you? Linus replied:

You should never sign off for somebody else.

You _can_ sign off as yourself, and just add a note of "From xxxx". That's what the (b) case is all about (ie "to the best of my knowledge it's already under a open-source license").

Of course, if it's a _big_ work with lots of original content, and you're unsure of exactly what the original author wanted to do with this, you obviously should _not_ sign off on it. But you knew that.

Elsewhere, under the subject: [PATCH][PPC64] Don't clear MSR.RI in do_hash_page_DSI, Paul Mackerras submitted a patch, using the "Signed-off-by" form suggested in Linus' original post. Linus replied:

Ok, looks like somebody has bought into the sign-off procedure. Great.

Except that I (and my tools) expected for the "Signed-off-by:" line to go into the comment section _before_ the patch (and after the "explanation") and obviously didn't make that part very clear.

The reason for that is partly because that's how all the current source control helper tools work by extracting the changeset comments (but that could certainly be changed), but more importantly because with a large patch, it's very very easy to overlook the sign-off at the end of the patch.

I only noticed after I applied this, so now you didn't get the distinction of being the first changeset ever to have the sign-off thing recorded ;^)

.. and the race is on.

(Seriously, while nobody has actually complained about the suggested rules, I don't think anybody should feel compelled to do the sign-off before we've had more time to let people argue over it. People who feel comfortable with the suggestion are obviously encouraged to start asap, though).

Albert Cahalan replied:

You're not known for bureaucracy.

The wordy mix-case aspect is kind of annoying, and for all that we don't get to differentiate actions. I count:

came up with the design ideas
wrote the original patch
reviewed and passed on
modified
blindly passed on

Maybe "blindly passed on" needs nothing. So I'm thinking, if we must bother with all this...

designed:
authored:
reviewed:
modified:

Add "pirated:" if you like, so that searching for pirated code is easier than checking the evil bit.

Linus replied:

I actually really really don't want to differentiate actions. There's really no reason to try to separate things out, and quite often the actions are mixed anyway. Besides, if they all end up having the same technical meaning ("I have the right to pass on this patch") having separate flags is just sure to confuse the process.

So what I want is something _really_ simple. Something that is unambigious, and cannot be confused with something else. And in particular, I want that sign-off line to be "strange" enough that there is no possibility of ever writing that line by mistake - so that it is clear that the only reason anybody would write something like "Signed-off-by:" is because it meant _that_ particular thing.

In contrast, your suggestion of "modified:" is something that people might actually write when they write a changelog entry.

One reason for uniqueness is literally for automatic parsing - having scripts that pick up on this, and send ACK messages, or do statistics on who patches tend to go through etc etc.

Mariusz Mazur announced linux-libc-headers version 2.6.5.2, saying:

Available at http://ep09.pld-linux.org/~mmazur/linux-libc-headers/. Changes:

a couple more macros for big endian machines in byteorder/swab.h (eg. hdparm on big endian machines should build)
missing include in linux/aio_abi.h added (don't remember what didn't build because of this)
removed a conflict between linux/if.h and glibc headers

I should be releasing 2.6.6.0 now, but I do not have enough time till the end of May, and the last bug in the above list proved to be a little too popular (at least that's what the number of bug reports would suggest). Since I've accidentally introduced it myself in the previous release, I can't force all those people to patch the thing by hand because of my mistakes :)

CREDITS File MAINTAINERS File Version Control

Adrian Bunk posted a patch to remove John A. Martin from the MAINTAINERS file, where he'd been listed as maintainer of the CREDITS file. John replied:

I cannot remember the last time I've seen a request for CREDITS file maintenance. The frequency of requests seemed to drop off sharply at some point several years ago, perhaps when access to CVS was broadened, IIRC.

I'd be more than happy to continue/resume maintenance of the CREDITS file if that is wanted.

Adrian replied:

The CREDITS file changes usually come from the people listed, and new people are most often added as part of the patch that adds the code they've written.

Your work in the past is appreciated, but effectively Linus and Andrew maintain the CREDITS file today.

End Of Thread.

User-Mode Linux Version Control

Jeff Dike said:

This patch updates UML to 2.6.6. Aside from the update, there were a few small bug fixes.

The 2.6.6-1 UML patch is available at
http://www.user-mode-linux.org/mirror/uml-patch-2.6.6-1.bz2

BK users can pull my 2.5 repository from
http://www.user-mode-linux.org:5000/uml-2.5

For the other UML mirrors and other downloads, see
http://user-mode-linux.sourceforge.net/dl-sf.html