Before even thinking about testing this thing, I'd like to see some (detailed?) explanation from you why exactly you think the changes in this patch are good and how + why they work.

IMHO it would be good to not apply ANY code to the stable kernel tree unless we understand what it does and what the author meant the code to do...

What the patch does is simply to push I/O as fast as we can.. we're by definition I/O bound and _can't_ defer it under any circumstance, for in this direction lies constipation. The only thing in the world which will make it better is pushing I/O.

If you test the patch, you'll notice one very important thing. The system no longer over-reacts.. as badly. That's a diagnostic point. (On my system under my favorite page turnover rate load, I see my box drowning in a pool of dirty pages.. which it's not allowed to drain)

Note that the Linux VM is certainly different enough that I doubt the comparisons are all that valid. Especially actual virtual memory mapping is basically from another planet altogether, and heuristics that are appropriate for *BSD may not really translate all that better.

I'll take numbers over talk any day. At least Mike had numbers, and possible explanations for them. He also removed more code than he added, which is always a good sign.

In short, please don't argue against numbers.

I have a client that wants to implement a webcache, but is very leery of implementing it on Linux rather than BSD.

They know that iMimic's polymix performance on Linux 2.2.* is half what it is on BSD. Has the Linux 2.4 networking code caught up to BSD?

Can I tell them not to worry about the Linux networking code strangling their webcache product's performance, or not?

I dont think raw network data helps. 2.2 and FreeBSD are basically the same speed for raw networking in the general case. So if someone was seeing real Linux/BSD differences Im concerned it might be a driver but also that it might not have been networking differences but perhaps VM or disk I/O performance. Clearly they saw something since its rather hard to mess up that kind of measuring. I wonder if it was networking though.

The extreme answer to the 2.4 networking performance is the tux specweb benchmarks but they dont answer for all cases clearly.

The locking order was rather nasty in it (mapping->i_shared_lock and mm->page_table_lock), and made a lot of the code much less readable than it should have been. And because none of the callers could know whether the vma existed after being merged, they ended up doing strange things that simply aren't necessary with the much simpler version.

This, coupled with the fact that many merges could be done trivially by hand (much faster), made me drop it. There were a few places where it was used where I couldn't make myself be sure that the locking was right: I could not prove that it was buggy, but I couldn't convince myself that it wasn't, either.

Note how do_brk() does the merging itself (see the comment "Can we just expand an old anonymous mapping?"), and that it's basically free when done that way, with no worries about locking etc. The same could be done fairly trivially in mmap too, but I never saw any real usage patterns that made it look all that worthwhile (The only "testing" I did was really running normal applications and then checking how many merges could be done on /proc/*/maps. Under normal load I did not see very many at all - I had something like six missed merges while running my normal set of applications (X, KDE etc). Others can obviously have very different usage patterns.). Handling the mmap case the same way do_brk() does it would fix the behaviour of this pathological example too..

Also note that the merging tests were not free, so at least under my set of normal load the non-merging code is actually _faster_ than the clever optimized merging. That was what clinched it for me: I absolutely hate to see complexity that doesn't really buy you anything noticeable.

Sorry, I don't believe that that this is correct in this case. I spoke on the telephone with the "Manager for Open Source Systems", and the concept of releasing documents to that a driver could be written whose source would be available was a concept too far. He kept on asking about NDAs- I kept on saying, yes, I'll sign an NDA (presumably so knowledge of advanced features, such as VLAN taggging, e.g., would not be released if they did not want it to be)- but the basic driver source would have to be OPEN! (this was for *BSD, but that's the same as linux in this case- we *all* want the damned source open). No meeting of minds. I have been trying this on and off for two years so that I can properly support the Wiseman && Livengood chipsets in *BSD. No luck, ergo, reverse engineering of what little they release with the Linux driver is the order of the day still. The Linux driver, btw, is pretty clearly a port of an NT driver- which is quite amusing.

FWIW.....I just think that the overall company policy within Intel, much like that of NetApp and others, is, "Open Source? Well, maybe, err,umm.. "... It's just not that important to them (as a company, they think). That said- if you can get access to said documentation (which I understand comes in a certain notebook that indicates releasing outside of Intel is a firing offense)- more power to you!

The way I've gotten so-called proprietary information in the past is to let the world know that "boneserver.analogic.com" 204.178.40.210 is an open ftp site in which I don't even log what's uploaded and downloaded.

I check it once or twice a week to see if somebody has sent me anything of interest. Sometimes, persons unknown to me, have deposited information that I need.

Now I seem to need some programming information on the Intel e-1000. I'll keep you informed if anything turns up.

No it is not strictly required.

But still the 2GB limit is annoying and together with the arch-independent maximum number of swap partitions/files it is pretty dumb.

So I would propose to first make a small patch to make MAX_SWAPFILES arch-dependent and bigger. (x86 would allow a muc higher MAX_SWAPFILES)

For 2.5 we could perhaps think about a new swapfile layout which allows bigger partitions.

The i386 actually support up to 4*16 = 64 swap files (or partitions) with this SWP_TYPE() definition, while include/linux/swap.h does define MAX_SWAPFILES to be 8 ... If that were a pointer array to kmalloc()ed blocks, the limit could be much higher. Indeed I think this is the only *static* limit anywhere in the current swap code.

Similarly it supports 2^24 PAGES of swap at i386 per file/partition. ( 16 million pages of 4k each = 64 GB -- should be enough ;) ) ( That would require vmalloc() to allocate 32 MB block, though. That might not be possible at every occasion -> swapon may fail. )

The more I read the documentation (= source and its comments), the more I am inclined to think that the beast *will* work with swap-partitions (and files!) larger than 2G.

Stephen Tweedie did this 'SWAPSPACE2' work for 2.4 series, what he might tell ? Is it really just a matter of fixing the mkswap utility ? Was Stephen just conservative saying: "Don't go over 2G" (I haven't tested it)

Reviewing thru the architectural definitions of these SWP_***() macroes, the shifts used for SWP_OFFSET seem to vary in between 7-12 and for Alpha and MIPS64: 40. Indeed things are not very easy to understand with 64 bit architectures. It looks like those architectures use the low 32 bits of swp_entry_t for something, while most use at most couple of bits.

Oh, even those 64-bit system seem to give at least 24 bits for PAGE_SIZE 'offset'. The lowest bitcount for 'offset' seems to be at s390 which gives "only" 2^20 * 4k pages, or 4 GB per swap file/partition. (SWP_OFFSET() shifts with 12, which is same as PAGE_SHIFT for the machine. Why SPARC64 uses PAGE_SHIFT in its own unique way, that I don't know.)

Somehow I suspect that the makers of each architecture port have not quite understood what the swp_entry_t bits are used for, and have blindly presumed them to be related to PAGE_SIZE ...

No, the present definition is terrible.

Read the mkswap source. A forest of #ifdefs, and still sometimes user assistance is required because mkswap cannot always figure out what the "pagesize" is.

There are two main problems:

• "new" swap is hardly larger than "old" swap
• the unit in which new swap is measured is a mystery

So, the next swap space has (i) a signature "SWAPSPACE3", (ii) (not strictly necessary) a size given as a 64-bit number in bytes. Moreover, the swapon call must not refuse swapspaces that are larger than the kernel can handle.

I have been using PPPoE in the 2.4.0 kernel for about 2 months now. It's very nice. I used

http://www.math.uwaterloo.ca/~mostrows/

just grab the tarball and compile. I bet it will work under 2.4.2 also.

For devices, the support is already there. See Documentation/pci.txt. Look for 'probe', 'id_table', etc.

I don't think there is support in the current tree for a controller that supports physical hotplugging of PCI adapters, yet. Compaq has a driver outside the tree to do such a thing (needing only very minor kernel patches), see http://opensource.compaq.com

Anybody who has any ideas or input, please holler. There are no actual BusLogic controller changes in the current 2.4.3-pre kernels at all, so there's something else going on.

There's a new aic7xxx driver there - did you enable support for that? I wonder if there could be some inter-action: the aic7xxx driver tries to probe every PCI SCSI controller because they are basically hard to ID any other way (no single vendor/id combination, or even a simple pattern). But it has some rather careful internal logic to filter out all non-aic7xxx controllers, so this really doesn't look likely.

If you didn't compile aic7xxx in, the only other SCSI change (apart from a lot of spelling fixes in comments etc) is some trivial error handling, like changing scsi_test_unit_ready to not have a result buffer (because it doesn't have a result except for the regular sense buffer). Which again certainly shouldn't be able to matter at all.

I'm getting a notable increase in people sending me patches that do major things and should be 2.5 stuff. Please if you want to rewrite the VM completely, redesign the scsi layer and the like wait until 2.5.

Right now I'm only collecting patches that are driver bugfix/updates, arch specific updates/fixes or bugfixes (not feature adds) for the core kernel code.

Anything else goes in the bitbucket

VM folks can post their patches to linux-mm@kvack.org, where we can play with things until 2.5 is forked.

I agree with Alan that we should keep all experimental stuff out of 2.4, probably even out of linux-kernel ...