Wine Traffic Brian Vincent

This is the 290th issue of the Wine Weekly News publication. Its main goal is to hate the new pipermail web interface. It also serves to inform you of what's going on around Wine. Wine is an open source implementation of the Windows API on top of X and Unix. Think of it as a Windows compatibility layer. Wine does not require Microsoft Windows, as it is a completely alternative implementation consisting of 100% Microsoft-free code, but it can optionally use native system DLLs if they are available. You can find more info at www.winehq.org
News News

We're back this week after taking a few weeks off. Things have been pretty hectic here and something had to give. In addition, the new Pipermail archives make building the section headings a complete pain in the ass. Since it changes the way I've gotten used to putting issues together, I'm not really sure what the publishing schedule of WWN will be going forward. It's added enough overhead and extra time to make things really frustrating. If you currently have, or would be willing to set up, Hypermail archives of wine-devel you'd be my best friend forever and ever.

I figured now might be a good time to look at where Wine's at and what's going to happen in the near future. You might remember that Alexandre set a deadline of September 30th for a beta release of Wine. Well, I sent Alexandre an email last week to see if things were on schedule. Apparently Alexandre is pretty busy right now and that deadline is going to slip by a few weeks. Interestingly, there's some large features being worked on (partly covered below), so there's a chance we might be able to sneak a few extra features in.

We also have the Wine To Do list and we haven't looked at it in a while. The last time we looked at it we had green (completed projects) beating red (unstarted projects) 36 to 8. We also had 27 projects in progress. Here's the current standings:

  • Green (completed): 50
  • Yellow (in progress): 15
  • Red (not started): 7

Some of the items that haven't been started aren't critical for a 0.9 release and a few could even be seen as only being nice to have. So overall, things are looking good.

Architecture Microsoft

Device drivers have always been a hot topic with Wine. The idea of some how running Windows device drivers with Wine comes up every few months. In recent years, some of the discussion has actually gotten pretty serious and some of the ideas have actually seemed somewhat plausible. For instance, CaptiveNTFS actually led to a real, working implementation of getting Microsoft's ntfs.sys to operate under Linux and let NTFS filesystems work under Linux. As you can imagine, these kind of things are hard.

Device drivers have been popping up more and more on wine-devel, especially in the context of things like the Safedisc(tm) (evil) copy protection driver. Now, it's not just enough to fake a device driver into loading; you also have to make it usable by the standard Win32 API. Vitaliy Margolen asked this week:

We need ntoskrnl to run some subset of drivers on wine. Notably cd protection drivers. They are not a hardware drivers, but a means to access something that is not accessible from user space. So in a sense ntoskrnl is just a process to run those drivers in, and send/receive requests to/from those drivers.

You need to keep in mind that this is a kernel level, so some things will not work, and some things have to work that normally do not in the user space. Also ntoskrnl for windows _is_ what wineserver for wine.

User space programs talk to these drivers in the following way: 1. At the start up, driver creates symbolic link in "\DosDevices\something". 2. Program calls CreateFile("\\.\something",..) to get a handle to the device. 3. Uses ReadFile/WriteFile/DeviceIoControl on that handle. (For now we need only DeviceIoControl. Read/Write could be added later.)

Here is a list of the most pressing problem and the way to solve them.

Any ideas and comments will be greatly appreciated, Vitaliy.

Problems:

  1. Open handle to a device - NtCreateFile.
  2. Call DeviceIoControl on a device. (This also includes (1) since we need to know that this is not a cdrom, vxd or something else.)
  3. Resolve user space handle to the DEVICE_OBJECT. (Assuming that we don't call ntoskrnl through wineserver).

Solutions for (1):

    1.1. Use a hack to determine that passed name is a device name. (Device names don't have colon in them).
    1.2. Check against all possible names other then device names. (A:..Z:, etc.)
    1.3. Try it last if all else fails. (since name spaces don't intersect this should work.)
    1.4. Use object manager to lookup the name to find out what is it. (Unless we open all handles using this method it will add one extra wineserver call.)

Solutions for (2):

  • ntdll->ntoskrnl
      2.1. Use named pipe to communicate with ntoskrnl process. Pass commands over the pipe in internal format.
      2.2. Use UNIX IPC to talk to ntoskrnl. (pipe, socket, etc.)
  • ntdll->wineserver->ntoskrnl
      2.3. Call wineserver which will call ntoskrnl using 2.1. or 2.2.
      2.4. Call wineserver which will call ntoskrnl using proper IoXXX functions from ntoskrnl to create an IRP. ntoskrnl will process them and notify wineserver when call is complete. (This is the most complicated and closer represents what windows does. There are still a problem communicating to ntoskrnl. It is probably the way to go in the future. But that will require number of things implemented which are not trivial.)

Solutions for (3):
MSDN quote: "Note that handles are not used for accessing device objects or driver objects."

  • 3.1. Use wineserver to resolve handle to some internal identifier and pass it to ntoskrnl.
  • 3.2. Pass handle and calling process id (pid, global handle, etc.) to ntoskrnl. ntoskrnl will call wineserver to resolve handle to internal id. (This handle will have to be either cached, or it will have to be created for every call.)
  • 3.3. (requires 2.3. or 2.4.) Wineserver will resolve handle and pass internal id to ntoskrnl. (Unless we want wineserver do much more than that.)

Mike McCormack felt the solution was even more complicated:

I think that the only way to make device drivers work properly is to eliminate wineserver completely, and make it what it really wants to be, which is a user mode implementation of ntoskrnl, something like User Mode Linux.

That would require being able to write to a process's memory from the ntoskrnl, which is what your really need to implement ReadFile and WriteFile so device driver can work properly.

That project is really beyond the scope of Wine... but an interesting one none the less :)

Peter Beutner wanted to know about one of the methods Vitaliy had mentioned, Why implement ntoskrnl as a seperate process plus inventing a new IPC protocol to talk to it? As you said: "ntoskrnl for windows _is_ what wineserver for wine.". So why not implement the needed ntoskrnl stuff into wineserver?

Ivan Leo Puoti (who's recently dealt with this concerning the large ntoskrnl patch he put together) answered, Great idea, but Alexandre doesn't want drivers running in wineserver.

Peter wanted to know the underlying reason and Alexandre replied, Stability is the obvious reason. And also of course the fact that we have most of the code we need in ntdll already and none of it in wineserver.

Marcus Meissner pointed out another reason why drivers loaded via the wineserver won't work, This ntoskrnl is meant to load binary windows drivers. It will never live in the wineserver due to this reason

Rob Shearman expanded on that and also answered a question about why Wine loading drivers would be more unstable than on Windows, Except that Windows doesn't have missing functions or functions that aren't quite completely implemented. As Alexandre says, there is a ton of stuff that would have to be duplicated in order to be able to load Windows drivers in the wineserver. For example, PE loader, virtual memory, exceptions / signal handling and threading. Also note that the wineserver is single threaded (and it would be a big task to make it thread-safe) and that whenever you are calling driver code then all Wine apps are blocked.

There was more discussion about why ntoskrnl needed to talk to different Windows processes and such. Vitaliy then summarized the discussion and asked for more ideas:

First let me thank everyone helping me and Ivan to get ntoskrnl moving forward.

Summary of all the ideas and requirements:

  1. wineserver should not be used to run drivers (absence of required functionality, single threaded, stability issues)
  2. ntoskrnl should be used to run drivers only and link to ntdll and not the other way around. Unless we implement int 0x2e handling and replace wineserver with ntoskrnl. (this is wine not reactos, ntdll already has all the required functionality)
  3. Drivers can not be ran in a process that requires them (drivers we are concerned about keep information that is shared between different processes, drivers require their own environment that is different from a user process environment, ntdll has no means nor should it have them to run a driver)

And that's it.

This validates original design by Ivan Leo Puoti for ntoskrnl. Now, there are still unanswered questions:

  1. How to talk to ntoskrnl (directly from ntdll or through wineserver).
  2. How to get information required for ntoskrnl from wineserver.
  3. How to identify this is a device operation and not a file/named pipe/mail slot, etc.

I know folks you have more ideas, keep them coming.

A few more ideas were discussed, but the architecture Vitaliy described seemed to be the consensus.

Project Management CodeWeavers Alexandre Julliard Jeremy Newman Marcus Meissner

One topic that comes up from time to time is Wine's organization. Open source projects have developed many different methods for getting code from a patch into the actual codebase. In Wine's case, Alexandre Julliard reviews and commits each patch himself. That's a pretty difficult model to pull off, especially when it comes to reviewing the number of patches that go into Wine.

Each year at WineConf we've discussed whether that model works or not. In fact, it's usually one of the first topics that gets addressed. Each year we've decided that the present model is working. Alexandre is comfortable handling the amount of patches coming in and everyone else thinks his judgement is correct. In the event a patch doesn't end up in CVS, Alexandre has said he has a not-quite-sure bin of patches where he keeps a running list of patches that he's unsure of. Sometimes they just need to be reworked slightly to get in, other times they're so large it takes a while to review them. It's not uncommon for a patch to end up there with no response. If it doesn't get committed, he suggests contacting him a few days after a patch hits wine-patches to find out why.

Now, one thing not addressed at WineConf is new developers. Figuring out that process may be quite frustrating for new developers and we don't really hear about it at WineConf since we don't have many new developers there. A few weeks ago there was a discussion that started out as someone that seemed to think there were tons of forks of Wine, meandered into car analogies, and ended up with a long discussion of the Wine management structure. It seems to point to a very real communication problem. At the end of the thread it appeared a few things could be done to help the process, however it doesn't appear anyone will step up to work on it. real solutions were offered.

Troy Rollo started the discussion with:

I should preface this by saying that I don't mean any criticism or offence by this. You asked the question and the following reflects my observations of current limits to the development model and what would have to be changed to improve scalability. That doesn't mean the model has not served well thus far or with the available pool of developers to date.

Having to pipe all the changes through one person limits scalability. Until recently the facilities didn't even exist for supporting the maximum scalability using that model (distributed branching), and even though they exist now few people are using them and they are not integrated into the central commit process.

Even as things are now there is a disincentive to new developers. Some developers don't bother submitting patches because they feel it's too much work to get them accepted, and sometimes the patch doesn't get written at all because there is not enough certainty that it will be accepted. Before you suggest that this is speculative, I have one developer here who has made patches to fix Wine bugs experienced at home and not submitted them for that exact reason. You can try getting guidance on what is likely to be accepted, but again you run into a bandwidth problem with Alexandre - one person cannot effectively give guidance to everybody who needs it, and such guidance as he is able to give doesn't always help that much. The process requires that developers risk their work amounting to nothing because it won't be accepted. How many times have you seen people say that "Alexandre doesn't always know what he wants, but he knows what he doesn't want"?. That perception is partly due to the fact that it is impossible for one person to take the time to really understand what is wanted for every area of Wine that is being worked on at any one time (humans are linear, it limits them).

I suspect the current model is either at or near its limits. It would certainly not cope with a significant number of commercial outfits putting in a serious level of contribution, nor does it encourage them to make the attempt.

To scale better you would need to divide the project into different areas of responsibility and have multiple committers. You could have overall guidelines, but you'd need to be willing to trust the managers of each area to make final decisions in that area - even so far as to let them make mistakes and let them learn from those mistakes. Almost every subdirectory of "dlls" could in principle be a separate area of responsibility with a separate group of committers, although it might be more convenient to have more-or-less related subdirectories under the umbrella of a single group. Each group responsible for an area would need to be prepared to give advice to people working on some patch in that area that if followed would make it virtually certain that the patch would be accepted.

A supporting facility you would probably need would likely include either separate mailing lists for each sub-project (things get lost in wine-devel now with consolidated read-only lists for people who want a convenient way of watching all of the discussions or a shift to an NNTP heirarchy with mailing gateways.

These structures are nothing new - most of the really big projects work this way - look at Gnome, KDE, and for the insane end of the scale, Debian.

Even with the Linux kernel, which is the only project that I am aware of that is comparable in size and follows a similar model to Wine, there appears to me to be some greater division of responsibility.

This is all relevant, of course, only if significant expansion of the pool of developers is a desirable goal.

Quite a few people immediately replied to point out things that they disagreed with. However, Ivan Leo Puoti agreed with "Alexandre knows what he doesn't want" comment, That's a problem Vitaliy and I now have, ntoskrnl can run kernel mode drivers just fine (Ok, it has for months but only recently it's started using real device handles), but Alexandre doesn't like some things about it and doesn't have any alternative solutions, so we're sort of stuck with 4100 lines of working code, but no immediate prospect of getting them committed. Also I've recently noticed that very few of the patches being submitted are being committed, mainly because Alexandre appears to be *very* busy with some work (mac support maybe?), so dependencies of ntoskrnl that could go in now are still pending approval, the result is development isn't as fast as it could be because we aren't sure what to do next, have no indication of what really needs to be changed in our code, and we have to wait several days to know if what little we have submitted is ok, and when you've got loads of patches that depend on each other that is a bit of an issue.

Bob Lunnon mentioned some his frustrations:

I must disagree, the LOTM (Lord Of The Manor) governance model may work for an small outfit but wine has already outgrown it. I have two or three withheld patches which are absolute show stoppers for running wine under Solaris. They are withheld despite the fact they work because they were refused, yet every second week I am forced to work around some portability problem introduced by someone else - not exactly ISO9001 quality assurance. This causes problems for both me and the wine project because:

  1. wine is NOT as portable as it should be.
  2. I am forced to become the LOTM for wine under Solaris since I am currently the only source (I know of) for Solaris wine.

There must have been half a dozen times where I have decided to abandon the wine project due to its governance model, only to be encouraged back to it by my customers. These days I submit my patches to comply with the LGPL, and if they go in all well and good, if not I no longer care... Is this how developers should be thinking about wine ?

It's also important to remember that many developers that contribute (Including myself) are volunteers, volunteers are hard to come by, but really easy to get rid of. You need a governance model that is not only fair and even handed with people, but is SEEN TO BE fair and even handed. This model is not that.

Marcus Meissner didn't think changing Wine's development model would help at all. He suggested resending patches and asking for feedback.

Bob followed up with another email discussing Wine's governance model. Among other things, he mentioned:

Now I'm not saying here that Alexandre isn't objective, just that his criteria of assessment is not clear to me, and after 2 years doing this I still reckon I have about a 50/50 chance of having anything major accepted. If it's not clear to me, then it must be positively opaque to someone new. You might say that the project is open, but the governance is closed in that the developer and user communities have little influence over governance.

Discussion then turned toward managing patches. Everyone's concern is we're losing patches because they don't get committed. Jeremy White mentioned it was something that had been thought of in the past:

We actually have a todo on Jeremy Newman's list to build a patch management system for wine-devel, for Alexandre.

Our hope was that we could adopt some of the CodeWeavers systems (we have a ticket system that's pretty slick, for example).

However, it became clear that the requirements were fairly substantial (the tight emacs integration became our first clue :-/), and that project got back burnered.

At the time we were discussing that, though, we didn't have many volunteer web programmers; maybe we should revisit that. Alexandre, would you be interested if folks other than Jer volunteered to help build such a system?

With that said, I have to ask - what open source projects are you guys working on that don't suffer from these problems? I'm now a successful contributor to the Linux Kernel (tweaked isofs for Windows CDs) and it took me 3 years and countless dropped emails, despite the personal help of Alan Cox and Andrew Morton before my patch got in (and I have another patch, a minor bug fix, that I despair will ever see the light of day). I had a similar situation with MythTV (and the #mythtv channel is actively hostile to anyone mildly clueless), and the list goes on.

Based on my experiences, I would say that Wine is a cut above, and Alexandre does a very fine job.

On the other hand, I've often thought that the developer section should have a big FAQ to help explain how Alexandre works (notably the fact that he uses the absolute minimum amount of communication required at any time) *grin*

That's pretty much where things were left. It's clear there's a few people who are unhappy with Wine's development model, but the majority of Wine's developers seem content. It's unclear whether the existing system is preventing new developers from joining, but certainly no one wants that. The two items Jeremy identified, a patch management utility and documentation concerning patch submission tips, didn't receive any volunteers.

Web/HTML Microsoft Jacek Caban

Microsoft and most other Windows software vendors began moving away from traditional help files years ago. Most help files are now created using Compressed HTML files. They offer significant benefits over regular help files and take advantage of capabilities built into Internet Explorer's HTML rendering libraries. As you can imagine, short of developing our own web browser, Wine is stuck without HTML support.

Well, fortunately we are building our own web browser. Rather, Jacek Caban is integrating a Gecko engine into Wine and building the associated interfaces around it. James Hawkins has been working in parallel to get HTML Help to work and gave a status update of where things are at:

As soon as the last hhctrl patch is committed (embedding in the correct window), HTML Help will be functional. That means that you can open up a chm help file with wine's help viewer, hh. It requires IE6 to be installed for the time being, until our shdocvw and mshtml are up to par. You'll also need a copy of native itss.dll (it has to be registered with wine):

    $ regsvr32 itss.dll
    $ WINEDLLOVERRIDES=shdocvw,mshtml,urlmon,shlwapi,itss=n wine hh helpfile.chm

A few things need to be completed, but not much. Most noticeably the toolbar buttons need to be drawn and submitted to wine-patches. If you would like to help in that area, send me an email and I'll let you know how it needs to be done. The tabs are blank and need filling in. Attached is a screenshot of hh in action. If anyone has any comments or suggestions, let me know and I'll address those.

Of course, screenshots are always nice to see and James included one of CHM rendering.

DirectX

Oliver Stieber dropped a note with a DirectX status update. You might remember that Oliver began working on DirectX 9 about nine months ago. He maintained much of that code outside of the Wine tree for a long time and only began submitting patches a few months ago. That was sort of frustrating for a lot of Wine developers, but things have really progressed. Oliver has broken his DirectX work into small, contained patches and Alexandre has committed almost all of them. Oliver wrote:

I thought it was about time I gave a status update on DirectX9/wined3d, so here goes.

Short of a couple of bug fixes (one to shaders, and another that causes some games to crash) and a couple of patches that need resending (support for non-power2 textures in warhammer 40k) wined3d and d3d9 is functionally as complete as the version on http://directxwine.sourceforge.net/, that means that as soon as pixel shaders are copied across from d3d8 it's more or less the end of adding features to wined3d and the start of fixing the remaining bugs and getting the performance up to spec.

Anyhow, as soon as I've sent in those patches I'll provide another status update with a bit more information about moving d3d8 over to wined3d and the improvements that have been made.

Status Updates Vitaliy Margolen Brad DeMorrow Tom Wickline cvs Marcus Meissner Laurent Pinchart

Ivan Leo Puoti announced this week that Safedisc copy protection now works with Wine. We've covered this in past issues and the approach being taken by Wine. Safedisc loads a special driver in order to operate. Rather than reverse engineering the driver or bypassing it, Wine "simply" needs to load it. Well, that means Wine needs to implement low-level API's used by the real Windows kernel, ntoskrnl. Blah, blah, blah.. show us some screenshots!

Finally we've got safedisc1 running on linux. Thanks go to Laurent Pinchart, Vitaliy Margolen, Brad DeMorrow, Marcus Meissner, and Alexandre for contributing time/code/ideas. The code is 100% DMCA compliant and hopefully can be cleaned up to be good enough for cvs over the coming days and weeks. Two games have been tested, they both use the same version of safedisc (don't ask how we know, we do)

Oh, and here's an actual in game shot, if you're a fan of the game you'll probably notice I'm out of practice.

One problem Ivan has been running into is implementing the code in a way Alexandre approves of. See the other thread in this issue about getting patches into Wine.

Tom Wickline asked which versions of Safedisc were supported. Ivan replied, 1.50.20 is the version we used to implement support in wine, so yes it supports secdrv.sys and dplayerx.dll and the encryption applied to the ICD.

WineHQ WineHQ Jonathon Wilson cvs

Jeremy Newman upgraded WineHQ a few weeks ago. We had some downtime as a result, but nothing major. Since there was an ISP change it took a while for the DNS change to propagate, but even that happened relatively quickly.

Jeremy announced what had changed:

The webserver is back online. There are still plenty of quirks to fix. Feel free to bug me here on wine-devel about any issues that crop up.

ChangeLog:

  • moved OS to Debian Sarge
  • now using - Apache 2.0.54 / PHP 4.3.10
  • switched from sendmail to Exim4
  • upgraded mailing lists from mailman 2.0 to 2.1
  • mailing lists now using pipermail for archiving, note: old links will work as the hypermail archives still exist

I'm still tweaking configs and all so expect the site to be a bit erratic over the next few days.

A few issues cropped up, such as cvsup coming online later in the day, but for the most part the move was seamless. Jeremy White was the first with congratulations:

Just because no one else has said it yet:

Nice work, Jer!

(Jer is masking all the behind the scenese wrangling he did with our two ISPs to arrange this, and the mad scramble to replace a raided drive that had fried, and the rebuild to Debian. )

It was darn nice work on his part to get it all essentially done in a day.

The change to Pipermail and the way it handles attachments seemed to cause some issues. Jeremy worked on the MIME magic and the problems seemed to go away. Later, Jonathon Wilson asked if GD2 was on the webserver and Newman replied it was. That opens the door for some AppDB improvements for things like screenshots.

Is Wine SMP safe? Well.. it certainly seems like a topic someone would have brought up if there were issues. Dan Kegel asked about a bug report:

I've seen conflicting reports about whether Wine is safe to run on SMP systems.

Have the issues described in http://www.winehq.org/hypermail/wine-devel/2005/01/0060.html been taken care of?

I just pinged the reporter of the only bug with SMP in the summary, http://bugs.winehq.org/show_bug.cgi?id=2708 to see if he can still reproduce it.

Michael Stefaniuc wasn't aware of any issues, I'm running it since 1.5 years on a SMP system (P4, HT enabled and smp kernel). I don't remember to have run into SMP problems. Or better said in the problems i tried to debug there wasn't a SMP bug.

MSI Aric Stewart Microsoft Ivan Gyurdiev

Microsoft Installer really seems to be the problem of last year. Mike McCormack and Aric Stewart spent a lot of time figuring out how MSI works and creating Wine's builtin library. While much of the framework was put in place last year, extensive work has been done this year fleshing out the API. Like many Microsoft API's, MSI has been overdesigned with tons of features (in this case, MSI's "Actions") that go unused. It seems like most of what the real world uses in MSI now works with Wine. Mike McCormack asked for people to go out and break Wine's MSI:

If anybody is aware of any cases of Wine's Microsoft Installer code not working, please log them in Wine's bugzilla (http://bugs.winehq.org/), assign them to me, and I will attempt to fix them.

Make sure to set "msi" = "builtin", "*msiexec" = "builtin" when testing your installers, and if possible test with the latest Wine CVS.

If the installer is available as a free download, please include a link to it.

Ivan Gyurdiev quickly pointed out bug #2899 which showed MSI problems with Half-Life 2.