The Linux kernel has enforced different behavior for processes starting with “X”

A nasty hack in the Linux kernel that has been in mainline for over three years has been called out. Due to a buggy X.Org Server / xf86 video mode setting DDX, the Linux kernel has imposed different behavior on whether a process starts with “X” and in turn disable support for atomic mode setting.

Linux security researcher Jason Donenfeld (who is also known for coming up with WireGuard), stumbled upon this nasty kernel hack.

Donenfeld commented this weekend on the core mailing list:

This reverts 26b1d3b527e7 (“drm/atomic: Take the atomic toys away from X”), a rootkit-like kludge that has no meaning in a general kernel. It’s the kind of debugging hack I want to use at a moment’s notice but never commit to, or some kind of babies-first-process-hides-malware trick.

The background is that some userspace code — xorg-server — has a mode setting DDX which is not really coded correctly. When no one wanted to maintain X11 anymore, instead of fixing the buggy code, the kernel was adjusted to avoid having to touch X11. A bummer, but fair enough: if the kernel doesn’t want to support any userspace API anymore, the right thing to do is to provide a graceful fallback where the userspace thinks it’s not available in a manageable way.

But the way * to do that is to just check `current->comm[0] == ‘X’`, and disable it for that case only. So that means it’s *not* just a matter of the kernel not wanting to support a particular userspace API anymore, but rather it’s the kernel not wanting to support xorg server, in theory but actually, it turns out , there are all processes that begin with ‘X’.

Playing power->communication games like this is obviously wrong and it’s quite shocking that this was ever committed.

The commit to this kernel with the “X” first character check was made back in September 2019. The argument in that Linux kernel commit at the time was:

-modesetting ddx has a completely broken idea of ​​how atomic works:
– does not disable old contacts, assuming they are automatically disabled as with the older setcrtc
– assumes ASYNC_FLIP is wired through for the atomic ioctl
– not a single call to TEST_ONLY

[In other words] the implementation is a 1:1 translation of legacy ioctls to atomic, which is a) broken b) pointless.

We already have errors in both i915 and amdgpu-DC where this prevents us from activating neat functions.

If anyone ever cares about atomic in X, we can easily add a new atomic level (req->value == 2) for X to bring back the shiny toys.

Since these broken versions of -modesetting have been shipped, there is really no other way to get out of this bind.

The “good” news is that since that time on the userspace page back in 2019, the xf86-video-mode-settings code went ahead and disabled atomic support by default. So technically, if you’re running an updated X.Org stack within the last three years, this core hack is no longer necessary since userspace just then avoids the atomic API.

We’ll see if Linus Torvalds is okay with this hack being removed, since after all it goes against his principle of “not breaking user space” by then regressing the system if he sticks to an outdated X.Org Server stack and want to drive. with a future core version. We’ll see, but surprisingly it took three years for this dirty code to be criticized.