Ideas on modernizing the open-source desktop

Welcome to LWN.net The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!

Scott Jenson has been working on user interfaces (UIs) and user experience (UX) for many years at Apple, Google, and other companies. Now, he's trying to convince open-source projects to experiment more and drive the desktop beyond the age-old "windows, icons, menus, pointer" (WIMP) model. At Akademy 2026, KDE's annual developer conference, he shared his complaints and ideas in a talk aimed at convincing those in attendance to take the lead on desktop design.

He introduced himself as someone who has been doing UX for a long time, much of that time at Google. He said that over that time he had noticed a cultural shift, "because back in 2005 when I joined, we weren't evil; we were really trying to do the right thing". He had worked on the Chrome team for a while, which he said was "filled with people who wanted nothing but open-source, open-web" projects. Things had changed radically, which led him to leave Google, "but I'm trying to atone for my sins working for these companies".

He now does work for Mastodon and Home Assistant, "just kind of trying to squeeze a little bit more good UX design into open source". Jenson said he did not want to be confrontational, but he thought that open source needs a lot of help in the area of UX design. That is challenging, because open-source projects are understaffed.

Innovation

He was at Akademy to discuss his ideas because he had given a talk similar to this at Ubuntu's conference last year, which proved to be popular. "It got more than half a million views on YouTube—which by YouTube standards is trivial, but by Scott standards is enormous." That told him that the topic of desktop UX was something that people wanted to talk about. This time around, he also wanted to show a few examples of prototypes he had to demonstrate his ideas.

He thinks that desktop UX is very different from other kinds of UX. Jenson related a story about working on the Macintosh's Finder application, which is a file manager. The team "wanted to do ellipses" in file names when showing a file in the Finder that had a name too long to display in its entirety. The developers wanted to put the ellipsis at the end of the file name, but he thought that would lose too much information, and suggested that the ellipses should be in the middle of the file names. "It's now considered to be one of those things that's an exciting little attention-to-detail issue the Macintosh does."

Another UX detail that he worked on is the way that clicking and dragging files from one window to another works on the Mac. A mouse click action has two parts: when the user depresses the button (mouse down) and when the user releases the button (mouse up). On the Mac, item selection happens on mouse down, but the window only raises on the mouse up action. In other words, if a user drags an icon from a Finder window into another window, the first window is not raised.

On most Linux systems, though, if a user is trying to click and drag a file from a window in the background it will cause the window to raise as soon as the user does the "mouse down" action. That complicates trying to click and drag a file from one window to another. That is a fundamental action that means that users can't move data into applications quite as easily on Linux as they can on the Mac.

He said that he had talked about that issue two years ago on Mastodon, which raised a response from a KDE developer who thought it was a good idea to separate the mouse down and mouse up actions when dragging data between windows. Then they implemented it a few hours later.

"I forgot who that was. So if anybody here did that", he said, but before he could finish the sentence a voice from somewhere in the room responded, "you're welcome". Jenson yelled "thank you! I've been wanting to meet you for two years!" That was an example, he said, of how open source can be awesome: someone can just say "that's a good idea, I'll just do it" and then rapidly implement and push the feature to the project.

Stabilized

The point he was really trying to make, though, is that the desktop UX had not really changed in 20 years. There were plenty of new ideas and improvements in the 1980s and 1990s, "and then it just stabilized, and not much has really changed". But people are doing much more with their computers today, and he thinks that there are ways for desktop UX to grow.

Much of the UX for Linux systems was originally copied from Windows and the Mac, "KDE even says it's for Windows users, that it wants to be similar to Windows for users". There's nothing wrong with that, but he felt that the Linux community "in general" had been leaning on Microsoft and Apple to do all of the UX research and testing, "and frankly, [make] the mistakes".

Some might point out that GNOME has conducted user research for some of its design changes, and KDE's human-interface guidelines (HIG) were also developed based on research by its contributors. However, it would be fair to say that these are exceptions rather than the norm; certainly, no one is funding or conducting regular UX design research and development for the Linux desktop overall.

He said that it is smart to let Apple and Microsoft do these things first "so we can draft behind them", but the problem is that "they've just kind of given up to an extent". Apple has been letting its desktop UX, and desktop hardware, "languish for years and years and years" because it had shifted its focus to the iPhone and iPad devices. "I mean, like, Apple wanted the Mac to die. They wanted the world to move over to the iPad. Guess what? It didn't."

Apple has since turned its attention back to the Mac, but the things they've done since—such as the Liquid Glass design language—have not been really popular with users. The features that Apple tends to add, he said, are "basically about tying it closer to the iPhone, tying you closer into their ecosystem". That isn't about improving the desktop, it's about increasing their moat.

Microsoft "is just making one mistake after another". He cited the company's attempts to force users to store files using its OneDrive cloud-storage service, the privacy-invasive Windows Recall feature that drew widespread criticism, and "ads everywhere" on the Windows desktop. His point was not to bash Apple or Microsoft, though the audience didn't seem to object to this, it was to point out that the Linux desktop could not follow their example any longer. "The whole Linux community has been just waiting for Apple and Microsoft, more or less, to take all the risks, and they're not taking any more risks."

Three pushbacks

Jenson emphasized that he didn't want to change everything, but he wanted to see experiments and additional development aimed at improving the desktop experience. However, he said that he gets pushback when he tells people that desktop innovation has stopped, and that the Linux desktop "can't just patch and bug fix our way into the future, we have to start taking leadership ourselves".

He grouped the pushback into three categories. The first objection he receives is "mobile won; the desktop is old, just do what mobile does". He dismissed that by saying that mobile has indeed won the consumer market, social media, and other areas, but "it didn't win productivity". People still do their work on computers, because "desktops are silently awesome; they have keyboards and big screens, and they can do amazing things". However, people have lost sight of that and don't really appreciate how good desktops are. "I think we should understand why they're good, and then what we can do to make them better."

The second objection he hears is that there are only so many ways that the WIMP model can work. "Just stop trying to fix it; it's done." He disagreed with that as well, and thought that there was more to be done. The final, particularly strong, objection from the tech community is "don't touch my stuff". People get angry when they have worked hard to get their desktop environment just as they like it and something changes to disrupt that. "And I get that. As I said before, I don't want to change everything." But the world is changing, "and we need to grow our way gently into some new things."

What is UX?

Thinking about where to go next with desktop design gets to the question "what is UX?" He said that term was misunderstood and that people, especially manager types, "they think it's just pixels, but UX is so much more than pixels". It helps to think of UX as having layers, he said. There is style, which consists of white space, iconography, color schemes, and yes, pixels. But there is also structure, he said, which is how the user moves through the application, the navigation model.

Strategy is another part of UX; understanding who the target user is, and determining what they want. Deciding what is important, and what is not; what will the development team prioritize and what will it ignore? "The fastest way a UX designer can save your team time is to help you say no."

There is also a layer he called "stuff": the lower-level limitations of the underlying technology that a platform has which influence UX. For example, he said, if an application is being created for DOS, "you're not going to write a nice Macintosh application. The technology dictates what you can do". The desktop has been evolving for 40 years and has "historical cruft" that carries expected standards.

We just have a bunch of stuff that we just accepted. And I think if we're going to fix the desktop, we have to understand this lower-level limitation, and how do we want to fix that? We just accept that mobile and desktop has stuff today, and we just don't think about fixing it.

People forget, he said, that the original Macintosh screen was "a whopping 342 pixels high". Naturally, windows overlapped a lot. Since then, "we just assumed that Windows have to overlap". But many people have "giant, giant screens", and the existing window-manager model does not work so well for those devices. "I can understand why people use tiled window systems and so forth, because this existing model just does not work very well."

Understanding the future

Early in the talk, he had referenced a saying by Alan Kay, who had led design of the first WIMP interface at Xerox PARC: "Perspective is worth 80 IQ points." Jenson now circled back to Kay: "Who, as you can tell, I really like a lot. [...] He talks about three steps to understand stuff". The first step, he said, was to think about the present, "what is happening right now?" The second is "don't forget the past, know what work has happened before", and the final step is to ensure that "your questions are really clear". Too many people rush to a solution, he said, but they should be rushing to questions. If the questions are clear, then the answers will be better.

He did not want to make the talk about AI, he said, but he had to address it. "I have to talk about AI, and it pisses me off because there's so many cool things we could be talking about, and AI just sucks up all the oxygen, and it's all you can talk about". But, he conceded, there are many things happening with AI on the desktop, albeit in a weird way.

He explained some of the current efforts to integrate AI with the desktop as "trying to sneak AI in" while keeping the UI exactly the same. Other efforts try to use AI to understand what is on the screen and use chatbots to interact with the interface, "which I think is kind of missing the point here". That is, until last summer when Anthropic introduced Claude Code. "It completely changed things, because was looking down into your filesystem" and drew context from the content of a user's files. That allowed it to do more interesting things and interact with the system.

After mentioning Claude, he paused to say he was only speaking about frontier models because everyone else is discussing them. "I have a lot of personal problems with the frontier models. They are ethical and environmental disasters. I am not very excited by them". He said he was excited by Apertus, which is billed as a fully open, responsible model for sovereign AI created by the Swiss AI Initiative. "I do think it's possible to talk about ethically trained small language models running locally, and you don't have to sell your soul to the devil."

The curse of direct manipulation

The key point he wanted to make was not about generative AI uses, but about how giving it access to the filesystem improved its usability. It now had context and could work with the user's data, but it was working at the filesystem level and not using the desktop UX.

He talked about the desktop building blocks, basically the familiar WIMP model plus the desktop clipboard, as the way that users move data around. Expert users can move data around quickly, he said, but there's a problem: the desktop is stateless. He called this the curse of direct manipulation; there is no working memory to the desktop. "If you copy a few too many things to the clipboard, oh sorry, it's gone".

Jenson theorized that the reason there are so many desktop environments, window managers, and Wayland compositors for Linux—as well as alternative clipboard managers—is because the desktop isn't quite working right for users. He thought it was really about working memory: "How do I manage my data in such a way that I can remember it across all of these things and actually use it." His perspective shifted to asking "what would we do to working memory to improve the desktop UX?"

AI may have given him the idea, but he added that he wanted to solve the desktop UX for people first. "If the AI can use it, great. I don't care. That comes later."

Remember your past

There is prior art in trying to create systems that help provide users with more context and working memory related to their data. He referenced a paper called "Lifestreams: a storage model for personal data". That paper made the case that desktop systems are "are ill-equipped to manage the electronic information and events of the typical computer user", and introduced the Lifestreams idea as a metaphor for dynamically organizing a user's computer workspace.

That paper influenced the ill-fated WinFS project from Microsoft, which was an attempt to merge data storage and management based on relational database systems. It was demonstrated in 2003 and scheduled to ship sometime later, but ultimately the project was shelved in 2006. On the free-software side of the house, there was the semantic desktop project NEPOMUK, which was a research project funded by the European Union.

A version of NEPOMUK was included in KDE 4, with the idea was that it would allow applications "to use information from all over the desktop, the web, other devices, and combine it into one coherent interface". It was later removed and replaced with the Baloo file-indexing and search framework for the Plasma desktop, which has some overlap with NEPOMUK's functionality but with a reduced scope of features.

Jenson said that he is concerned that people see the failure of those projects as proof that the ideas were wrong. He argued that is not the case, "I would say that it is the hardware and systems at the time let the vision down. I think it's time to rethink these projects" in light of newer hardware and better systems.

Prototypes

Having visited the present and the past, as Kay recommended, he had come up with a few questions that he thought worth exploring related to desktop UX. The first was "what would happen if we only designed for large monitors?" What would it unlock if there were desktop designs that ignored laptops and only focused on widescreen monitors? Another question he had was how could the desktop capture a user's intent over time?

Those questions led to the prototypes that he said he was a little nervous to show the audience "because technical people love to find mistakes". The demos are about 29 minutes into the video of Jenson's talk. His slides have not yet been published but he told me at the event that they would be at some point.

The first demo was to show how a desktop might make better use of a widescreen display. He noted that the center of a widescreen monitor is good for working with an application, but it becomes more difficult to work on the sides of the monitor. "It's good for peripheral vision, but not good for working".

He demonstrated a desktop layout, a screenshot of which is below, that would put the focus on the window or windows in the middle of the monitor. He had included a grid in the background that tapers off from the middle of the screen to simulate a widescreen monitor. The idea was that a widescreen-first desktop would emphasize information in the middle of the screen, and then use an Exposé-like tiling model to arrange the rest of the desktop's windows. This would make the windows that were not in use easier to interact with, he thought, and be a better model than virtual desktops for managing lots of windows. "I'm not against virtual desktops. Virtual desktops are used by very organized and intelligent people. I know my audience." However, he said, most people can't deal with virtual desktops.

He also demonstrated dragging a music-player window all the way to one side of the screen, which he called "the stash area". As it moved to the edge, it transformed into a widget with just a play button instead of showing the full window. He thought that it would be possible to do the things he was demonstrating with Wayland as it is today.

Jenson put up another demonstration of an improved clipboard model that copied ideas from the Obsidian editor's Canvas feature. Canvas allows users to drag and drop images, text, and files to create a visual layout of information. A user could create a document and drag information from a web browser into the document, drop a file in from another source, "the idea here is that this is now effectively a collection of stuff that I have gathered for this document. And if I close it and then come back tomorrow, it's all still there". He added that a local AI would be able to come in and organize the information a user had gathered, "oh, look, you've got a bunch of hotels, let me organize them for you".

The final idea he discussed was a way of gathering "privacy-preserving data" to help give users a better visualization of things like web-browsing history. A web browser will only give the user a view of their browsing history that "is not very useful" he said, so he tried to gather data "not using AI, just using simple math" that would use attention signals—such as how long the user was on a web page—to tell a story of where they spent their time on the desktop. "I basically came up with this spatial associative episodic memory prompt" to organize data in a helpful, interesting way. He thought that working memory was a good way to explore new ideas for desktop UX.

So the goal with this talk is to say, let's start to take this apart. And I really hope that my prototypes cause you guys to come to me and say, We forgot about this, and let's talk about that. [...] So I just want us to try, because no one else is going to try, and I think someone's got to start.

There was time for a single question. An audience member asked how Jenson's model would protect against all of the privacy concerns that Microsoft faced with Windows Recall. "How are we going to make sure that this history remains only accessible by the user and cannot be hacked in like it could happen with Windows Recall?"

Jenson answered that his idea was to gather telemetry data, "not interesting information", which would significantly reduce the attractiveness of the data to would-be attackers. He said he did not want to underplay the issue, though, and wanted to build prototypes to see if the idea was good first, and then explore ways to encrypt or protect the data. He agreed that even this data could be valuable information. He added that the biggest problem with Windows Recall was "how stupidly they protected it" and he was sure others could do a lot better job.

[I would like to thank the Linux Foundation, LWN's travel sponsor, for its assistance with my trip to Graz, Austria for Akademy 2026.]