Concept: add classlike methods to processes
Processes or library routines could have API hooks through which other programs can request information about the process without effecting the process itself, or perhaps the method could cause the process to change its behavior.
Example: tar would have hooks for an external program to check the name of the current file being tar'd, see the percentage complete for the file, and see the percentage complete for the whole batch of files. Any external progam (with appropriate permissions) can ask tar to report its status. When the method is called, a thread/fork is created which will not crash tar if the thread/fork crashes. No one would need to rewrite or copy code from tar to create a new graphical status bar for their archival program.
Concept: OS built on structured datatypes + XPath file access methods
Unix is currently built on unstructured text, with logic separated into different programs with their own data structures. When connecting different programs together to work on each others' data, one program formats the data into plain text and the other parses the plain text back into data. This is wasted CPU time.
In a hypothetical operating system, communications between processes may include standardized data structures that both processes understand. I hear that CORBA does this, but I've never taken a look at it and hear that it's extremely difficult to write. I would extend this data-structure inter-process communication concept down to the filesystem and the shell tools.
People promoting this sort of thing generally describe it as "XML" because the XML world includes a standardized and readily available set of formats and APIs for accessing this kind of information. The XPath language in particular makes specifying a place in a hierarchy as easy and powerful as regular expressions make specifying part of a string.
How an XML-enabled filesystem would work:
You could run: cat /etc/passwd[@uid=0]@login
to get a list of user names whose UID is 0. No need to use sed or learn regular expressions, and the password file does not need to be in traditional Unix format -- the system might put logins in the eighth field and the command would still work.
The xpath would be parsed by some level of the operating system, not the program. The passwd file would be converted into a nodeset containing a series of login names. The nodeset might be converted into plain text for printing by the shell, or perhaps the console might also be XML-enabled and allow you to perform operations on the login names by clicking on them to bring up a context menu.
If another process wanted to get a standard Unix text-formatted passwd file, it would get it, and a saved file would be converted into whatever the storage format is if it is well formed. XML lets you do this sort of thing. Also, each field in the passwd file might have its own security settings. For instance, normal users might not be able to access the @password field. This would make shadowing unnecessary.
To get these capabilities, every program on Unix will need to be rewritten to take nodesets. The C language may even need to be rewritten to add syntactic sugar for programming with XML. The data structures being passed around would basically be the structs and classes, with the compiler taking care of exposing and integrating them into the system.
Concept: Event-based filesystem
Allow users and administrators to script onLoad, onClose, onExecute events onto files. The scripting is done at the filesystem level, so no recompiling of programs is necessary. Scripts can be run with the same UID as the user, or can be suid (logging scripts). Scripts can be placed in a directory (such as, say, /) so that actions on any file within the directory trigger the event.
In a hypothetical system, the event model may be so fine-grained that entry into and exit from specific functions in an executable can be scripted, again without editing or recompiling the executable.