Nils Klarlund -- ReX, towards a foundation for multimodal, multimedia applications

Solving a universal user interface problem, the ReaX programming language forces the buttons

to always work. (The button marked "|<" is the undo function, the button marked "<<" is the rewind function, the button ">>" is the forward function, and the button ">|" is the redo function.)

Modern user interfaces are often needlessly irking to use, simply because the prosaic problem of providing consistent and workable undo and seek mechanisms have been ignored by programmers. Speech enabling applications not already designed with undo leads to offensive designs. Moreover, small buttons on new devices and generally difficult-to-use applications make the lack of a general undo mechanism a basic problem that we show is readily solvable. Our main technical contribution is a new technique for tying declarative reasoning about time and synchronicity to the basic notion of event.

The simpleness of HTML contributed to the explosive growth of the Internet. But what is the content format that will do the same for multimedia content? The format must combine temporal notions for media coordination, such as SMIL (a W3C recommendation), and a reactive programming language, such as Statecharts or a multithreaded version of Ecmascript.

However, a well-known and fundamental semantic conflict exist between a declarative style of specifying temporal relationships and the programming that addresses user and system events. We propose a new multi-threaded, XML-based scripting language—ReaX—that introduces a notion of reaction tree for capturing causality among events. We show how the explicitly tree-oriented semantics of ReaX overcome semantic problems in reactive languages and SMIL.

Our initial work with Jennifer Beckham and Giuseppe di Fabbrizio outlined how a SMIL-like notation, then called ReX, would form a basis for a reactive language with declarative, temporal features. You can see our ideas in our paper.

The

survey talk (html) [Powerpoint] explains my general interests in user interface technology and how they lead to fundamental questions about events and synchronicity.

ReaX—towards a foundation for multimodal, multimedia applications