|
1
|
- Nils Klarlund
- Clairgrove, LLC
|
|
2
|
- Provoke reflection on spoken language for operating machinery
- Were we deluded by
- HAL and Startrek?
- The idea that speech input would be inherently natural when working on
the computer (a la “NaturallySpeaking)”?
- Even if yes, there is a fascinating opportunity to make speech
recognition viable as a tool with no assumptions about the presence of
intelligence outside the brain
|
|
3
|
- Typing injury (> 10 years)
- The question “Did humans evolve so that they could operate complex
machinery only through their fingers?” becomes of interest
- The answer is “no” already by virtue of our feet
- Pedal control has limited bandwidth, so the overriding question is:
|
|
4
|
- Do the following tasks
- Capitalize the next three words
- Move to the beginning of the line and replace two words there with “A
new”
- Insert two new lines and backup to the place between the new lines
- Delete what is inside the parenthesis that enclose text where the mouse
pointer is and insert “practically speaking”
- Place cursor at last “l” in the preceding word “travelling”
- Split the current window to show the file “newtrace”
|
|
5
|
- You are editing a paper with a colleague
- Your colleague has the mouse and the keyboard
- You have an editing intention in mind
- What happens:
- “Uh, move this word over here, no, no, one line down, and then insert
‘the’, which you should capitalize, oh no, but not that ‘the’, the next
‘the’…”
- At which point you physically take position of the mouse while you’re
both yanking at the keyboard
|
|
6
|
- NL has no meaning!
- NL is verbose!
- “Move to the beginning of the line and replace two words there with ‘A
new’”
- NL is not a tool!
- Since tools allow concise sequencing of atomic actions
- NL commands are confusable with NL dictation!
- Current dictation system are most unnatural to use as a result (because
pauses must be inserted by user
|
|
7
|
- The belief that natural language is a feasible substitution for tools
whose use is characterized by the unconscious sequencing of well-defined
atomic operations
- The four reasons natural language doesn’t work
- No meaning issue
- Verbosity
- Syntax too complicated
- Mode problem (for dictation systems)
|
|
8
|
- Precise, specific concepts (instead of no meaning)
- “copy selection,” “move cursor to beginning of line”,…
- Concise symbolization as keys (instead of verbosity)
- Sequences of keys can be arbitrarily composed (instead of complicated
syntax)
- No confusion as what are commands (instead of mode problem)
- But Shift, Ctrl, Alt keys are slow and results in chaotic naming of
concepts
|
|
9
|
- These languages can be written but not read, some “sentences” or
“paragraphs” perhaps are
- C-x C-b M-x v i tab w e q RET C-e (Emacs)
- j j j a the RET ESC (vi)
- C-left C-S-left C-x C-right right C-v (Windows)
|
|
10
|
- Director of the Office of Scientific Research and Development, Dr.
Vannevar Bush
- In July of 1945, Bush published a remarkably prescient outline of
research directions “As we may think”
- A universal language is one that should be understood and spoken by
every one
- Next, the excerpt. ..
|
|
11
|
- Predicts speech-to-text apparatus, inspired by theVocoder, a Bell Labs
invention
- “Our present languages are not
especially adapted to this sort of mechanization, it is true. It is
strange that the inventors of universal languages have not seized upon
the idea of producing one which better fitted the technique for
transmitting and recording speech.”
|
|
12
|
- Speech recognition for natural language works really quite well
(especially if user trained)
- No need for universal constructed language at least on phonetic grounds
(most of us will be relieved!)
- Instead, we need specialized languages for controlling the computer
itself. Such languages are simple and semi-universal for their domain
(each is known by many people).
|
|
13
|
- Robert A. Heinlein, a sci-fi author
- Reference thanks to Jeffrey Bilmes
- A perfect language is one that mirror the true nature of objects, the
precise nature of ideas (Umberto Eco: “The search for a perfect
language”)
- A perfect language need not to be universal (and vice versa)
- Next, the ideas in the novel “Gulf”…
|
|
14
|
- 850 words sufficient to express anything that could be expressed by
"normal" human vocabularies: Ogden’s Basic English (not
sci-fi)
- "Blood, toil, tears and sweat" becomes "Blood, hard work,
eyewash and body water”
- Phoneticians analyzed all human tongues into about 100-odd sounds,
represented by the letters of a general phonetic alphabet
- Each capable of variation: length, pitch, …
|
|
15
|
- “Without much refinement of accepted phonetic practice, it was possible
to establish a one-to-one relationship with Basic English so that
- one phonetic symbol was equivalent to an entire word in a “normal”
language,
- one word was equal to an entire sentence”
- “Speedtalk was not shorthand basic English…other languages made
scientific, multi-valued logic almost impossible to achieve; in Speedtalk it was difficult not to
be logical.”
|
|
16
|
- “In the syntax of speech talk the paradox of the Spanish barber could
not even be expressed, safe as a self-evident error”
- Paradoxes are verbal, they do not exist in the real world.
|
|
17
|
- A pool of 215,999 words (603-1) available for specialized
meaning, most of them monosyllabic
- “The ability to learn Speedtalk at all is proof of supernormal
intelligence” (and your “association speed” is 3x that of normal humans)
- “Speedtalk enabled him to manipulate symbols approximately seven times
as fast as English symbols could be manipulated”
- So, total gain is 7 x 3 = 21!!
|
|
18
|
- Well, nothing really, but principles of sentence formation apply them to
verbalize keyboard languages!
- 1 billion users of such non-verbalized languages
|
|
19
|
- Verbalize the symbols of keyboard languages?
- Yes but keyboard languages are much too poor
- Instead, start over, make a constructed, perfect language
- Phonemes must carry units of editing meaning
- Syllables are words
- Words are command phrases
- That is what ShortTalk is (although not quite as perfect as Speedtalk)
|
|
20
|
- ai
- Go forward from cursor as in syllable nairx (next)
- The word nairx-rye means “go to next ‘)’ after the cursor”
- gnaith “go neither”, i.e. “down arrow”
- oo
- Go backward from cursor
- noorx-rye means “go to the nearest ‘)’ before the cursor”
- goop “go up”
|
|
21
|
- ..ft
- The end of what is found
- “chaiw-aift the most”: chew up (delete forward) one character after the
first occurrence of “the most” after the cursor
- b..f
- The beginning of what is found
- “chaiw-baif the most”: chew up (delete forward) one character before
the first occurrence of “the most” after the cursor
|
|
22
|
- ghin, ex,…
- chaiw, choock
- Delete a character (forward direction, backward direction)
- hare, tair
- Here (cursor), there (mouse)
- ane, twain, traio,.. (strictly
speaking traio is not a syllable)
- smack, grab,…
- Delete, make a copy at cursor,…
- line, senten, inner,…
- Line, sentence, inside of parenthesis or quotes
|
|
23
|
- Capitalize the next three words
- Go to the “l” in the preceding word “travelling”
- Split the current window to show the file “new trace”
- sploot-menu new trace
- Note: this ShortTalk word that has an English sentence fragment as an
argument
|
|
24
|
- Move to the beginning of the line and replace two words there with ‘a
new’”
- ghin-line, smack-twain, a new
- Insert two new lines and backup to the place between the new lines
- Delete what is inside the parenthesis that enclose text where the mouse
pointer is and insert “practically speaking”
- smack-inner-gook, practically speaking
|
|
25
|
- ShortTalk constructed so that none of the command words is confusable
with a common fragment of English
- Without this property, the mode problem would reappear and destroy any
hope of fluency
- Demo:
- letter.wmv (Windows), letter.rm (Real Networks)
|
|
26
|
- Estimate time for various actions and sum them to provide indication of
how long specific edit tasks take
- Combining mouse and keyboard is very expensive due to need to home hands
- ShortTalk + pointing for many tasks several times faster because homing
is eliminated and because of much more succinct expression of editing
intention
- But analysis does not include cognition effort
|
|
27
|
- From data collected over a couple of months
- n log n estimate of information rate is about 16 bits/second for
ShortTalk commands (assuming four syllables/second speech output)
- About 1000 different commands used
|
|
28
|
- Repetitive work is often unfeasible by voice:
- “page down, page down, page up, page down”
- Clearly feet are underutilized when sitting at the computer
- State of the art: at most one or two pedals offered for control of say
voice recorder or for delete key or mouse button
- Loose pedals, not combined with foot rests, and completely insufficient
coverage of repetitive functions
|
|
29
|
- ShortStep is my 7th or so attempt at producing a foot
keyboard that
- Covers virtually all control functions
- Shft-tab, Alt-tab, Alt-Shft-Tab, left, Ctrl-b, Shft-left, PgUp,
Ctrl-Alt-s, Ctrl-Esc, Backspace, …
- Doesn’t hurt
- Is fast (foot pedals on an organ are spaced far apart)
- Doesn’t require you to take your socks off
|
|
30
|
- Browsing, reading
- Process your email
- Process your photos
- Some need to be rotated others not
- Edit your text
- Just pressing the return key and the backspace key by feet saves time
and effort (because they’re a little out of the way)
- CAD applications
- Programming
|
|
31
|
|
|
32
|
|
|
33
|
- Pointing (and hence the mouse) cannot be replaced in general
- Fortunately, a foot keyboard offers a very significant advantage:
- The mouse buttons can be eliminated
- Why important: think of a graphic pen and the bothersome button on the
side
- Similar problem with mouse…
|
|
34
|
- Problem: ever-present tension because fingers must be constantly lifted
in order not to press the buttons
- Consequently, the more “ergonomic” the mouse is (in terms of large
button and light pressure) the more injurious it may be
- Without buttons the mouse can be made small, which makes it possible
for the whole hand to rest on the table, again significantly reducing
static tension
- These theories are medically unproven (so far…)
|
|
35
|
- There is so much to be done for computer users who are disabled, even
for those who have just sores and pains
- Questions raised about whether natural language is really necessary---at
what level of user sophistication or need does natural language offer
any advantages?
- The USI (Universal Speech Interface research at CMU) is less radical
than ShortTalk but also advocates simple keywords for controlling
interactive voice response systems
|
|
36
|
- Hallelujah & Congratulations!... I have been thinking along these
lines for years now
- Very exciting video using ShortTalk inside of Emacs...
- I would really like to try your ShortTalk product.
- I am very impressed with your work...I have to warn you, I'm really really really hoping I can
motivate you to find a way so that I can try ShortTalk
|
|
37
|
- Were we deluded by HAL? Yes and no.
- Certainly, in my opinion yes, when it comes to operating text
editors. Natural language makes
no sense.
- The human mind must be trained and efficiency is the very best
motivation
- Research question: Is there something in the brains or conditioning of
humans that makes our hands inherently better suited over voice for
editing and similar very complex sequencing of operations?
|
Notes
