Voice Communication Between Humans and Machines (1994) / Chapter Skim
Currently Skimming:
The Roles of Language Processing in a Spoken Language Interface
The Roles of Language Processing in a Spoken Language Interface
Pages 217-237
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 217... ...
As a result, most current systems use a loosely coupled, unidirectional interface, such as N-best or a word network, with natural language constraints as a postprocess, to filter or resort the recognizes output. However, the level of discourse context provides significant constraint on what people can talk about and how things can be referred to; when the system becomes an active participant, it can influence this order.
|
From page 218... ...
A number of questions were raised during the discussion, including whether a single system could provide both understanding and constraint, what the future role of discourse should be, how to evaluate performance on interactive systems, and whether we are moving in the right direction toward realizing the goal of interactive human-machine communication.i Background: The ARPA Spoken Language Program Much of the research discussed at the natural language understanding session was done in connection with the Advanced Research Projects Agency's (ARPA) Spoken Language Systems program.
|
From page 219... ...
To date, there is still no agreed upon metric for the rich multidimensional space of interactive systems, which includes the system's ability to communicate effectively with the user, as well as an ability to understand what the user is trying to accomplish. The remainder of this paper is divided into four sections: "The Dual Role of Language Processing" discusses the role of language processing in providing both understanding and constraint; "The Role of Discourse" outlines several sources of discourse and conversational constraints that are available at the inter-sentential level; "Evaluation" returns to the issue of evaluation and how it has affected research; and the final section, "Conclusions," summarizes these issues in terms of how they affect the development of deployable spoken language systems.
|
From page 220... ...
Given recognizer output from the ATIS task, the observers could not reliably distinguish correctly transcribed output from incorrectly transcribed output, due to the irregularities that characterize spontaneous speech. 5Actually, partial analysis is equally critical for large-scale text-understanding applications, as documented in the Proceedings of the Fourth Message Understanding Conference (1992)
|
From page 221... ...
However, if the language-understanding component is to provide an additional knowledge source to help in choosing the "right answer," it must have access to multiple hypotheses from the recognizes. The N-best interface (Chow and Schwartz, 1990; Schwartz et al., 1992~6 has proved to be a convenient vehicle for such experimentation: it makes it easy to interface the recognition and understanding components, it requires no change to either component, and it permits off-line exploration of a large search space.
|
From page 222... ...
but a somewhat larger decrease in sentence recognition error (1.7 percent, from 48.9 percent for N = 1, to 47.2 percent for N = 5~.9 Alternatively, if the language-processing system can provide scores for alternate hypotheses, the hypotheses could be (re~ranked by a weighted combination of recognition and language-understanding score. Use of an LR parser produced over 10 percent reduction in error rate for both sentence error and word error when used in this way (Goddeau, 1992~.1° In summary, there have been some preliminary successes 7This model has not yet been coupled to the recognizes, to determine its effectiveness in the context of a complete spoken language system.
|
From page 223... ...
However, more experiments need to be done to demonstrate that it is possible to use the same language-processing system in both recognition and language understanding to produce word error rates that are better than those of the best current systems. It is clearer that language processing can be used to improve understanding scores, given alternatives from the recognizes.
|
From page 224... ...
sentence error, which is a recognition measure requiring that all the words in a sentence be correctly transcribed; 2. natural language understanding, which uses the transcribed input to compute the understanding error rate (100 - % Correct)
|
From page 225... ...
_ NL Understanding Error Word Recognition Error FIGURE 1 Error rate decrease over time for spoken language systems. (SLS, Spoken language system; NL, natural language.)
|
From page 226... ...
These figures lead to several conclusions: . Both speech recognition and language understanding have made impressive progress since the first benchmarks were run in 1990; the understanding error rate has been reduced by a factor of 3 for both spoken language and natural language in less than 2 years.
|
From page 227... ...
THE ROLE OF DISCOURSE The preceding section discussed the state of language understanding and the limited constraint it provided. The systems discussed above focused mainly on within-sentence constraint.
|
From page 228... ...
semantic information in over 90 percent of the cases. This mode of interaction clearly imposes very strong constraints for recognition and understanding, but these con straints have not yet been incorporated into running systems in the ARPA community, in part because the evaluation metrics have discouraged use of interactive dialogue (see discussion in the section "Evaluation" below)
|
From page 229... ...
People tend to perform tasks in a systematic way, which provides ordering to their exploration of the search space. For example, in looking at opening sentences for tasks in the ATIS domain, we find that 94 percent of the initial sentences contain information about departure city and destination city, as in "What is the cheapest one-way fare from Boston to Denver?
|
From page 230... ...
Even though these types of constraint look promising, it is always hard to predict which knowledge sources will complement existing knowledge sources and which will overlap, producing little improvement. Building spoken language systems is very much an iterative trial-and-error process, where different knowledge sources are exploited to see what effect they have on overall system performance.
|
From page 231... ...
However, it is now time to look again to extending our suite of evaluation methods to focus research on new directions. The preceding section argued that natural language understanding could contribute more constraint if we go beyond individual sentences to look at discourse.
|
From page 232... ...
During this period, four of the ARPA sites con i8Unevaluable utterances are still not counted in the understanding scores, but they are included as part of the input data; system responses for unevaluable queries are currently just ignored. For recognition rates, however, all sentences are counted as included in the evaluation.
|
From page 233... ...
CONCLUSIONS We can draw several conclusions from the preceding discussion about the role of language understanding in current spoken language systems: · Current systems can correctly answer correctly almost 90 percent of the spoken input in a limited domain such as air travel planning. This indicates that natural language processing has become robust enough to provide useful levels of understanding for spoken language systems in restricted domains.
|
From page 234... ...
When evaluating interactive systems, there seems to be no obvious way of factoring the user out of the experiment. Such experiments require careful controls for subject variability, but without such experiments we may gain little insight into what techniques help users accomplish tasks.
|
From page 235... ...
For language technology, this is still largely a manual procedure. Even for recognition that uses automated procedures for training, a significant amount of application-specific training data are required.20 To support widespread use of spoken language interfaces, it is crucial to provide lowcost porting tools; otherwise, applications will be limited to those few that have such a high payoff that it is profitable to spend significant resources building the specific application interface.
|
From page 236... ...
Goodine, and M Phillips, "Integrating Syntax and Semantics into Spoken Language Understanding," Proceedings of the DARPA Speech and Natural Language Workshop, P
|
From page 237... ...
"February 1992 DARPA ATIS Benchmark Test Results Summary," Proceedings of the Fifth DARPA Speech and Natural Language Workshop, M Marcus (ed.)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.