Task Definition and Contribution

MM 0430/2018

A new task of machine comprehension of spoken content was proposed.

TOEFL listening comprehension test is taken as an corpus.

The questions in TOEFL can not be answered by simply matching the words in the question. The key information is usually buried by many irrelevant utterances.

A listening comprehension model, Attention-based Multi-hop Recurrent Neural Network (AMRNN) framwork for the TOEFL task was proposed.

In this proposed approach, the audio of the stories is first transcribed into text by ASR, and the proposed model is developed to process the transcriptions for selecting the correct answer out of 4 choices given the question. The initial experiments showed that the proposed model achieves encouraging scores on the TOEFL listening comprehension test.

The attention-mechanism can be applied on either word or sentence levels. sentence-level attention was found to achieved better results on the manual transcriptions without ASR errors, but word-level attention outperformed the sentence-level on ASR transcriptions with errors.

[0]

B. H. Tseng, S. S. Shen, H. Y. Lee, L. S. Lee, ``Towards machine comprehension of spoken content: Initial TOEFL listening comprehension test by machine," Towards machine comprehension of spoken content: Initial TOEFL listening comprehension test by machine, 2016.