Academic
Publications
Word confidence calibration using a maximum entropy model with constraints on confidence and word distributions

Word confidence calibration using a maximum entropy model with constraints on confidence and word distributions,10.1109/ICASSP.2010.5495606,Dong Yu,Sh

Word confidence calibration using a maximum entropy model with constraints on confidence and word distributions   (Citations: 4)
BibTex | RIS | RefWorks Download
It is widely known that the quality of confidence measure is critical for speech applications. In this paper, we present our recent work on improving word confidence scores by calibrating them using a small set of calibration data when only the recognized word sequence and associated raw confidence scores are made available. The core of our technique is the maximum entropy model with distribution constraints which naturally and effectively make use of the word distribution, the raw confidence-score distribution, and the context information. We demonstrate the effectiveness of our approach by showing that it can achieve relative 38% mean square error (MSE), 39% negative normalized likelihood (NNLL), and 23% equal error rate (EER) reduction on a voice mail transcription data set and relative 35% MSE, 45% NNLL, and 35% EER reduction on a command and control data set.
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.
    • ...Some material contained in this paper was presented at ICASSP 2010 [46], [47]...

    Dong Yuet al. Calibration of Confidence Measures in Speech Recognition

    • ...As pointed out in [14], using a generic model is not desirable for two reasons...
    • ...Second, some information is applicationspecific and is hence not available in training the generic confidence measure [14]...
    • ...The key technique used in this work is the maximum entropy (MaxEnt) model with distribution constraints (MaxEnt-DC) [9], which has been successfully used to improve the quality of word confidence measures as described in the companion paper [14]...
    • ...Different from the work in [14], this paper focuses on the semantic confidence score which requires different set of features, dependents on the word confidence scores, and is more important to the spoken dialog applications...
    • ...Following [14], in this paper the quality of confidence measure is evaluated using the mean square error (MSE), negative normalized log-likelihood (NNLL), equal error rate (EER), and the detection error trade-off (DET) curve [3] defined on a set of �0 confidence scores �? �E and the associated labels �U �E �<�: �? �E�‐�>�r�·�s �?�· �U �E�‐�<�r�·�s �=�; ��� ��EL�s�· �Æ�· �0�= with �U �EL�s indicating ...
    • ...The exact definitions of these criteria can be found in [14]...
    • ...We have explained the MaxEnt-DC model and the special treatment needed for the continuous and multi-valued nominal variables in detail in [14]...
    • ...As shown in [14], the quality of word confidence scores can be greatly improved using the calibration algorithm described therein...
    • ...The W1 and W2 settings use word confidence measures calibrated using the approach described in [14], with the word labels unadjusted and adjusted, respectively, as described in Section 3. All the results reported in the table are obtained using the lowest three (i.e...

    Dong Yuet al. Semantic confidence calibration for spoken dialog applications

    • ...Some material contained in this paper has been presented at ICASSP 2010 [46][47]...

    Dong Yuet al. Calibration of Confidence Measures in Speech Recognition

    • ...Alternatively, a generic classifier can be trained and then recalibrated in a separate step on domain specific data possibly using domain specific features [6, 7]...

    Michael Levit. Leveraging call context information to improve confidence classificati...

Sort by: