Academic
Publications
Improving the Generation of Ground Truths based on Partially Ordered Lists

Improving the Generation of Ground Truths based on Partially Ordered Lists,Julián Urbano,Mónica Marrero,Diego Martín,Juan Lloréns

Improving the Generation of Ground Truths based on Partially Ordered Lists  
BibTex | RIS | RefWorks Download
Ground truths based on partially ordered lists have been used for some years now to evaluate the effectiveness of Music Information Retrieval systems, especially in tasks related to symbolic melodic similarity. However, there has been practically no meta-evaluation to measure or improve the correctness of these evaluations. In this paper we revise the methodology used to generate these ground truths and disclose some issues that need to be addressed. In particular, we focus on the arrangement and aggrega- tion of the relevant results, and show that it is not possi- ble to ensure lists completely consistent. We develop a measure of consistency based on Average Dynamic Re- call and propose several alternatives to arrange the lists, all of which prove to be more consistent than the original method. The results of the MIREX 2005 evaluation are revisited using these alternative ground truths.
Cumulative Annual
View Publication
The following links allow you to view full publications. These links are maintained by other sources not affiliated with Microsoft Academic Search.