The queries used in this evaluation were selected in such a way that they can be answered using data provided by at least one ontology. Precision equals the percentage of correctly answered questions from a given corpus of questions. An answer is described as correct with respect to a query over an ontology or set of ontologies. In order for an answer to be correct, PowerAqua has to align the vocabularies of both the asking query and the answering ontologies. Therefore a valid answer is the one considered correct in the ontology world. PowerAqua fails to give an answer if the knowledge is in the ontology(ies) but it can not find it. Note that a conceptual failure (the knowledge is not in the ontology) is not considered as a PowerAqua failure because the ontology does not cover the information needed to map all the terms or relations in the user query. Moreover, a correct answer, corresponding to a complete ontology-compliant representation, may give no results at all if the ontology is not populated.
Total recall can not be measured in this open scenario, as we dont know in advance how many ontologies can potentially answer the users query. Therefore recall is measured in terms of getting at least an answer or not
We have tested our prototype on a collection of ontologies saved into online repositories and indexed by PowerMap. The collection includes high level ontologies, like ATO, TAP, SUMO, DOLCE, and very large ontologies like SWETO_DBLP or SWETO  with around 800.000 entities and 1.600.000 relations. In total, we collected around 2GBs of data stored in 130 sesame repositories that are accessible through http://kmi-web07.open.ac.uk:8080/sesame. (The running times are an approximation, and it can vary depending on the load of the server and network connections)
The questions used during the evaluation were selected as follows. We asked seven members of KMi, familiar with the Semantic Web and ontologies, to generate questions for the system that were covered by at least one ontology in our collection. We have pointed out to our colleagues that the system is limited in handling temporal information; therefore we asked them not to design questions which required temporal reasoning (e.g. today, last month, between 2004 and 2005, last year, etc). Because no quality control was carried out on the questions, it was admissible for these to contain some spelling mistakes and even grammatical errors. Also, we pointed out that PowerAqua is not a conversational system. Each query is resolved on its own with no references to previous queries. We collected a total of 69 questions listed in what follows.
The current version of PowerAqua can answer correctly 69.5% of the queries. The average time is 15 secs per query. All the repositories used in this evaluation are online at: Sesame repositories (no longer available)
Q1: Give me all papers authored by Enrico Motta in 2007
Q2: What is the title of the paper authored by Enrico Motta in 2007?
Q3: Give me all albums of Metallica
Q4: Which are the albums of the rock group Metallica?
Q5: Give me all californian dry wines.
Q6: Which californian wines are dry?
Q7: Which religion does Easter belong to?
Q8: Which are the main fasting periods in Islam?
Q9: Is there a railway connection between Amsterdam and Berlin?
Q10: What are the main rivers of Russia?
Q11: Which russian rivers flow into the Black Sea?
Q12: Which prizes have been won by Laura Linney?
Q13: What are the symptoms of Parkinson?
Q14: Who stars in Bruce Almighty?
Q15: Who presented a poster at ESWC 2006?
Q16: What is the capital of Turkey?
Q17: Who are the people working in the same place of Paolo Buquet?
Q18: Give me all the articles written by people from KMi.
Q19: Give me the list of restaurants which provide italian food in San Francisco
Q20: Which restaurants are located in San Pablo Ave?
Q21: Which cities are located in the region of Sacramento Area?
Q22: What is the apex lab??
Q23: Who believe in the apocalypse?
Q24: What is skoda doing?
Q25: Where are Sauternes produced?
Q26: Give me the papers written by Marta Sabou.
Q27: which organization employs Enrico Motta?
Q28: where is Odessa?
Q29: which russian cities are close to the Black Sea?
Q30: give me oil industries in Russia.
Q31: Which sea is close to Volgograd?
Q32: Name the president of Russia.
Q33: which countries are members of the EU?
Q34: What is the religion in Russia?
Q35: Which sea does the russian rivers flow to?
Q36: what is activated by the Lacasse enzyme?
Q37: What enzymes are activated by adrenaline?
Q38: what enzymes are used in wine making?
Q39: give me the actors in "the break up"
Q40: Give me fishes in salt water
Q41: Give me the main companies in India.
Q42: What is the birthplace of Albert Einstein
fails to map "birthplace", and it splits the compound "Albert Einstein".
Q43: Show me museums in Chicago.
Q44: Give me types of birds.
Q45: In which country is Mizoguchi?
It can not infere from Riichiro Mizoguchi who is affiliated to the Osaka University from Ontology 1 to the fact that Osaka is in Japan from the Ontology 2. PowerAqua can not do a "double cross-ontology jump" to link two linguistic terms within the same linguistic triple.
Q46: Find all rivers in Asia
15.501 secs (partial answer because, for efficiency reasons, as there is a direct relation between rivers and Asia in the ontology it does not look for the indirect relation river-country-Asia which would have produce more answers within the same ontology)
Q47: Find all Asian cities
It fails to map "Asia" to "Asian", the query can be reformulated to "find me cities in Asia"
Q48: Which universities are in Pennsylvania?
Q49: Who won the best actress award in the toronto film festival?
Q50: Which desserts contain fruits and pastry?
It can not map "dessert" to "dessertDishes" and "fruit" to "fruitDishes"
Q51: Are there any rivers in Russia that flow to Caspian and Black Sea?
Out of coverage.
Q52: What ways are there for weight management?
Q53: What kinds of brain tumour are there?
It can fin the literal "Brain Tumor SY NCI" which is a synonym of the class "Brain Neoplasm".
Q54: Where is Lake Erie?
Q55: Which islands belong to Spain?
Q56: List some Spanish islands.
It can not map "Spanish" to "Spain". It can be reformulated to "list me some islands in Spain"
Q57: Which islands are situated in the Mediterranean sea?
Q58: Which Spanish islands lie in the Mediterranean sea?
Q59: How many airports exist in Canada?
Q60: Which terrorist organization performed attacks in London?
The ontology is not well modeled (redundant terms not connected between themselves). The literal "London, United Kingdom" is discarded by the exact instance mapping "London", which is not related neither to the instance "United Kingdom" nor to the literal "London, United Kingdom", being those two last ontological terms the only ones that link to the class "terrorist organization".
Q61: Which are the main attacks that took place in the United Kingdom?
Q62: What are the terrorist organizations that are active in Spain?
Q63: What type of buildings exist?
It does not find the term "building" in the sweto ontology.
Q64: Which RBA banks are situated in Switzerland?
Q65: What are the bank types that exist in Switzerland?
Q66: How many tutorials were given at iswc-aswc2007?
It can not find "tutorialEvent" as a mapping for "tutorials". The query can be re-formulated to "how many tutorial events were given at iswc-aswc 2007?" (in this case it only maps the term "2007" which is the localname of the entity "/iswc-aswc/2007" with no label).
Q67: How can you treat acne?
It finds the class "acne" which is not connected to any other entity, while it discards the approximate mapping "acneTreatment" that would have lead to the answer.
Q68: What drugs can be used for reducing fever?
it tries to find mappings for the terms “drugs”, “reducing” and “fever” while the answer is contained in an unique class, namely “FeverReducingDrug”
Q69: Which drug can treat colds and reduce headache?
It can not map the term “drug” to the class “drugMedicine” and the term “colds” to the class “coldCoughCareProduct” to obtain for example (ibuprofen, is-a, coldCoughCareProduct) (ibuprofen, is-a, drugMedicine). Nevertheless it maps “cold” to “CommonCold”, however the class “CommonCold” is not connected to “drugMedicine” or to “coldCoughCareProduct”.
Many of PowerAqua failures are because the relevant mappings can not be found, or even if they were found, they were discarded by PowerMap filtering heuristics,, which is a compromise between good performance and recall (exploring all possible mappings that can lead to a solution). Also, in many cases these errors are the consequence of bad modelled ontologies (ontologies with redundant terms not connected between themselves). The linguistic coverage should also be extended by augmenting the grammars or the type of queries PowerAqua can understand.