We decided to construct this benchmark taking the TREC 9 and TREC 2001 (http://trec.nist.gov/) standards as a starting point, because this provide us with an independently produced set of queries and document judgments. The IR collection we took as basics comprises 10 GB of Web documents known as the TREC WT10G collection, 100 queries, corresponding to real user logs requests, and the list of document judgments related to each query. These judgments allow the quality of the information retrieval techniques to be calculated using standard precision and recall metrics. The aims behind selecting an IR collection are twofold. On the one hand, to be able to evaluate the query results retrieved by querying the Semantic Web. On the other hand, to be able to evaluate the advantage of using semantic information for document retrieval in terms of precision and recall.
In particular, PowerAqua can be used as standalone, and the semantic results obtained form the queries can be evaluated in terms of how well can they used to perform document retrieval by using query expansion. Or it can be integrated with an domain independent IR mechanism, like the one provided by the [1], that uses the semantic information to retrieve documents by applying semantic ranking models.
To evaluate semantic search systems in this benchmark, we required a text collection, a set of queries and corresponding document judgments, ontologies that cover the query topics, and knowledge bases that populate the ontologies. The source of this semantic data should be independent of the query and text collection. However, as the Semantic Web is still sparse and incomplete, many of the query topics associated with WT10G are not yet covered by it. Indeed, we have only found ontologies covering around 20% of the query topics, the ones used for this evaluation. Moreover, due to sparseness, most of these ontologies are barely populated, and they are not enough to perform large-scale semantic search. To overcome this limitation and provide a medium-scale test experimentation of our algorithms, some of the selected ontologies have been semi-automatically populated from an independent information source: Wikipedia.
As a result, we have sourced 40 public ontologies on the Semantic Web, through Watson and Swoogle, which potentially cover a subset of the TREC domains and queries. These ontologies are grouped in 370 files comprising 400MB of RDF, OWL and DAML. In addition to the 40 selected ontologies, our experiments also access to other 100 repositories (2GB of RDF and OWL) stored in sesame and indexed with PowerMap indexing structures, and in a near future, they can also access to other non-predefined ontologies at execution time using the WATSON search engine. For the remaining TREC topics, ontology-based technologies can not be used yet to enhance traditional search methodologies.
Thus, these evaluation do not limit the amount of semantic information used, and, therefore, they do not restrict the amount of topics that that can potentially be covered. Neither the documents, nor the ontologies, nor the links between the two are handpicked or handcrafted.
Here we list the selected TREC topics, each topic has associated a set of basic NL questions (obtained from the title, description and narrative provided by TREC) and ontologies that cover the answer. All the ontologies can be found online under http://kmi-web03.open.ac.uk:8080/sesame (no longer available).
The results of this evaluation are published in:
Fernandez, M., Lopez, V., Motta, E., Sabou, M., Uren, V., Vallet, D., Castells, P. Semantic Search meets the Web. In Proc of the IEEE international conference on Semantic Computing (2008).
[1] Castells, P., Fern�ndez, M., and Vallet, D. An Adaptation of the Vector-Space Model for Ontology-Based Information Retrieval. IEEE Transactions on Knowledge and Data Engineering 19(2), February 2007, pp. 261-272.
TOPICS
Number: 451
What is a Bengals cat?
Description:
Provide information on the Bengal cat breed.
Narrative:
Item should include any information on the Bengal cat breed, including description, origin, characteristics, breeding program, names of breeders and catteries carrying bengals. References which discuss bengal clubs only are not relevant. Discussions of bengal tigers are not relevant.
Questions:
Provide information on the bengal cat breeders
ontologies (domain) : tapfull (animals)
Number: 452
Do beavers live in salt water?
Description:
Describe the normal habitat for beavers; note exceptions, if any.
Narrative:
Relevant documents describe the habitat range as well as references to specific areas and bodies of water.
Questions:
Describe the habitat for beavers.
ontologies (domain) : tapfull (animals)
Number: 454
Parkinson's disease
Description:
What are the symptoms and treatment of Parkinson's Disease, and what segments of the population have this disease?
Narrative:
Documents discussing research projects and funding for research projects were considered relevant only when clinical trials were included. Documents regarding legislation which discussed funding and programs were considered irrelevant.
Questions:
What are the symptoms of Parkinson?
What is the treatment for Parkinson?
ontologies (domain) :tapfull (diseases)
Number: 457
CHEVROLET TRUCKS
Description:
Find documents that address the types of Chevrolet trucks available.
Narrative:
Relevant documents must contain information such as: the length, weight, cargo size, wheelbase, horsepower, cost, etc.
Questions:
Find chevrolets.
ontologies (domain) :tapfull, autos (autos)
Number: 465
Deer
Description:
What kinds of diseases can infect humans due to contact with deer or consumption of deer meat?
Narrative:
Documents explaining the transference of Lyme disease to humans from deer ticks are relevant.
Questions:
What deer diseases can infect humans?
What human diseases are transferred by deers?
ontologies (domain) : tapfull (diseases)
Number: 467
dachshund dachshunds "wiener dog"
Description:
Identify documents that contain information on buying and owning dachshund dogs.
Narrative:
Documents that discuss general dog information which is directly applicable to buying and owning dachshunds (i.e., how to chose a breeder)
are relevant. Documents that list names of dachshund breeders and names of clubs for dachshund owners are relevant.
Questions:
Show me all information about dachshund dog breeders
ontologies (domain) :danchundogs, tapfull (animals)
Number: 476
Jennifer Aniston
Description:
Find documents that identify movies and/or television programs that Jennifer Aniston has appeared in.
Narrative:
Relevant documents include movies and/or television programs that Jennifer Aniston has appeared in.
Questions:
Show me the movies of Jenifer Aniston.
ontologies (domain) : movie_database (cinema)
Number: 484
auto skoda
Description:
Skoda is a heavy industrial complex in Czechoslovakia. Does it manufacture vehicles?
Narrative:
Relevant documents would include references to historic and contemporary automobile and truck production. Non-relevant documents would pertain to armament production.
Questions:
Show me the auto production of Skodas
ontologies (domain) :auto (AUTOS)
Number: 489
calcium
Description:
How do members of the medical profession view the effectiveness of calcium supplements?
Narrative:
Any document which cites the benefits of humans using calcium supplements or advises how calcium supplements should be used are relevant. A relevant document must establish that the information comes from a qualified medical source and not from the claims of a manufacturer or vendor of calcium supplements or from the opinion of anyone not recognized by the medical profession.
Questions:
What is the effectiveness of calcium supplements?
What are the benefits of calcium?
ontologies (domain) :fungalv2 (MEDICINE)
Number: 491
Japanese Wave
Description:
Identify occurrences in which a Japanese wave or tsunami caused loss of life or damage.
Narrative:
Any reports that describe the occurrence of a Japanese wave or tsunami causing loss of life or damage are relevant. A relevant report must describe an actual event occurring at any location.
Questions:
Show me all tsunamis.
Describe disasters produced by tsunamis. (TODO)
ontologies (domain) : phenomenon (NATURAL DISASTERS)
Number: 494
nirvana
Description:
Find information on members of the rock group Nirvana.
Narrative:
Descriptions of members' behavior at various concerts and their performing style is relevant. Information on who wrote certain songs or
a band member's role in producing a song is relevant. Biographical information on members is also relevant.
Questions:
Show me all members of the rock group nirvana.
What are the members of nirvana?
ontologies (domain) : tapfull, music (MUSIC)
Number: 504
information about what manatees eat
Description:
Find documents that describe the diet of the manatee.
Narrative:
Relevant documents will identify any foods providing sustenance to the manatees.
Questions:
What is the diet of the manatee? (no answer)
ontologies (domain) :tap (animals)
Number: 508
hair loss is a symptom of what diseases
Description:
Find diseases for which hair loss is a symptom.
Narrative:
A document is relevant if it positively connects the loss of head hair in humans with a specific disease.
In this context, "thinning hair" and "hair loss" are synonymous. Loss of body and/or facial hair is irrelevant,
as is hair loss caused by drug therapy.
Questions:
of what diseases hair loss is a symptom (TODO-linguistic coverage)
Find diseases for which hair loss is a symptom (TODO � linguistic coverage)
What diseases have symptoms of hair loss?
ontologies (domain) : biomedical(medicine)
Number: 511
diseases caused by smoking?
Description:
What diseases does smoking cause?
Narrative:
A relevant document must describe smoking tobacco products as a cause of a disease. Diseases caused by second-hand smoke and smokeless tobacco are not relevant.
Questions:
What diseases does smoking cause?
What diseases are caused by smoking?
ontologies (domain) : biomedical (medicine)
Number: 512
how are tornadoes formed?
Description:
How are tornadoes formed?
Narrative:
A relevant document will provide the meteorological and atmospheric conditions necessary to create a tornado and explain how the conditions interact to form the funnel-shaped cloud.
Questions:
how are tornadoes formed
Describe the formation of tornadoes
ontologies (domain) : phenomenon (natural disasters)
Number: 513
earthquakes?
Description:
What causes earthquakes, and where do they occur most often?
Narrative:
A relevant document will discuss scientific causes of earthquakes or tremors and/or report geographic areas where earthquake activity occurs most frequently.
Questions:
what causes earthquakes?
where do earthquakes occur?
ontologies (domain) : phenomenon (natural disasters)
Number: 516
halloween?
Description:
When, where, and how did Halloween evolve?
Narrative:
A relevant document will discuss the origin of Halloween and the original
customs of Halloween. Modern day trick-or-treating stories are not
relevant.
Questions:
What is the origin of halloween?
What are the original customs of halloween?
ontologies (domain) :stconcepts (festivities)
Number: 519
info on where frogs live
Description:
Find documents that describe the habitat of frogs.
Narrative:
A relevant document will identify the natural habitat of any type of frog. A frog's diet is not relevant.
Questions:
Where do frogs live?
Describe the habitats for frogs?
ontologies (domain) :animals-wh (animals)
Number: 523
facts about the five main clouds?
Description:
How are the five main types of clouds formed?
Narrative:
A document that explains the process of cloud formation for any of the five main types of clouds is relevant. A document that discusses clouds, but does not explain their formation processes is not relevant.
Questions:
How are the clouds formed?
Describe the formation of clouds.
Explain the process of cloud formation
ontologies (domain) :phenomenon (natural world)
Number: 524
how to erase scar?
Description:
What methods are used for removal of scar tissue?
Narrative:
A relevant document must disclose the name of a procedure or describe it, or identify the instrument used to remove scar tissue or skin defects. Mere references to "surgical removal" are insufficient.
Questions:
How to erase a scar?
How to remove a scar?
ontologies (domain) : galen (medicine)
Number: 526
bmi
Narrative:
Any document that gives defines or explains BMI is relevant.
Questions:
what is BMI?
ontologies (domain) : form_demo (medicine)
