Kyoto-project.eu

Berlin-Brandenburgische Akademie der Wissenschaften, J¨agerstr. 22/23, 10117 Berlin, Germany ontology, ontology development, ontology evaluation, rigidity, type, role, WordNet In this paper we present Rudify, a set of tools designed for the semi-automatic evaluation of ontologicalmeta-properties based on lexical realizations of these meta-properties in natural language. We describe thedevelopment of Rudify, provide an evaluation of initial output, and describe how this output can be used inconjunction with OntoClean (Guarino and Welty, 2002) to produce clean ontological hierarchies. In particularwe show how a Rudify evaluation of concepts for the meta-property of rigidity can facilitate modelling typesand roles.
Section 4 discusses the meta-property of rigidity andits relation to the type–role distinction. Section 5 dis- Developing an ontology requires paying especial at- cusses the development of Rudify. In section 6 the tention to the hierarchical relations.
notion of base concepts is briefly introduced. A set lar, taking into consideration certain meta-properties of base concepts was used for the evaluation of the of the concepts modelled in the ontology can help Rudify output (section 7). Finally, section 8 provides the developer avoid formal contradiction and un- specific examples of how Rudify output can be used sound inheritance of properties (Guarino and Welty, to “clean up” hierarchical relations within an ontol- 2004). However, manually determining ontological meta-properties of concepts within large ontologiesis time consuming and has been shown to producea low level of agreement amongst human annotators (V¨olker et al., 2005). A further difficulty around theannotation of meta-properties is that evaluating themeta-properties of concepts can be difficult for non- The Kyoto project is a content enabling system that ontologists while evaluating technical concepts from performs deep semantic analysis and searches and a specific domain may be difficult for ontologists who that models and shares knowledge across different do- mains and different language communities. Seman- In this paper we present Rudify, a set of tools tic processors are used for concept and data extrac- for the semi-automatic determination of ontological tion, and the resulting knowledge can be used across meta-properties. Rudify has been used for ontology the different linguistic communities. A wiki environ- development within the Kyoto project (Herold et al., ment allows domain specialists to maintain the sys- tem. Kyoto is currently being targeted toward the en- Section 2 of this paper provides an overview of the vironmental domain and will initially accommodate Kyoto project with particular emphasis on the role of seven languages, namely, English, Dutch, Spanish, the ontology. Section 3 contains a brief description Italian, Basque, Chinese, and Japanese. The system of OntoClean, a method for evaluating hierarchical depends on an ontology that has been linked to lex- relations in an ontology (Guarino and Welty, 2002).
ical databases (wordnets) for these languages. The role of the ontology is to provide a coherent, stable tween types and roles, since every type is a rigid con- and unified frame of reference for the interpretation cept and every role is a non-rigid concept. Second, it of concepts used in automatic inference. For more is relatively easy to find lexical patterns for rigidity.
information on the Kyoto project see (Vossen et al., The lexical patterns are a crucial prerequisite for the 2008) and http://www.kyoto-project.eu/.
programmatical determination of meta-properties as Kyoto should be able to accommodate, not only a done by Rudify (see section 5). Third, AEON (V¨olker variety of languages and domains of knowledge, but et al., 2008) also concentrated on rigidity, so there is also changes in scientific theories as both the world and our knowledge of the world change. We, there-fore, require an ontology that is not idiosyncratic, butrather one that can accommodate 1. a variety of languages and their wordnets, The notion of rigidity relies on the philosophical no-tion of essence. An essential concept is one that nec- essarily holds for all of its instances. For example, being an animal is essential to being a cat since it is 5. can serve as the basis of sound, formal reasoning.
impossible for a cat to not be an animal, while beinga pet is not essential because any cat can, in theory, Because the end users will be able to maintain and roam the streets and, thereby, not be a pet. The idea extend the ontology, it is crucial that the ontology is of essence contains an idea of permanence; Fluffy the extended in a clean and consistent manner by non- cat is an animal for the entire duration of his life.
However, the notion of essence is stronger than per- With this aim in mind we have developed Ru- manence. While Fluffy can be a pet for his entire life, dify. We are using Rudify in conjunction with Onto- it nevertheless remains possible for him to cease be- Clean in order to build and maintain a clean ontology.
By evaluating the ontological meta-properties of con- Armed with the notion of essence, we can now cepts, Rudify facilitates a major step in the construc- define rigidity. A rigid concept is a concept that is es- tion and maintenance of clean hierarchies.
sential to all of its possible instances, i. e., every thingthat could be a cat is in fact a cat. Therefore, “cat” is arigid concept. However, “pet” is a non-rigid concept since there are individual pets that do not have to be apet.
OntoClean (Guarino and Welty, 2002) is a method for Non-rigidity further subdivides into two meta- evaluating ontological taxonomies. It is based on on- properties: semi-rigidity and anti-rigidity. Those con- tological meta-properties of the concepts that appear cepts that are essential to some, but not all, of their in the ontological hierarchy. These meta-properties – instances are semi-rigid, while those that are not es- namely, rigidity, unity, identity, and dependence – are sential to any of their instances are anti-rigid. We do both highly general and based on philosophical no- not focus on this distinction in our work although Ru- tions. Although OntoClean uses meta-properties to dify can be used to evaluate these meta-properties as evaluate ontological taxonomies, it is not intended to provide a way of determining the meta-propertiesthemselves. Instead it shows the logical consequences of the users modelling choices, most notably on-tological errors that may result in taxonomies af- We are currently using Rudify to develop a central on- ter modelling choices have been made (Guarino and tology, to separate type- and role-hierarchies in On- toWordNet (Gangemi et al., 2003), and also to help automatically assigning meta-properties to concepts the end user keep type- and role-hierarchies in the do- based on how the concepts are expressed in natural main ontology separate. This section provides a dis- cussion of the relation between rigidity and type–role Of the four types of ontological meta-properties used by OntoClean, we focus on rigidity. There are Types and roles are the two main subdivisions of several reasons for this choice. First – and most im- sortal concepts. A sortal concept is a concept that portant in the context of the Kyoto project, the no- describes what sort of thing an entity is.
tion of rigidity plays a large role in the distinction be- ample “cat,” “hurricane,” and “milk” are sortal con- cepts while “red,” “heavy,” and “singing” are not. In to the maintenance stop of the originally implemented an ontology, sortal concepts are those concepts that one by Google, and a more flexible input facility was carry the meta-property identity (for a discussion of needed instead of the purely OWL based one.
identity, see (Guarino and Welty, 2004)). Further- In the following technical description and discus- more, sortals usually correspond to nouns in natural sion of Rudify we focus on the meta-property of rigid- language. We work on the assumption that the con- ity as this has been the most important property in the cepts represented in the noun hierarchy of WordNet context of the Kyoto project so far.
(Fellbaum, 1998, see also section 6) are sortal terms, The first step in the Rudify process is the identi- since this is generally the case. Types are rigid sor- fication of adequate LRs for the concepts that are to tals, while non-rigid sortals are generally roles. Fur- be tagged. Due to polysemeous word forms there is thermore, roles cannot subsume types.
no one to one mapping between concepts and LRs.
In order to see that roles should not subsume Also, the actual number of recorded senses for a given types, we can consider the following (erroneous) hi- LR may vary across lexical databases and across ver- sions of a specific lexical database. The results re-ported here are based on the English WordNet (Fell- baum, 1998) version 3.0. A further complication are concepts that do not have LRs at all. Typically, this applies mostly for concepts of the top levels of ontolo-gies, although there are some (rare) examples like themissing English antonym for “thirsty” meaning “not According to this hierarchy, if Fluffy ceases to be a thirsty” which constitutes a lexical gap.
pet, then Fluffy also ceases to be a cat, which is im- A set of linguistic patterns that represent positive or negative evidence for a single meta-property needs From this last point in conjunction with the above to be developed. Each pattern specifies a fixed se- assumption that nouns usually represent sortals, it fol- quence of word forms. For little inflecting languages lows from the OntoClean principles that amongst sor- like English with relatively fixed word order this ap- tal terms, non-rigid sortals should not subsume rigid proach works reasonably well. Further refinement of sortals. In other words, non-rigid nouns generally the patterns will be needed for languages with more should not subsume rigid nouns. There are excep- free word ordering. For rigidity, we found only pat- tions to this rule. However, this general conclusion terns representing evidence against rigidity. Thus, the allows us to evaluate concepts only for rigidity and default assumption when tagging for rigidity is that non-rigidity, which in turns saves us the computation- rigidity applies. A concept C is considered non-rigid ally expensive task of evaluating non-rigid terms as only if enough evidence against rigidity has been col- either semi- or anti-rigid over large sets of concepts.
lected for C. Obviously, sparse data for occurrencesthe LR for C will distort the results and produce askew in the direction of rigidity.
For rigidity, a typical pattern reads “would make a good X” where X is a slot for a concept’s LR. This The general idea behind Rudify is the assumption that may be a single token, a multiword or even a com- a preferred set of linguistic expressions is used when plex syntactic phrase (as is frequently the case in Ro- talking about ontological meta-properties. Thus, one mance languages). Over-generation of patterns is pre- can deduce a concept’s meta-properties from the us- vented by enumerating and excluding extended pat- age of the concept’s lexical representation (LR) in terns. The non-rigid pattern “is no longer (—/a/an) X” natural language. This idea has been developed and over-generates phrases like “there is no longer a cat programmatically exploited first in the AEON (Au- (in the yard/that could catch mice/. . . )” from which tomatic Evaluation of ONtologies) project (V¨olker we cannot deduce non-rigidity for “cat.” et al., 2008). AEON was developed for the auto- Another frequent over-generation is found for LRs matic tagging of existing ontologies in terms of Onto- that occur as part of a more complex compound noun Clean meta-properties. The Kyoto project decided to as in “is no longer an animal shelter” where animal rewrite the software based on the principles published is not an instance of the concept “animal.” As the re- by (V¨olker et al., 2005) for several reasons: there sults returned from web search engine are often mere was no active development of the tool any more and fragments of sentences such instances can only be ex- the software was released as a development snapshot cluded based on part-of-speech tagging and not based only, the web service interface had to be changed due From a linguist’s point of view, the first three of theseproblems are discussed in more detail by (Kilgarriff, Rudify currently uses 25 different patterns as evi- dence against rigidity. The results of the web search Rudify now is a highly configurable modular tool queries based on these patterns form a feature vector with parameter sets developed for English and Dutch.
for each LR that is then used for classification, i. e.
Work is under way for the development of parame- the mapping from the feature vector to the appropri- ter sets for the remaining European languages of the ate rigidity tag. Technically this is a ternary decision Kyoto consortium (Italian, Spanish, Basque). The between rigid, non-rigid and uncertain.
software is written in Python and NLTK (Bird et al., All classifiers were trained on a hand crafted and 2009) is used as the linguistic backend. Classifier cre- hand tagged list of 100 prototypical LRs of which 50 ation, training and application is done using Weka 3 denote rigid concepts and 50 denote non-rigid con- (Witten and Frank, 2005), but can be easily delegated cepts. They cover a broad range of domains and are to any software suite capable of manipulating ARFF recorded as monosemeous (having a single sense) in files. Rudify will be released as free and open source Four different algorithms have been used for clas- • decision tree (J48, an implementation ov C4.5) • multinomial logistic regression (Rosch et al., 1976) empirically showed the presence • nearest neighbor with generalization (NNge) of basic level concepts (BLC) in human cognition. In a conceptual taxonomy, for each concept C its subor- locally weighted learning, instance based dinate concepts Cn are typically more specific than C.
In evaluating the output we considered the results of The increase in specificity is due to at least one added all four classifiers and ranked the results according the feature for Cn that is compatible with C but allows for degree of consensus amongst them (see section 6 for discrimination between all Cn. BLCs mark the border between the most general concepts comprising only Both Rudify and AEON rely on the World Wide few features and the most feature rich concepts.
Web as indexed by Google as the hugest repository of Base concepts (BC) are described by (Izquierdo utterances that is accessible to the research commu- et al., 2007) as those concepts within a semantically nity. This is done in order to minimize sparse data structured lexical data base that “play the most im- effects. We are aware of the theoretical implications that data extracted from Google or other commercial vague notion is effectively a rephrase of the BLC.
web search engines entails. The most crucial prob- BCs, though, are conceived as a purely computation- ally derived set based on semantic relations encoded in hierarchical lexical databases. BCs are those con- Results are unstable over time. The indexing pro- cepts that are returned by the following algorithm: cess is rerun regularly and results retrieved at any for each path p from a leaf node (a node with no given point in time may not be exactly repro- hyponym relation to other nodes) up to a root node (a node with no hypernym relation to other nodes) • The query syntax may be unstable over time and choose the first node C with a local maximum of implements boolean searches rather than linguis- specific relations to other nodes as a BC. This al- gorithm can be adapted by defining the set of spe- cific relations (e. g. only hyponymy, all encoded re- There are arbitrary limitations of the maximum lations including lexical relations) and by defining a number of results returned and of the meta-data minimally required number of subsumed concepts a possible BC must contain. BC sets depend from the specified parameters and the hierarchical structure of • The data repository is in principle uncontrolled as the lexical database. Thus, different sets are com- write access to the World Wide Web and other puted for different versions of WordNet and for other parts of the Internet is largely unrestricted. Com- national wordnets. Software and data for comput- mercial search engines work as additional filters ing BCs from WordNet are freely available online at on the raw data with their filter policy often left undocumented and subject to changes as well.
WordNet (Fellbaum, 1998) is an electronic lex- and also frequently used in figurative language (exam- in terms of semantic relations including synonymy (“car”–“automobile”), hyponymy (the relation among • “Mount Si High School teacher Kit McCormick general and specific concepts, like “animal” and “cat,” that results in hierarchical structures), and (generalization from a school mascot to a school meronymy (the part-whole relation, as between “cat” and “claw”). Linking words via such relations resultsin a huge semantic network.
• “Also the 400 CORBON is no longer a wildcat.” We have added a set of BCs to the middle level of the Kyoto ontology thereby providing the ontol- • “He nearly gave in and became a Wildcat before ogy with a generic set of concepts that can be used finally deciding to honor his original commitment for inter-wordnet mappings and wordnet to ontology Rudify was evaluated on the set of BCs derived from WordNet 3.0 considering only hypernym rela- • “For example, the dog is no longer a wolf, and is tions and with a minimum of 50 subsumed concepts for each BC (BC-50). These parameters result in a (example discusses changing relations between set of 297 concepts. Inspecting the BC-50 set we found LRs that are highly unlikely BLCs though they • “For four years, the space agency had been plan- fulfill the formal criteria for BCs. A striking exam- ning, defining, or defending some facet of what ple is “moth.” In WordNet, much effort was spent to record a high number of different insects as dis- tinguished concepts thus effectively shifting the basiclevel downwards in the taxonomic tree. (Tanaka and • “Others figuring prominently in the county’s his- Taylor, 1991) report on a similar effect of basic level tory were Edward Warren, who established a trad- shifts for BLCs that can be shown for experts in their ing post near what is now Apollo [. . . ]” • “The patron of the city is now Apollo, god of light, We tested Rudify on four different English language • 50 region terms (handcrafted by environmental We classify the Rudify output on the BC-50 set ac- cording to the agreement amongst the four classifiersused. We refer to those cases in which all four clas- • 236 Latin species names (selected by environmen- sifiers reached agreement as decisive. Rudify yielded decisive output for 215 BCs. Whenever there is dis- • 201 common species names (selected by environ- agreement amongst the classifiers, we refer to this output as difficult. There are 82 difficult cases thatsubdivide into two further cases. When three out of • 297 basic level concepts (BLC-50) four classifiers reached agreement, we refer to thisoutput as indecisive. Rudify yielded indecisive out- put for 56 BCs. When two classifiers evaluate a termas rigid and two as non-rigid, we refer to this as un- Classifiers correctly classified all region terms and all decided. Rudify is undecided with respect to 26 BCs.
Latin species names as rigid concepts. This holds also These figures are summarized in table 1.
for the common English species names with three ex- An evaluation of Rudify output for the 215 deci- ceptions: “wildcat” was misclassified as denoting a sive cases indicates that Rudify produces a high level non-rigid concept by all four classifiers and “wolf” of accuracy for decisive cases (see table 2). 85 % of and “apollo” (a butterfly) were mis-classified by all the terms evaluated as rigid were correctly evaluated, classifiers except NNge. This mis-classification is due and 75 % of the terms evaluated as non-rigid are cor- to the fact that those LRs are not monosemeously de- rectly evaluated. Many of the Rudify errors either noting a single concept (a species) but are polysemous came from high level concepts, e. g., “artifact” and Table 1: General overview of the classification on the BC- Table 2: Overview of the decisively classified BC-50 con- “unit of measurement,” which are ordinarily dealt with manually, or else they dealt with polysemouswords, which was an anticipated difficulty (see sec- In 3 % of the decisive output we used Rudify to de- termine whether a concept is rigid or non-rigid, e. g.
for “furniture.” Since not every concept is ontologi- cally clear cut, and since some concepts lie within ar-eas of ontology in which the alternative theories havenot yet been properly worked out (e. g., the ontol- ond uses Rudify in conjunction with OntoClean prin- ogy of artefacts), we have determined that Rudify can be occasionally helpful in making modelling choicesbased on the common sense uses of the concepts in language. For these cases the evaluation remains un-clear.
We consider BCs that can reasonably be considered For 56 concepts Rudify yielded indecisive output.
amouts of matter. Amounts of matter are generally re- Exactly 50 % of these cases are incorrect (28 out of ferred to by mass nouns; ‘milk,’ ‘mud,’ and ‘beer’ are 56). For this reason we do not regard the indecisive a few examples. Once again we begin by provision- ally modelling the concepts taken from WordNet as The decisive Rudify output on the BCs within the the upper level concept “amount-of-matter” into the OWN hierarchy yields five OntoClean errors, if we following hierarchy, which includes rigidity assign- count the hypernyms, and 22 errors if we count in- ments from Rudify. R+ indicates a rigid concept, R− stances of hypernym relations. This is based only on the Rudify output prior to evaluating the correctnessof this output, but it gives us an idea of the OntoClean results if we uncritically use Rudify to evaluate con- cepts in the ontology (for more details, see (Herold et al., 2009b)). In short, Rudify output coupled with the OntoClean methodology provides a useful tool for drawing attention to problems in the backbone hierar- In summary, our evaluation of Rudify output on Using the Rudify data, we can clean up this hierar- BCs is that Rudify is successful with respect to the chy. First we notice that Rudify has evaluated “nutri- decisive output. It produces decisive output with a rel- ment” as non-rigid. This indicates that it is probably atively high degree of accuracy (83 %) and an overall a role rather than a type. In order to verify this, we re- accuracy on the BC-50 set of 69 % (table 3). Further- fer to the definition taken from WordNet: “a source of more, Rudify has also proven useful in deciding how materials to nourish the body.” That is, the milk in my refrigerator is a nutriment only if it nourishes a body.
If you bathe in milk, like Cleopatra, it is a cosmetic.
“Nutriment,” therefore, is a role that milk can play, so it does not belong in the type hierarchy. We therefore,move it to the role hierarchy as subclass of “amount of In this section we illustrate with two examples how matter role.” We pause to notice that in this case, the Rudify results can be used to inform ontology design.
decision was made using Rudify results and human The first example uses Rudify independently, the sec- verification of the output. This case does not invoke OntoClean, i. e., there would be no OntoClean errors 2008) for human inter-annotater agreement. For spe- if “nutriment” were subsumed by “amount of matter.” cialized domain terms, agreement was substantially This contrasts with the second example, which yields higher: only 3 out of 201 English species terms had a formal error within the hierarchy itself.
The evaluation of the results reported here shows potential for further improvement. Word sense disam-biguation will increase the accuracy for polysemeouswords. First experiments involving hypernyms of LRs Notice that Rudify evaluates “drug” as non-rigid, and in the retrieval of evidence for or against ontological “antibiotic” as rigid. However, the current hierarchy meta-properties give already promising results.
subsumes the rigid concept under the non-rigid con- For future reference and stability of the results it cept. This results in a formal error in the hierarchy.
will be beneficial to use a controlled linguistic corpus Because “drug” and “antibiotic” are both sortal terms, of appropriate size instead of commercial web search this means a role subsumes a type, which, as we have seen above leads to inconsistency. Consider the an-tibiotic penicillin. Penicillin is only a drug if it is ad-ministered to a patient, but it is always an antibioticdue to its molecular structure. By subsuming “an- tibiotic” under “drug,” the ontology erroneously statesthat if some amount of penicillin is not administered The development of Rudify and its application to the to a patient, then it is not an antibiotic. The solution Kyoto core ontology has been carried out in the EU’s then, is to move “drug” out of the type hierarchy and 7th framework project Knowledge Yielding Ontolo- into the role hierarchy. “Drug” then becomes a “sub- gies for Transition-based Organizations (Kyoto, grant stance role,” and an antibiotic is subclass of “amount of matter” that can play the role “drug.” The authors would like to thank Christiane Fell- Because “chemical compound” and “oil” are both baum for many fruitful discussions and the Kyoto evaluated as rigid we do not need make any changes members for their kind collaboration.
The result is the following hierarchy fragments under “amount of matter” and “amount of matter Bird, S., Klein, E., and Loper, E. (2009). Natural Language Processing with Python. O’Reilly.
Fellbaum, C., editor (1998). WordNet: An Electronic Lexical Database. The MIT Press.
Gangemi, A., Guarino, N., Masolo, C., and Oltramari, A.
(2003). Sweetening wordnet with dolce.
Guarino, N. and Welty, C. (2002). Evaluating ontological decisions with ontoclean. Communications of theACM, 45(2):61–65.
Guarino, N. and Welty, C. (2004). An overview of ontoclean. In Staab, S. and Studer, R., editors,Handbook on Ontologies, pages 151–172.
We presented Rudify – a system for automatically de- Herold, A., Hicks, A., and Rigau, G. (2009a). Central ontology version 1. Kyoto project deliverable D6.2.
riving ontological meta-properties from large collec-tions of text based on the lexical representation of in- Herold, A., Hicks, A., Segers, R., Vossen, P., G. Rigau, G., dividual concepts in natural language. This approach Agirre, E., Laparra, E., Monachini, M., Toral, A., andSoria, C. (2009b). Wordnets mapped to central yields valueable results for use in consistency check- ontology version 1. Kyoto project deliverable D6.3.
ing of general large scale ontologies such as the Kyotocore ontology. On the basis of 297 basic concepts Izquierdo, R., Su´arez, A., and Rigau, G. (2007). Exploring the automatic selection of basic level concepts. In derived from the English WordNet 69 % agreement Proceedings of the International Conference on with human judgement could be demonstrated. This Recent Advances on Natural Language Processing closely matches the figures reported by (V¨olker et al., (RANLP’07), Borovetz, Bulgaria.
Kilgarriff, A. (2007). Googleology is bad science.
Computational Linguistics, 33:147–151.
Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., and Boyes-Braem, P. (1976). Basic objects in naturalcategories. Cognitve Psychology, 8:382–439.
Tanaka, J. W. and Taylor, M. (1991). Object categories and expertise: Is the basic level in the eye of thebeholder? Cognitve Psychology, 23:457–482.
V¨olker, J., Vrandecic, D., and Sure, Y. (2005). Automatic evaluation of ontologies (AEON). In Proceedings ofthe 4th International Semantic Web Conference(ISWC2005), volume 3729 of LNCS, pages 716–731,Berlin/Heidelberg. Springer.
V¨olker, J., Vrandecic, D., Sure, Y., and Hotho, A. (2008).
AEON — an approach to the automatic evaluation ofontologies. Applied Ontology, 3(1-2):41–62.
Vossen, P., Agirre, E., Calzolari, N., Fellbaum, C., Hsieh, S., Huang, C., Isahara, H., Kanzaki, K., Marchetti,A., Monachini, M., Neri, F., Raffaelli, R., Rigau, G.,and Tescon, M. (2008). Kyoto: A system for mining,structuring and distributing knowledge acrosslanguages and cultures. In Proceedings of LREC2008, Marrakech, Morocco.
Witten, I. H. and Frank, E. (2005). Data Mining: Practical machine learning tools and techniques. MorganKaufmann, San Francisco, 2nd edition.

Source: http://kyoto-project.eu/www2.let.vu.nl/twiki/pub/Kyoto/RudifyApplication/KEOD_2009_18_CR.pdf