Comp.lancs.ac.uk

An ecologically valid study of categorisation by designers
Thomas C. Ormerod, Ryan RummerLancaster University Abstract
This paper reports the use of ethnographic and experimental methodologies to investigatethe conceptual representation of design expertise. An ethnographic study was firstconducted in a commercial software development group. This study yielded a wide rangeof data, from which thirty design episodes were selected. Members of the design team thenundertook a card sort task, in which they sorted the episodes into natural categories.
Differences in the sorts produced by designers reflect their specialist roles within the designteam. The same design information is evaluated under different metrics across roles. Thiscorroborates previous studies that show how conceptual design representations differ inindividual and team contexts. The results also show how combining the outcomes ofethnography with a traditional cognitive method enable us to verify objectively theobservations made in ethnographic studies.
Introduction
The use of ethnography in design research has proliferated in recent years (e.g.,Bucciarelli, 1988). Often ethnographies are carried out to inform the development of designtools and methods. While ethnographic methods provide rich and naturalistic data, theyface the criticism that observations are open to subjectivity. We have argued (Ball &Ormerod, in press) that where the goals of an ethnography are purposive, then there is arequirement for objectivity. One approach to enhancing objectivity is to use experimentalmethods of cognitive psychology. However, examples of the successful use ofexperimentation to explore design are few. This is hardly surprising given the complexityof the domain, and the consequent difficulty and validity of imposing experimental controlsupon expert behaviours in pursuit of objective validation of hypotheses. In this paper, weargue that ethnography can play a role in enhancing the ecological validity ofexperimentation in design research. We report a study in which the ecological validity of anexperimental method is enhanced by the provision of realistic materials from anethnographic analysis.
A basic tenet of the ethnographic approach is the adoption of a non-interventionist research tradition. Most cognitive psychology, on the other hand, is of an interventionistnature, gaining its strength from the systematic manipulation and control of variables thatmight affect performance. Although ethnographic methods provide rich and naturalisticdata, observations are open to subjective sampling and interpretation. Conversely, attemptsto use experimental methods to explore design immediately confront problems of ecologicalvalidity. Notwithstanding these problems, we argue that both approaches have a role toplay in design research. The role of controlled experimentation is demonstrated in usabilitytesting of products and prototypes where objectivity in observation and interpretation isessential. Similar requirements for objective assurance face the researcher in developingtheories of design expertise, accounts of optimal design practice, or tools and methods forsupporting design. We suggest that ethnographic and experimental methods can be used totriangulate observations across methodologies.
In the remainder of this paper, we report an empirical study which combines ethnography with the use of a sort method for investigating the categories designers use toclassify information. Participants are given sets of cards that contain domain descriptionsand are required to sort these into categories significant to them. The method originates inresearch into memory and the mental representation of concepts. The rationale is that, byexamining the nature of sorts, one can infer participants’ mental representations ofconceptual knowledge. In applied research, the method is used to study expert/novicedifferences (e.g., Davies, Gilmore and Green, 1995). It has been suggested (e.g., Burton,Shadbolt, Rugg, & Hedgecock, 1990) that the sort method provides an informative butcost-effective method for eliciting expertise in the development of expert system.
However, the sort method is not without its problems. In early psychological research intoconceptual categories, there was an assumption that what was being elicited through a sortmethod reflected relatively static conceptual representations. However, this view has been challenged by Barsalou’s (1985) description of goal-directed categories. In this view,categories are not fixed, but are determined by the task faced by the individual at any onetime. If this view is correct, then the assumption that a sort method elicits staticrepresentations of concepts is unfounded.
A particular problem facing the use of the sort method concerns the nature of the items to be sorted. In traditional sort tasks, the items consist of object descriptions, often varyingin familiarity, representativeness, or prototypicality. In applied research, they may containdescriptions of domain objects or terminology (e.g., programs - Davies et al, 1995;archaeological shards - Burton et al, 1985). However, it is in the selection of sort items thatthe method can lose ecological validity. If concepts are goal-directed, then the sorts thatexperts produce are likely to be a function of the items they are given. One way around thisproblem is to use as sort materials items produced as part of expert performance. To date, we are unaware of any studies employing the sort method that have used items produced bythe study participants themselves. This is not surprising given the difficulty of collectingsuch items. We suggest that ethnography can be used to elicit design episodes that provideecologically valid materials for a study employing the sort method to explore designexpertise.
The aim of the study reported here was to investigate whether natural categories can be identified that are used by expert designers to classify design information. The study ispart of a wider project to develop a computer based indexing system for supporting designreuse. Thus a further aim was to identify natural categories to provide a notation forencoding design information within the system. The study took place at a majormanufacturer of personal computers (PCs), and involved two phases, ethnographic datacollection and the experimental study.
Phase 1: The ethnographic study
In the first phase, our researcher spent two weeks conducting an applied ethnographic
study of design practices within the company’s software development team. A full report
of the first phase is available elsewhere (Ball & Ormerod, in press).The general remit of the
team was to develop software networking solution. The team had twelve full-time
members. As well as the team leader, two members were principally involved in
managerial and high-level planning or commercial liaison activities. Four members were
specialist designers, each with their own area of expertise (e.g., local area networks,
hardware systems). The remaining members were programmers. The team had an
hierarchical management structure, in which one or two programmers and one specialist
designer worked under the remit of one of the three senior team managers. At the beginning
of each week, the whole team had a two hour meeting in which current projects were
reviewed and team goals for the week were shared. Otherwise, team members worked
mixture of single and shared office spaces, with numerous informal conversations and
meetings taking place but otherwise few formal exchanges.
The researcher followed a standard ethnographic approach, collecting a mixture of field notes, taped interviews and meetings, texts and other company documentation, whilstnoting personal observations and commentary. At the end of this period, we analysed theresulting data with a view to establishing a set of design materials from the data that couldform the basis of a sort study. Previous work has suggested that the 'design episode'forms a useful unit for describing design activity. As a metric for determining the scale of adesign episode, we chose the Questions, Options and Criteria approach of (MacLean,Young, Bellotti, & Moran, 1991). The identification of episodes was made on the basis ofjudgements by two researchers as to the extent to which a segment of data reflected thepursuit of a single design question. Thirty episodes were chosen to reflect the types ofdesign information that were evident in the ethnographic data.
Phase 2: The experimental study
The aim of the experimental study was to investigate the natural categories of design
episode information produced by the designers who had been the focus of our observations
in the ethnographic phase.
MethodNine members of the design team took part in this phase (three managers, three specialistdesigners and three programmers). Materials consisted of a thirty episodes, each printedon an A5 size card (examples as in Table 1). Participants carried out the sorts individually.
Each participant reviewed a instructions that included an example sort. They were thengiven30 cards, each displaying one episode. Participants were asked to sort the episodesinto natural categories segregated by a dimension of their own choosing, and that captureda view of the episodes that was deemed important or relevant to their own design activities.
They were told to use as many or as few categories as they found necessary to representeach grouping. Each participant completed two sorts, and the dimensions used todifferentiate categories and the cards assigned to each category were recorded.
Example episodes
(audiotaped conversation): “Actually, ‘Jim’ said to me yesterday, or Friday last weekmaybe, that ‘John’ wanted to talk about a way of formalising the process from Lotus Notesto NBP. You know, how do we link these and have an actual cut off that says.here’s thepoint where we stop talking about this stuff and say that’s the decision. Now put it to aWord Pro doc and up to the library, and make that a design point.” (email): TARDIS Web Page & Logic DiagramAs the Web page idea has crystallised and grown!, so has the need to firm up the design. Ithink the logic diagram I put on the [.] Web site [.address.] will describe where we areheaded. You will see how the Web page will navigate the user to the right download site .
It will also give him the right code that will automatically route his support request to thecorrect Helpcenter. The response has been no less than enthusiastic from all and we want tokeep up the momentum.
(field notes): 'Bill' describes to 'Mike' during an informal morning meeting in Mike'soffice how The GUI and ‘engine’ of TARDIS v0.1 are distinct but closely coupled -always running on the same system. TARDIS v0.2 removes this restriction by introducinga new GUI in the form of a combination of HTML web pages and Java applets to allow[. deleted.]. Bill is seeking clarification of whether he should continue to work onthe new GUI. 'Mike' queries whether he is the right person to give this clarification.
Data analysis focuses upon the category assignments for each sort and category labelsprovided by designers. The average number of categories produced in each sort was 5.8(standard deviation = 2.3). Much of the variability comes from a single designer, whoproduced two sorts of 10 and 11 categories. Example category labels produced are shownin Table 2. From these, dimensions that govern distinctions between category labels wereinferred, mainly from concurrent verbal reports of the designers.
The data were subjected to multi-dimensional scaling (MDS) and hierarchical cluster analysis methods using Statistica™ using two data matrices. The first was a similaritymatrix scoring the number of designers making each possible episode pairing. The secondwas a dissimilarity matrix in which pairing of episodes was shown for each participant.
The first matrix allowed for analyses of episode clustering across dimensions, whilst thesecond matrix allowed for the analysis of participant groupings and outliers. Figure 1shows a mapping produced by an MDS run from the similarity matrix. Note that thedimensions are those determined by MDS, not those inferred from the category labels.
However, these dimensions fit reasonably closely to the dimensions of Product (with ascale from 'general' to 'specific') and Role (with scale from 'managerial' to'implementor'). The boundaries drawn around episode groups were identified as a result ofconducting an hierarchical cluster analysis on the matrix data (using a complete linkagealgorithm). Similar mappings were produced for the dissimilarity matrix data. However,space limitations preclude their inclusion.
Dimensions of category labels produced by designers
(e.g., "technical group; systems; bureaucratic; testgroup; methods; status meeting") (e.g., "process; marketing; function; logistics;program development") (e.g., "obvious procedures; press release;discussion of specification; general discussion;specification details; general narratives; long-windedinfo of unknown quality; status notes") (e.g., "TARDIS v0.1; POLAXv0.2; not product-specific") (e.g., "trivial items; info relevant only toparticipants; sensible suggestions; important workitems; hi-level design descriptions; detailed designinfo that does not affect designers") (e.g., "Things already done; defined optionsrequiring decision; early design discussions;oddballs") Figure 1. Two dimensional scaling solution for the episode similarity
matrix ('e' refers to episode).
Discussion
It is notable that, of the 17 sorts generated (one designer was only prepared to undertake asingle sort), 15 are captured by just four dimensions. What is interesting is how some ofthese dimensions, and more specifically the categories generated within them, corroboratethe observations made by Bucciarelli (1988). Notably, ‘relevant to specific group/role’corresponds with his notion of object worlds, and ‘design process/stage’ corresponds withhis notions of specification and constraint. Thus, the sort study is able to verify theobservations that emerge from this ethnographic study. In the limited space available, it isinappropriate to comment upon the clustering of cards in each category across designersand consistency of designer’s sorts beyond a few broad observations. Designers werehighly consistent in their category assignments under the dimensions of ‘relevant tospecific group/role’ and ‘product’, but less so under dimensions such as ‘episode contenttype’ and ‘design process/stage’. The role played by each designer in the softwaredevelopment team had a strong influence on their categorisation. For instance,programmers produced ‘critical’ evaluative categories (e.g., “trivial information”; “excellentprogram structure”), while managers produced open evaluative categories (e.g., “detaileddecision-making”; “external influences”; “common building blocks”). Differences in sortperformance between the various specialist roles within the design team, and theimplication that the same design information is evaluated under different metrics acrossroles, corroborate our observations from the ethnographic study that design knowledgerepresentations differ in individual and team contexts.
Acknowledgements
This research was supported by an ESRC Cognitive Engineering Programme grant (No:L127251027). We are grateful to our grant co-holder (John Mariani) and to Gary Spiers fortheir contributions and the company and design team members for their co-operation. Asynopsis of the study appears as part of Ball and Ormerod (in press).
References
Ball, L.J., & Ormerod, T.C. (in press) 'Applying ethnography in the analysis and support of expertise in engineering design'. Design Studies. Barsalou, L. W. (1985) 'Ideas, central tendency, and frequency of instantiation as determinants of graded structure in categories'. Journal of Experimental Psychology:LM&C, 11, pp.629-654.
Bucciarelli, L. L. (1988). 'An ethnographic perspective on engineering design'. Design Studies, 9, pp. 159-168.
Burton, A. M., Shadbolt, N. R., Rugg, G., & Hedgecock, A. P. (1990) 'The efficacy of knowledge acquisition techniques: A comparison across domains and levels ofexpertise'. Knowledge Acquisition, 2, pp167-178.
Davies, S. P., Gilmore, D. J., & Green, T. R. G. (1995) 'Are objects that important? Effects of expertise and familiarity on classification of object-oriented code'. Human-Computer Interaction, 10, pp. 227-248.
MacLean, A., Young, R. M., Bellotti, V. M. E., & Moran, T. P. (1991) 'Questions, options and criteria: Elements of design space analysis'. Human Computer Interaction,6, pp. 201-250.

Source: http://www.comp.lancs.ac.uk/~jam/desperado.d/pdf.d/orb.pdf