Nonreactive 1

NONREACTIVE MEASURES

IN STUDENT AFFAIRS RESEARCH

Rose M. Abler and William E. Sedlacek

Research Report #5-86

Nonreactive 2

COUNSELING CENTER

UNIVERSITY OF MARYLAND

COLLEGE PARK, MARYLAND

NONREACTIVE MEASURES IN STUDENT AFFAIRS RESEARCH

Rose M. Abler and William E. Sedlacek

Research Report #5‑86

Summary

Relying exclusively on questionnaire and interview data can cause the student affairs researcher to overlook nonreactive methodology: unobtrusive methods of collecting data in which participant reaction to the process of data collection does not interfere with the response. Such methods of data collection have been used in other fields, and the student affairs professional can improve research techniques and implement a multi‑method assessment strategy by incorporating nonreactive methods into current research programs. Examples in which nonreactive methodology has been used in student affairs research are discussed'; and practical steps are presented to help in beginning a program of nonreactive research.

Nonreactive 3

Nonreactive Measures in Student Affairs Research

Many student affairs professionals encounter difficulties in conducting research, such as lack of time, funding, or confidence, which can lead them to avoid research efforts altogether (Brown. 1986; Hood, Karr, Riggers & Elliott, 19$6). However, "if we are to survive through the next century. . . it is imperative that we support research and evaluation that examines our services and programs and what impact they have on students" (Brown, p. 195). Methodology can pose particular problems. Those beginning a research effort may find that the first data collection methods which come to mind involve questionnaires or interviews. Certain inherent difficulties with these methods are obvious, such as the time consuming nature of constructing and scoring questionnaires and/or conducting interviews, or the expense involved in printing costs and supplies.

Another problem with interview and questionnaire methods that is often overlooked has been noted by Webb, Campbell, Schwartz, Sechrest, and Grove (1981). They describe such methods as "reactive" because when these methods are used, the participant's reaction to the process influences his or her responses. This is a serious problem because when people know that they're being observed or that their responses are being counted it some way, they do not always respond accurately or honestly. An example relevant

Nonreactive 4

to student affairs work might be one in which researchers are interested in examining the degree of racial discrimination present in a college sports program. To answer their questions, they interview coaches and athletes about the alleged problem and administer various racial attitude questionnaires. However, given the demand characteristics of the situation (coaches ‑ the desire for their programs to appear in a positive light; athletes ‑ the pressure not to jeopardize their athletic careers.), the racial problems are minimized, and, receiving these results, the administration continues current policies when more accurate information would have suggested charges.

Nonrereactive Methodology

In response to such a problem, student affairs professionals can employ nonreactive methodology: unobtrusive methods of collecting data that do not interfere with the response itself (Webb et al., 1981 ) . In the previous example of racial discrimination in a college athletic program, one simple nonreactive approach might be to use team records try review the racial make‑up of past teams. Do any patterns emerge with respect to what positions are played by minorities? Have minority players ever consistently assumed leadership roles such as quarterback? (See McGehee & Paul, 1984). This method involves minimal effort and is nonreactive.

Nonreactive 5

At this point, the reader may be struck by the obvious nature of the nonreactive method just described. It is not a difficult approach; most people collect information nonreactively every day, but are simply not aware that they are accumulating potentially useful information.

Examples in Student Affairs

In the past, student affairs professionals have occasionally employed nonreactive methodology, resulting in innovative research resigns. To assess the success of an assertiveness training program, McFall and Marston (1970) posed as magazine salespersons and telephoned former program participants in order to unobtrusively analyze their assertiveness. In a nonreactive approach to questions about police bias in monitoring traffic violations? Heussenstamm (1971) had student research associates with unblemished driving records put Black Panther bumper stickers on their cars. Results indicated that this group received more traffic tickets than did a control group, supporting the hypothesis that the police discriminated against this group when issuing traffic citations. In order to identify a noncognitive predictor of student Success (other than, for example, SAT scores) Sedlacek, Bailey and Stovall (1984) analyzed types of errors made by prospective students in completing freshman orientation applications. Those who followed directions were

Nonreactive 6

more likely to stay in school. In a symposium presented at the University of Maryland, several additional ideas were presented and discussed (Abler, Bandalos Boyer Sedlacek, Sergent, Thomas & Thompson, 1986. Observers went to target locations in the student union to count the number of students present during random time slots. Frequencies of handouts taken from a commuter affairs office were used to determine program needs. Conference presentations were evaluated by observing audience seating patterns and participation rates. In a discussion of implications for future research on counseling center retention programs, Weiss and Gidden (1986) report on recent nonreactive studies which use archival data to assess the relationship between counseling center programs and student attrition.

Classifying Nonreactives Studies

Given the variety of nonreactive research being conducted, attempts have been made to create a classification system of nonreactive methodology. Webb, Campbell, Schwartz and Sechrest (1966) used the categories of physical traces (such as the errors left on orientation applications used by Sedlacek, Bailey and Stovall, 1984), archives (such as the team records used by McGehee and Paul, 1984) and observation (such as the student union and audience seating observations described above) to describe the various types of nonreactive data. Sechrest and Phillips (1979) note that this system was

Nonreactive 7

"ad hoc and soley for the sake of convenience in writing ...not carefully thought out as a conceptual framework" (p. 9). In an alternative approach, they proposed a matrix as a step toward a taxonomy of nonreactive methodology. One axis of the matrix consists of a list of possible purposes of assessment (for example, to determine interest, category membership, or affective state); the other axis consists of nonreactive characteristics of the responses to be observed (such as frequency or magnitude). The first axis is used to categorize the research question. The second axis provides a variety of ways to operationalize the research question using nonreactive data. Table 1 illustrates an application of this matrix using a student affairs example.

Insert Table 1 about here

Advantages and Disadvantages

As with any research methodology, advantages and disadvantages exist for the student affairs researcher employing nonreactive techniques. Advantages, as previously discussed, include the low cost and less extensive efforts required and, most important, that participant reactions to the data collection procedures cannot bias their responses. Ethical considerations bring up a potential disadvantage in

Nonreactive 8

nonreactive research: is privacy and the right to informed consent violated when participants are not aware of the data collection procedures? Such questions can only be answered on an individual basis, the circumstances involved in each research enterprise being unique. Another potential disadvantage is that there is no guarantee that the data which the researcher has nonreactively collected actually represent the construct of interest. The solution to such a problem leads to the conclusion drawn by Webb et al. (1981): namely, that nonreactive methods are not meant to replace the traditional questionnaires and interviews but rather to supplement and cross‑validate them. In this way, a multi-method research program as advocated by Campbell and Fiske (1959) can be implemented.

Steps Doing Nonreactive Research

First, bring up the idea of nonreactive research at a staff meeting. Review your current data collection procedures and then brainstorm as to how to nonreactively collect the information you need. Second, when planning programs, ask yourself the question, "How can I tell how well I'm doing without asking the recipients of this service?" Whatever nonreactive procedures you implement can be cross-validated against the paper‑and‑pencil evaluation forms normally used. Third, use the "critical incident" technique (Flannagan, 1954) in your daily work and think nonreactively

Nonreactive 9

in order to develop ideas. In this approach, direct observations of human behavior (critical incidents) are used as a springboard for solving practical problems. The incident may be an outstandingly positive or negative example of the issue to be studied. For example, the Sedlacek et al. (1984) study of application errors and student success developed out of a critical incident: an obviously lost student wandered into a staff meeting (the critical incident) which began a staff discussion of how to better predict successful students and eventually led to the nonreactive study then conducted. Student affairs professionals who take steps such as those described above and employ nonreactive techniques may find this methodology a useful addition to their work.

Nonreactive 10

References

Abler, R., Bandalos, D., Boyer, S., Sedlacek, W., Sergent, M., Thomas. T. & Thompson C. (1986) . Nonreactive measures in student affairs research. Symposium conducted at the University of Maryland Student Affairs Conference, College Park, Maryland.

Brown, R. (1986). Research: A frill or an obligation? Journal of College Student Personnel, 27, 195.

Campbell, D. & Fiske, D. (1959). Convergent and discriminant validation by the multi‑trait, multi‑method matrix. Psychological Bulletin, 56, 81‑105.

Flannagan, J. (1954). The critical incident technique. Psychological Bulletin, 51, 327-358.

Heussenstamm, F. (1971). Bumperstickers and the cops. Transaction, 8, 32‑33.

Hood, A., Kerr, B., Biggers, D. & Elliott J. (1986). Paper presented at the American College Personnel Association Convention, hew Orleans, Louisiana.

McFall R. & Marston, A. (1970). An experimental investigation of behavior rehearsal in assertive

training. Journal of Abnormal Psychology, 76, 295‑303.

McGeehee R., & Paul, M. (1984). Racial make-up of central, stacking and oither playing positions in Southeastern Conference on Sport and Society, Clemson University, Clemson, South Carolina.

Sechrest, L. & Phillips, M. (1979). Unobtrusive measures, an overview. In L. Sechrest (Ed.), Unobtrusive measurement today (pp.1-17). San Francisco: Jossey Bass.

Sedlacek, W., Bailey, B. & Stovall, C. (1984). Following directions: An unobtrusive measure of student success. Journal of College Student Personell, 25, 556.

Webb, E., Campbell, D. Schwartz, R. & Sechrest, L. (1966). Unobtrusive measures: noreactive research in the social sciences. Boston: Houghton Mifflin.

Weiss, S. & Gidden, N. (1986). Applications of cost effectiveness to counseling center retention programs. Journal of College Student Personnel, 27, 260-268.

Noncognitive 12

Table 1:

Applying a Student Affairs Example to Sechrest & Phillips’ (1979)

Illustrative Matrix for a Generative Taxonomy of Nonreactive Measures

(EXAMPLE)

Research Question: Are University students interested in additional campus computer terminals?

Purposes of Assessment		NONREACTIVE CHARACTERISTICS OF RESPONSES TO BE OBSERVED
		Frequency of Response	Magnitude of Vigor of Response	Choice Response	Guilty Knowledge Response	Biased Response
	Interest or Involvement	# complaints received at chancellor's office	length of time the complaints & letters have been occurring	# students willing to increase their activity fee to pay for new facilities	whether students have studied the exact costs of extra terminals	whether students over-estimate the number of people who can't find available terminals to complete their work
		# letters to editor about problem	extremeness of language in letters
		# people waiting to use terminals on any given day
	Value
	Ability
	Affective State
	Category Membership