COUNSELING
CENTER
COLLEGE
PARK, MARYLAND
Issues
in Advancing Diversity Through Assessment*
William
E. Sedlacek
Research
Report #5-93
*
Portions of this article are from an invited address to the Association for Assessment
in Counseling at the American Counseling Association National Convention, March
15, 1993, Atlanta.
COUNSELING
CENTER
UNIVERSITY
OF MARYLAND AT COLLEGE PARK
COLLEGE
PARK, MARYLAND
Issues
in Advancing Diversity Through Assessment
William
E. Sedlacek
Research
Report 45-93
Summary
Five problems relating to diversity that exist in the
counseling and personnel professions are presented. A discussion of how those
concerned with assessment can address each problem is presented.
The first problem (The Quest for the Golden Label)
concerns operationally defining diversity through measurement rather than to
seek the perfect label. Measures such as the Situational Attitude Scale (SAS)
and Noncognitive Questionnaire (NCQ) are discussed as examples.
The "Three Musketeers" problem is a tendency to
feel that fairness means using a single measure for all. Nontraditional
populations may present their abilities and attributes in different ways,
necessitating different measures to achieve equal results.
The "Horizontal Research" problem deals with our
tendency to study nontraditional populations as afterthoughts in the design of
research studies. Hence, we do not achieve developmental or inductive research
programs for nontraditional groups.
The "Bias is Bias" problem involves a discussion
of sampling bias and how it can be an example of racism in how we interpret
results of studies.
The lack of professionals trained in assessment and an
understanding of cultural and racial variables is called the "I'm OK, You're Not," problem.
Overall suggestions for change are also made such as
more joint training programs and projects among professional groups.
This title suggests an important focus which is recent in
our profession; using assessment as a proactive tool to initiate change rather
than in a more passive, defensive way to insure that we aren't harming someone.
The Association for Assessment in Counseling (AAC) has
developed a new document on multicultural assessment standards (Prediger 1993)
which is a compilation from a number of sources. It is a detailed listing of
ethical and professional standards which I recommend for your study. I will not
attempt to go over the details of that publication but I will concentrate on
some problems relating to diversity that are compatible with that publication
that I feel exist in the counseling and personnel professions, and how those
concerned with assessment can promote diversity through their efforts.
While there are many problems that could be addressed, I
will concentrate on five here today. The first problem I call the "Quest
for the Golden Label". The idea here is that if we could just find the
right label, our problems would be solved. The Quest for the Golden Label Problem
There is often much confusion about how we define
diversity, what groups should be included and what terms we should apply to
those groups. Questions arise such as; Should we include gays and lesbians in
our conceptualization? Can we use Black and African-American
interchangeably? Is the whole issue just a matter of being politically correct?
1
While labels and terminology are important and contain
many symbolic and connotative elements, those concerned with assessment can
help cut through some of the ambiguity by helping to operationally define
terms. Differences or similarities among groups can be demonstrated through
measurement.
Westbrook and Sedlacek (1991) studied the labels used to
describe what they called "nontraditional students" in the Education
Index over forty years. Terms have varied from a focus on acculturation in
the 1950's to disadvantaged in the 1960's, to culture-specific
differences in the 1970's to multicultural in the 1980's. We could add
diversity as the probable term for the 1990's. While these terms may suggest
different approaches to the groups discussed, operationally we may still
be discussing the same people; those with cultural experiences different than
those of White middle class males of European descent; those with less power to
control their lives; and those who receive discrimination in the United States.
But does it make sense to include such variables as gender, sexual orientation,
or status as an athlete as aspects of cultural experience?
Here is where we can apply some principles of assessment.
As one example of how I have tried to address this question, I and students and
colleagues have developed measures in two areas and applied them to many
different groups. The first is the Situational Attitude Scale (SAS) which
measures prejudicial attitudes toward different groups including
racial/cultural groups ( Balenger, Hoffman, and Sedlacek, 1992; White and
2
Sedlacek, 1987), women (Minatoya and Sedlacek, 1983; older
people (Schwalb and Sedlacek, 1990), gays and lesbians (Washington, 1993)
people with physical disabilities (Stovall and Sedlacek, 1983), and athletes
(Engstrom and Sedlacek, 1991). If a group is the target of prejudicial
attitudes they meet my first criterion for inclusion as a nontraditional group.
The second measure that we have developed is the
Noncognitive Questionnaire (NCQ) which is designed to assess attributes that
are not typically measured by other instruments and that may be common ways for
persons with nontraditional experiences to show their abilities (Sedlacek,
1989). I will discuss the NCQ further under another problem. If the NCQ
systematically correlates more with the success of members of a given group
(e.g., Blacks) than it does for traditional White males, then the group meets
my second criterion for inclusion as a nontraditional group.
In summary, if a group receives prejudice and demonstrates
abilities in ways different from those with traditional experiences, they are
operationally defined as nontraditional. Even though groups as different as
athletes and older people may show their diversity in different ways, the
variables underlying their problems in dealing with their development, and in
coping with a traditional system that was not designed for them, may have some
similarities.
3
There are many who would disagree with my conclusions or
my definition of nontraditional. For example, Helms (1992) sees visible
racial/ethnic groups (VREGS) as having some distinct problems and
characteristics. Also Sue, Arredondo and McDavis (1992) express concern that
the term cross-cultural can be defined so broadly that it allows
counseling professionals to avoid dealing with what they see as the four major
minority groups in our society: African Americans, American Indians, Asian
Americans, and Hispanics and Latinos.
My concern is not that we conclude that one conception or another
is best but that we base our decisions on measurement principles and empirical
evidence wherever possible. It is quite possible that we can identify relevant
variance based on nontraditionality and more specific group memberships as
well.
The rallying cry of "all for one and one for
all" is one that we use often in developing what we think of as fair and
equitable measures. Our interpretation of how to handle diversity is to hone
and fine-tune our measures so they are equally valid for everyone.
However, if different groups have different experiences and different ways of
presenting their attributes and abilities, it is unlikely that we could develop
a single measure, test item etc. that could be equally valid for all. If we
concentrate on results rather than intentions, we could conclude that it is
important to do an equally good job of assessment for each group, not that we
need to use the same
4
measures for all to accomplish that goal. We want equality
of results, not process.
Sternberg's (1985, 1986) work on intelligence might prove
instructive here. He suggests that there are three kinds of intelligence.
Componential intelligence is the ability to interpret information in a hierarchical
and taxonomic fashion in a well-defined and unchanging context. People
who do well on standardized tests such as the ACT or GRE have this type of
intelligence. Experiential intelligence involves the ability to interpret
information in changing contexts; to be creative. Standardized tests do not
measure this type of intelligence, according to Sternberg. Sternberg calls his
third type of intelligence contextual; it has to do with the ability to adapt to
a changing environment; the ability to handle and negotiate the system.
If we apply Sternberg's types of intelligence to what we
typically do in college admissions for example, we heavily concentrate on
componential intelligence. Work in assessing nontraditional variables with the
NCQ suggests that nontraditional people tend to show their abilities more often
through experiential and contextual intelligence. (Boyer and Sedlacek, 1982;
Sedlacek, 1989, 1991; Sedlacek and Adams-Gaston, 1992; Tracey and Sedlacek,
1984, 1985, 1987, 1988, 1989; White and Sedlacek, 1986). Much of this is out of
necessity since nontraditional people must learn to be "bicultural"
and examine issues from different perspectives, and be able to negotiate a
5
system that was not designed for them. Having long range
goals, a self-concept that includes how the system views you and an
ability to handle racism are some of the scales on the NCQ. Institutional
racism is defined as the negative consequences that accrue to a member of a
given group because of the way a system or subsystem operates in the society
(e.g., college admissions) regardless of any other attributes of the individual
(Sedlacek, 1988). So if we concentrate on componential intelligence we do less
valid assessments for-nontraditional persons than for those with more
traditional experiences in the system and we have an example of institutional
racism.
It is not that componential intelligence is not important
to nontraditional people, it may be that experiential and contextual abilities
may be prerequisite (Westbrook and Sedlacek, 1988). If one is struggling with
racism in the system it doesn't allow the time and energy to show ones
componential talents.
My point here is to illustrate that we need to think
differentially in terms of measures if we wish to achieve equitable assessments
for all. We likely have a classic oxymoron here in thinking that we can assess
diversity of experience with a single measure. Notice that this approach is
also positive and proactive. I am not advocating
lowering standards of
assessment. I am suggesting that we develop and use the most valid measures we
can for all groups that we can operationally define.
6
The point could be made that we seem to be getting more
and more literature on assessments with nontraditional groups; isn't this
showing great process? The answer to me is a little yes but mostly no.
Unfortunately, the most common paradigm for a study that involves
nontraditional persons is probably "we are doing the study anyway so lets
add some (Blacks, women, Hispanics, etc.) to it and see how they look".
The nontraditional subjects are often "throw ins". We didn't design
the study or the measures with them in mind but we include them anyway. Our
motives are probably quite varied; from being politically correct to having a
genuine interest in a given nontraditional group. But as I stated above,
results not intentions are what count.
If we continue to employ this methodology over and over
again in different settings, we get the same results over and over again. The
research isn't developmental, or sophisticated. It's not vertical or inductive;
its not going anywhere. It also tends to give us a false sense of security or
"self righteousness" by having us count the number of studies and
reach an erroneous conclusion. For example, this is what I feel has happened in
college admissions testing. We have drawn samples of available nontraditional
students on campus after campus, correlated their scores with grades and
reached conclusions. Much less often have we built our studies upon theory and
earlier research and developed new measures or higher order constructs.
7
But I do not want to leave the impression that this only
applies to admissions testing. I was recently attending a presentation by a
colleague who was developing a new instrument in the career area. An African-American
graduate student of mine was also attending and I introduced her to my
colleague indicating that the student was interested in career issues for
African-Americans. I asked him if he had any information on how useful
the instrument was for African-Americans. He said he had not checked that
out yet but that was a good idea. I then said that if some of the constructs he
was measuring differed for African-Americans he might have excluded some
important issues, such as handling racism, which might need to be included in
the development of an instrument that was useful with African-Americans. He
admitted he had never thought of that and that perhaps we could talk further. I
mention the students' presence here to illustrate a point I will make later
about how we train people to do assessments.
I believe my story is a typical situation. My colleague is
a highly respected, well trained, humanistic person. He would not intentionally
engage in racism, but the effect of what he was doing was just that. It is
likely that his new instrument will not be as valid for African-Americans
as for others and I believe he was caught in the horizontal research problem.
There was no developmental or sophisticated thinking or planning relative to
diversity that was taking place here.
8
Any good methodological reference will tell us to guard
against loss of subjects in doing assessment research and in developing norms
for our instruments ( e.g., Mehrens and Lehmann, 1991). However, in practical
situations, we almost always have missing or incomplete data, and we commonly
make the assumption that the missing subjects do not cause our sample to be
biased. When the missing subjects are from nontraditional groups, however, that
assumption frequently will be incorrect.
When I took my first undergraduate course in statistics, I
remember the instructor pointing out the distinction between statistical bias
"a consistent overestimate or underestimate of a parameter," and
prejudice or social "bias" of some sort. I wonder if that was a good
idea. Much of the work we do in assessment has become so esoteric and molecular
that we tend to forget or perhaps never think of, the larger implications of
the bias issue.
An example of sampling bias was documented by the
institutional research office at a university. In order to develop regression
equations for use in student selection, data were included for all students
whose SAT scores and high school grades were
in the university's data base.
The equations resulting from this process were found to predict equally well
for all races. However, upon further probing, it was noted that students with
incomplete data were not included in the equation generating data set and
students in special program were less
9
likely to be included. The rationale was that the number
of exclusions was small and that the researchers were using all the data
available.
Analyses of the data from the missing groups showed that
the relationships between predictors and criteria were very different for these
students. On closer inspection, it was discovered that students for whom the
SAT and high school record did not predict as well were less likely to have
data in student data base, either because they didn't turn it in, or because
they didn't take the SAT and/or were in a special program that generated its
own data. Such students were also much more likely to be nontraditional. In
this example, the loss of subjects resulted in more sampling bias for
nontraditional students than for traditional students. The regression equations
produced on the biased data set were used to select students for admission for
several years. Consequently, in this case, sampling bias resulted in the use of
selection procedures which were less valid or invalid for many traditional
applicants.
Sampling bias which invalidates assessments for
nontraditional persons are common. For example, the problem of differential quality
of samples when studying African-Americans in career development studies
was noted by Carter and Cook (1992) .
Sampling bias can be reduced by checking the
characteristics of population members with missing data, asking ourselves what the effects might be on people
with similar characteristics
if
10
the missing data are ignored, and working to get data from
all members of the sample or population. If it is not possible to get complete
data, extraordinary efforts should be made to get data on a well-drawn
sample of those missing, so that the extent of the sampling bias can be
estimated.
Overall parameter estimates for the population can then be
modified or, at the very least, cautions can be given so that the biased
equations or norms are treated as more fallible. This should result in less
rigid applications of the results of assessment studies to people for whom they
may not be appropriate.
However, as I discuss in the next section the lack of
knowledge of nontraditional populations by assessment specialists often
prevents a sophisticated examination of variables on which sampling bias my
occur.
At the present time there are few assessment specialists
who are adequately trained in developing measures for use with nontraditional
populations. There are many graduate programs that turn out methodologists with
good technical and quantitative skills, but few of these graduates have even a
rudimentary appreciation of racial and cultural variables, let alone any sense
of how to measure them. Conversely, those most interested in cultural and
racial variables tend to have little training or interest in quantification or
assessment. As a result of the dearth of well-trained professionals
working in the area, there
11
are very few assessment techniques available for work with
nontraditional groups.
It may be that differences in vocational interests explain
the lack of quantitatively-oriented programs working on issues relevant
to nontraditional persons. Holland's (1985) research on vocational preferences
suggests that "investigative" and "realistic" types tend to
be interested in assessment and quantification, while "social" types
tend to be interested in issues related to culture, race, and nontraditionality.
Helms (1992) feels that the relatively small numbers of
VREG members of the American Psychological Association (3.1% and no
psychometricians; see Russo, Almedo, Stapp and Fulcher, 1981) requires that
those trained under the Eurocentric tradition in testing must develop the
needed culturally sensitive measures.
Obviously, requiring those interested in assessment to
take courses in multicultural issues, and requiring those with multicultural
interests to take measurement courses is one answer. However, this already
takes place in many training programs and it is not nearly enough to change
much behavior in the profession. We need to provide more practice,
opportunities for collaboration in instrument development through grants,
workshops, etc., as well theses, dissertations and everyday professional
activities.
For example, in the situation I discussed earlier
involving the African-American graduate student and the colleague there
are several things that I am trying to
do. First, I have begun a collaboration
with the student and the colleague on this topic. The student could use the
experience and the colleague needs some input. I am playing a consulting role.
Had I not suggested they work together, the student may not have been able to
identify a problem with the measure and the issue of use of the measure for
nontraditional populations may have never come up for my colleague. However,
simply putting the two together without some further effort on my part may not
reduce the skepticism of the student or the reluctance of my colleague.
I believe there are things AAC can do to help on this
issue. More publications, either free standing or part of the journal or
newsletter are recommended. Also regular and preconvention programs could be
supported, including more with the Association for Multicultural Counseling and
Development (AMCD). Obtaining grants and contracts from or with the American
Counseling Association (ACA) to provide such training for AAC members or with
knowledgeable AAC members working with others all appear worthwhile. Like the
national deficit it took us a long time to develop our problems in this area,
but the sooner we begin to solve them the better off we will all be. I have
discussed several problems that work against promoting diversity. These
problems are in no sense independent. Some are easier to reduce or control than
others, but I have offered some suggestions as to how we can attempt to remedy
each problem.
What we do about these problems and many others that are undoubtedly present in our procedures, may be conceptualized in
13
terms of the classical statistical error model: Type I vs.
Type II. If we proceed to implement some of the approaches discussed here and
the hypotheses I have proposed are false, we would be making a Type I error and
wasting considerable time, effort, and money. Conversely, if the hypotheses are
true and we conclude that differences based on nontraditionality are not
present or at least not worth investing further, we would be making a Type II
error, but we would also be running the risk of producing an extremely negative
series of outcomes for many groups by perpetuating biased approaches to testing
and measurement. I believe the very least we can do, if we are serious about
promoting diversity, is to tolerate the possibility of a Type I error, and
increase the power of our currently used procedures by working to eliminate the
molar problems presented here as well as more molecular problems such as
differential item bias.
14
Balenger, V. J., Hoffman, M. A., & Sedlacek, W. E.
(1992). Racial attitudes among incoming white students: A
study of ten-year trends. Journal
of College Student Development 33 (3) 245-252.
Boyer, S. P., & Sedlacek, W. E. (1988). Noncognitive
predictors of academic success for international
students: A longitudinal study. Journal
of College Student Development, 29, 218-222.
Carter, R. T., & Cook, D. A. (1992). A culturally
relevant perspective for understanding the career paths of
visible racial/ethnic group
people. In H. D. Lea & Z. B. Leibowitz (Eds.), Adult Career Development:
Concepts, Issues and Practices (pp. 192-217). Washington, D.C.:
National Career Development Association.
Engstrom, C. M., & Sedlacek, W. E. (1991). A study of
prejudice toward university student-athletes. Journal
of Counseling and Development, 70,
189-193.
Helms, J. E. (1992). Why is there no study of cultural
equivalence in standardized cognitive ability testing?
American Psychologist, 47,
1083-1101.
Holland, J. (1985). The Self Directed Search.
Odess, FL: Psychological Assessment Resources.
Mehrens, W. A., and Lehmann, I. J. (1991). Measurement
and evaluation in education and psychology. Ft Worth:
Holt Rinehart and Winston.
15
Minatoya, L. Y., & Sedlacek, W. E. (1983). The
Situational Attitude Scale toward women: (SASW): A means to
measure environmental sexism. Journal
of the National Association for Women Deans, Administrators, and Counselors,
47 (1), 2630.
Prediger, D. J. (1993). Multicultural assessment
standards: A compilation for counselors. Alexandria, Va.
American Counseling Association.
Russo, N: F., Olmedo, E. L., Stapp, J., & Fulcher, R.
(1981). Women and minorities in psychology. American
Psychologist, 36,
1315-1365.
Schwalb, S. J., & Sedlacek, W. E. (1990). Have college
student attitudes toward older people changed? Journal
of College Student Development, 31,
127-132.
Sedlacek, W. E. (1988). Institutional racism and how to
handle it. Health Pathways, 10 (9), 4-6.
Sedlacek, W. E. (1989). Noncognitive indicators of student
success. Journal of College Admissions, 1 (Fall)
(125), 2-9.
Sedlacek, W. E. (1991). Using noncognitive variables in
advising nontraditional students. National Academic
Advising Association Journal, 2 (1),
75-82.
Sedlacek, W. E., & Adams-Gaston, J. (1992).
Predicting the academic success of student-athletes using SAT and
noncognitive variables. Journal
of Counseling and Development 70 (6), 724-727.
16
Sternberg, R. J. (1985). Beyond 10. London:
Cambridge University Press.
Sternberg, R. J. (1986). What would better intelligence
tests look like? In Measures in the
College
Admissions Process (pp. 146-150).
New York: The College Entrance Examination Board.
Stovall, C., & Sedlacek, W. E. (1983). Attitudes of
male and female university students toward students with
different physical disabilities.
Journal of College Student Personnel, 24, 325-330.
Sue, D. W.,
Arredondo, P., & McDavis , R. J. (1992). Multicultural counseling
competencies and standards: A
call to the profession. Journal
of Counseling and Development, 70, 477-486.
Tracey, T. J., & Sedlacek, W. E. (1984). Noncognitive
variables in predicting academic success by race.
Measurement and Evaluation in
Guidance, 16, 171-178.
Tracey, T. J., & Sedlacek, W. E. (1985). The
relationship of noncognitive variables to academic success: A
longitudinal comparison by race.
Journal of College Student Personnel, 26, 405-410.
Tracey, T. J., & Sedlacek, W. E. (1987). Prediction of
college graduation using noncognitive variables by
race. Measurement and
Evaluation in Counseling and Development, 19, 177-184.
17
Tracey, T. J., & Sedlacek, W. E. (1988). A comparison
of White and Black student academic success using
noncognitive variables: A LISREL
analysis. Research in Higher Education, 27, 333-348.
Tracey, T. J., & Sedlacek, W. E. (1989). Factor
structure of the Noncognitive Questionnaire-Revised across
samples of Black and White
college students. Educational and Psychological Measurement, 49, 637-647.
Washington, J. E. (1993). An investigation of attitudes
of heterosexual identified resident assistant towards
students based on
the sexual orientation of the student. Unpublished doctoral dissertation,
University
of Maryland, College Park, Md.
Westbrook, F. D., & Sedlacek, W. E. (1988). Workshop
on using noncognitive variables with minority students
in higher education. Journal
for Specialists in Group Work, 13, 82-89.
Westbrook, F. D., & Sedlacek, W. E. (1991). Forty
years of using labels to communicate about nontraditional
students: Does it help or hurt? Journal
of Counseling and Development, 70, 20-28.
White, T. J., & Sedlacek, W. E. (1986). Noncognitive
predictors of grades and retention for specially
admitted students. Journal of
College Admissions, 3, 20-23.
18
White, T. J., & Sedlacek, W. E. (1987). White student
attitudes toward Blacks and Hispanics: Programming
implications. Journal of
Multicultural Counseling and Development, 15, 171-182.
19