COLLEGE PARK, MARYLAND
Issues in Advancing Diversity Through Assessment*
William E. Sedlacek
Research Report #5-93
* Portions of this article are from an invited address to the Association for Assessment in Counseling at the American Counseling Association National Convention, March 15, 1993, Atlanta.
UNIVERSITY OF MARYLAND AT COLLEGE PARK
COLLEGE PARK, MARYLAND
Issues in Advancing Diversity Through Assessment
William E. Sedlacek
Research Report 45-93
Five problems relating to diversity that exist in the counseling and personnel professions are presented. A discussion of how those concerned with assessment can address each problem is presented.
The first problem (The Quest for the Golden Label) concerns operationally defining diversity through measurement rather than to seek the perfect label. Measures such as the Situational Attitude Scale (SAS) and Noncognitive Questionnaire (NCQ) are discussed as examples.
The "Three Musketeers" problem is a tendency to feel that fairness means using a single measure for all. Nontraditional populations may present their abilities and attributes in different ways, necessitating different measures to achieve equal results.
The "Horizontal Research" problem deals with our tendency to study nontraditional populations as afterthoughts in the design of research studies. Hence, we do not achieve developmental or inductive research programs for nontraditional groups.
The "Bias is Bias" problem involves a discussion of sampling bias and how it can be an example of racism in how we interpret results of studies.
The lack of professionals trained in assessment and an understanding of cultural and racial variables is called the "I'm OK, You're Not," problem.
Overall suggestions for change are also made such as more joint training programs and projects among professional groups.
This title suggests an important focus which is recent in our profession; using assessment as a proactive tool to initiate change rather than in a more passive, defensive way to insure that we aren't harming someone.
The Association for Assessment in Counseling (AAC) has developed a new document on multicultural assessment standards (Prediger 1993) which is a compilation from a number of sources. It is a detailed listing of ethical and professional standards which I recommend for your study. I will not attempt to go over the details of that publication but I will concentrate on some problems relating to diversity that are compatible with that publication that I feel exist in the counseling and personnel professions, and how those concerned with assessment can promote diversity through their efforts.
While there are many problems that could be addressed, I will concentrate on five here today. The first problem I call the "Quest for the Golden Label". The idea here is that if we could just find the right label, our problems would be solved. The Quest for the Golden Label Problem
There is often much confusion about how we define diversity, what groups should be included and what terms we should apply to those groups. Questions arise such as; Should we include gays and lesbians in our conceptualization? Can we use Black and African-American interchangeably? Is the whole issue just a matter of being politically correct?
While labels and terminology are important and contain many symbolic and connotative elements, those concerned with assessment can help cut through some of the ambiguity by helping to operationally define terms. Differences or similarities among groups can be demonstrated through measurement.
Westbrook and Sedlacek (1991) studied the labels used to describe what they called "nontraditional students" in the Education Index over forty years. Terms have varied from a focus on acculturation in the 1950's to disadvantaged in the 1960's, to culture-specific differences in the 1970's to multicultural in the 1980's. We could add diversity as the probable term for the 1990's. While these terms may suggest different approaches to the groups discussed, operationally we may still be discussing the same people; those with cultural experiences different than those of White middle class males of European descent; those with less power to control their lives; and those who receive discrimination in the United States. But does it make sense to include such variables as gender, sexual orientation, or status as an athlete as aspects of cultural experience?
Here is where we can apply some principles of assessment. As one example of how I have tried to address this question, I and students and colleagues have developed measures in two areas and applied them to many different groups. The first is the Situational Attitude Scale (SAS) which measures prejudicial attitudes toward different groups including racial/cultural groups ( Balenger, Hoffman, and Sedlacek, 1992; White and
Sedlacek, 1987), women (Minatoya and Sedlacek, 1983; older people (Schwalb and Sedlacek, 1990), gays and lesbians (Washington, 1993) people with physical disabilities (Stovall and Sedlacek, 1983), and athletes (Engstrom and Sedlacek, 1991). If a group is the target of prejudicial attitudes they meet my first criterion for inclusion as a nontraditional group.
The second measure that we have developed is the Noncognitive Questionnaire (NCQ) which is designed to assess attributes that are not typically measured by other instruments and that may be common ways for persons with nontraditional experiences to show their abilities (Sedlacek, 1989). I will discuss the NCQ further under another problem. If the NCQ systematically correlates more with the success of members of a given group (e.g., Blacks) than it does for traditional White males, then the group meets my second criterion for inclusion as a nontraditional group.
In summary, if a group receives prejudice and demonstrates abilities in ways different from those with traditional experiences, they are operationally defined as nontraditional. Even though groups as different as athletes and older people may show their diversity in different ways, the variables underlying their problems in dealing with their development, and in coping with a traditional system that was not designed for them, may have some similarities.
There are many who would disagree with my conclusions or my definition of nontraditional. For example, Helms (1992) sees visible racial/ethnic groups (VREGS) as having some distinct problems and characteristics. Also Sue, Arredondo and McDavis (1992) express concern that the term cross-cultural can be defined so broadly that it allows counseling professionals to avoid dealing with what they see as the four major minority groups in our society: African Americans, American Indians, Asian Americans, and Hispanics and Latinos.
My concern is not that we conclude that one conception or another is best but that we base our decisions on measurement principles and empirical evidence wherever possible. It is quite possible that we can identify relevant variance based on nontraditionality and more specific group memberships as well.
The rallying cry of "all for one and one for all" is one that we use often in developing what we think of as fair and equitable measures. Our interpretation of how to handle diversity is to hone and fine-tune our measures so they are equally valid for everyone. However, if different groups have different experiences and different ways of presenting their attributes and abilities, it is unlikely that we could develop a single measure, test item etc. that could be equally valid for all. If we concentrate on results rather than intentions, we could conclude that it is important to do an equally good job of assessment for each group, not that we need to use the same
measures for all to accomplish that goal. We want equality of results, not process.
Sternberg's (1985, 1986) work on intelligence might prove instructive here. He suggests that there are three kinds of intelligence. Componential intelligence is the ability to interpret information in a hierarchical and taxonomic fashion in a well-defined and unchanging context. People who do well on standardized tests such as the ACT or GRE have this type of intelligence. Experiential intelligence involves the ability to interpret information in changing contexts; to be creative. Standardized tests do not measure this type of intelligence, according to Sternberg. Sternberg calls his third type of intelligence contextual; it has to do with the ability to adapt to a changing environment; the ability to handle and negotiate the system.
If we apply Sternberg's types of intelligence to what we typically do in college admissions for example, we heavily concentrate on componential intelligence. Work in assessing nontraditional variables with the NCQ suggests that nontraditional people tend to show their abilities more often through experiential and contextual intelligence. (Boyer and Sedlacek, 1982; Sedlacek, 1989, 1991; Sedlacek and Adams-Gaston, 1992; Tracey and Sedlacek, 1984, 1985, 1987, 1988, 1989; White and Sedlacek, 1986). Much of this is out of necessity since nontraditional people must learn to be "bicultural" and examine issues from different perspectives, and be able to negotiate a
system that was not designed for them. Having long range goals, a self-concept that includes how the system views you and an ability to handle racism are some of the scales on the NCQ. Institutional racism is defined as the negative consequences that accrue to a member of a given group because of the way a system or subsystem operates in the society (e.g., college admissions) regardless of any other attributes of the individual (Sedlacek, 1988). So if we concentrate on componential intelligence we do less valid assessments for-nontraditional persons than for those with more traditional experiences in the system and we have an example of institutional racism.
It is not that componential intelligence is not important to nontraditional people, it may be that experiential and contextual abilities may be prerequisite (Westbrook and Sedlacek, 1988). If one is struggling with racism in the system it doesn't allow the time and energy to show ones componential talents.
My point here is to illustrate that we need to think differentially in terms of measures if we wish to achieve equitable assessments for all. We likely have a classic oxymoron here in thinking that we can assess diversity of experience with a single measure. Notice that this approach is also positive and proactive. I am not advocating lowering standards of assessment. I am suggesting that we develop and use the most valid measures we can for all groups that we can operationally define.
The point could be made that we seem to be getting more and more literature on assessments with nontraditional groups; isn't this showing great process? The answer to me is a little yes but mostly no. Unfortunately, the most common paradigm for a study that involves nontraditional persons is probably "we are doing the study anyway so lets add some (Blacks, women, Hispanics, etc.) to it and see how they look". The nontraditional subjects are often "throw ins". We didn't design the study or the measures with them in mind but we include them anyway. Our motives are probably quite varied; from being politically correct to having a genuine interest in a given nontraditional group. But as I stated above, results not intentions are what count.
If we continue to employ this methodology over and over again in different settings, we get the same results over and over again. The research isn't developmental, or sophisticated. It's not vertical or inductive; its not going anywhere. It also tends to give us a false sense of security or "self righteousness" by having us count the number of studies and reach an erroneous conclusion. For example, this is what I feel has happened in college admissions testing. We have drawn samples of available nontraditional students on campus after campus, correlated their scores with grades and reached conclusions. Much less often have we built our studies upon theory and earlier research and developed new measures or higher order constructs.
But I do not want to leave the impression that this only applies to admissions testing. I was recently attending a presentation by a colleague who was developing a new instrument in the career area. An African-American graduate student of mine was also attending and I introduced her to my colleague indicating that the student was interested in career issues for African-Americans. I asked him if he had any information on how useful the instrument was for African-Americans. He said he had not checked that out yet but that was a good idea. I then said that if some of the constructs he was measuring differed for African-Americans he might have excluded some important issues, such as handling racism, which might need to be included in the development of an instrument that was useful with African-Americans. He admitted he had never thought of that and that perhaps we could talk further. I mention the students' presence here to illustrate a point I will make later about how we train people to do assessments.
I believe my story is a typical situation. My colleague is a highly respected, well trained, humanistic person. He would not intentionally engage in racism, but the effect of what he was doing was just that. It is likely that his new instrument will not be as valid for African-Americans as for others and I believe he was caught in the horizontal research problem. There was no developmental or sophisticated thinking or planning relative to diversity that was taking place here.
Any good methodological reference will tell us to guard against loss of subjects in doing assessment research and in developing norms for our instruments ( e.g., Mehrens and Lehmann, 1991). However, in practical situations, we almost always have missing or incomplete data, and we commonly make the assumption that the missing subjects do not cause our sample to be biased. When the missing subjects are from nontraditional groups, however, that assumption frequently will be incorrect.
When I took my first undergraduate course in statistics, I remember the instructor pointing out the distinction between statistical bias "a consistent overestimate or underestimate of a parameter," and prejudice or social "bias" of some sort. I wonder if that was a good idea. Much of the work we do in assessment has become so esoteric and molecular that we tend to forget or perhaps never think of, the larger implications of the bias issue.
An example of sampling bias was documented by the institutional research office at a university. In order to develop regression equations for use in student selection, data were included for all students whose SAT scores and high school grades were in the university's data base. The equations resulting from this process were found to predict equally well for all races. However, upon further probing, it was noted that students with incomplete data were not included in the equation generating data set and students in special program were less
likely to be included. The rationale was that the number of exclusions was small and that the researchers were using all the data available.
Analyses of the data from the missing groups showed that the relationships between predictors and criteria were very different for these students. On closer inspection, it was discovered that students for whom the SAT and high school record did not predict as well were less likely to have data in student data base, either because they didn't turn it in, or because they didn't take the SAT and/or were in a special program that generated its own data. Such students were also much more likely to be nontraditional. In this example, the loss of subjects resulted in more sampling bias for nontraditional students than for traditional students. The regression equations produced on the biased data set were used to select students for admission for several years. Consequently, in this case, sampling bias resulted in the use of selection procedures which were less valid or invalid for many traditional applicants.
Sampling bias which invalidates assessments for nontraditional persons are common. For example, the problem of differential quality of samples when studying African-Americans in career development studies was noted by Carter and Cook (1992) .
Sampling bias can be reduced by checking the characteristics of population members with missing data, asking ourselves what the effects might be on people with similar characteristics if
the missing data are ignored, and working to get data from all members of the sample or population. If it is not possible to get complete data, extraordinary efforts should be made to get data on a well-drawn sample of those missing, so that the extent of the sampling bias can be estimated.
Overall parameter estimates for the population can then be modified or, at the very least, cautions can be given so that the biased equations or norms are treated as more fallible. This should result in less rigid applications of the results of assessment studies to people for whom they may not be appropriate.
However, as I discuss in the next section the lack of knowledge of nontraditional populations by assessment specialists often prevents a sophisticated examination of variables on which sampling bias my occur.
At the present time there are few assessment specialists who are adequately trained in developing measures for use with nontraditional populations. There are many graduate programs that turn out methodologists with good technical and quantitative skills, but few of these graduates have even a rudimentary appreciation of racial and cultural variables, let alone any sense of how to measure them. Conversely, those most interested in cultural and racial variables tend to have little training or interest in quantification or assessment. As a result of the dearth of well-trained professionals working in the area, there
are very few assessment techniques available for work with nontraditional groups.
It may be that differences in vocational interests explain the lack of quantitatively-oriented programs working on issues relevant to nontraditional persons. Holland's (1985) research on vocational preferences suggests that "investigative" and "realistic" types tend to be interested in assessment and quantification, while "social" types tend to be interested in issues related to culture, race, and nontraditionality.
Helms (1992) feels that the relatively small numbers of VREG members of the American Psychological Association (3.1% and no psychometricians; see Russo, Almedo, Stapp and Fulcher, 1981) requires that those trained under the Eurocentric tradition in testing must develop the needed culturally sensitive measures.
Obviously, requiring those interested in assessment to take courses in multicultural issues, and requiring those with multicultural interests to take measurement courses is one answer. However, this already takes place in many training programs and it is not nearly enough to change much behavior in the profession. We need to provide more practice, opportunities for collaboration in instrument development through grants, workshops, etc., as well theses, dissertations and everyday professional activities.
For example, in the situation I discussed earlier involving the African-American graduate student and the colleague there are several things that I am trying to do. First, I have begun a collaboration with the student and the colleague on this topic. The student could use the experience and the colleague needs some input. I am playing a consulting role. Had I not suggested they work together, the student may not have been able to identify a problem with the measure and the issue of use of the measure for nontraditional populations may have never come up for my colleague. However, simply putting the two together without some further effort on my part may not reduce the skepticism of the student or the reluctance of my colleague.
I believe there are things AAC can do to help on this issue. More publications, either free standing or part of the journal or newsletter are recommended. Also regular and preconvention programs could be supported, including more with the Association for Multicultural Counseling and Development (AMCD). Obtaining grants and contracts from or with the American Counseling Association (ACA) to provide such training for AAC members or with knowledgeable AAC members working with others all appear worthwhile. Like the national deficit it took us a long time to develop our problems in this area, but the sooner we begin to solve them the better off we will all be. I have discussed several problems that work against promoting diversity. These problems are in no sense independent. Some are easier to reduce or control than others, but I have offered some suggestions as to how we can attempt to remedy each problem.
What we do about these problems and many others that are undoubtedly present in our procedures, may be conceptualized in
terms of the classical statistical error model: Type I vs. Type II. If we proceed to implement some of the approaches discussed here and the hypotheses I have proposed are false, we would be making a Type I error and wasting considerable time, effort, and money. Conversely, if the hypotheses are true and we conclude that differences based on nontraditionality are not present or at least not worth investing further, we would be making a Type II error, but we would also be running the risk of producing an extremely negative series of outcomes for many groups by perpetuating biased approaches to testing and measurement. I believe the very least we can do, if we are serious about promoting diversity, is to tolerate the possibility of a Type I error, and increase the power of our currently used procedures by working to eliminate the molar problems presented here as well as more molecular problems such as differential item bias.
Balenger, V. J., Hoffman, M. A., & Sedlacek, W. E. (1992). Racial attitudes among incoming white students: A
study of ten-year trends. Journal of College Student Development 33 (3) 245-252.
Boyer, S. P., & Sedlacek, W. E. (1988). Noncognitive predictors of academic success for international
students: A longitudinal study. Journal of College Student Development, 29, 218-222.
Carter, R. T., & Cook, D. A. (1992). A culturally relevant perspective for understanding the career paths of
visible racial/ethnic group people. In H. D. Lea & Z. B. Leibowitz (Eds.), Adult Career Development: Concepts, Issues and Practices (pp. 192-217). Washington, D.C.: National Career Development Association.
Engstrom, C. M., & Sedlacek, W. E. (1991). A study of prejudice toward university student-athletes. Journal
of Counseling and Development, 70, 189-193.
Helms, J. E. (1992). Why is there no study of cultural equivalence in standardized cognitive ability testing?
American Psychologist, 47, 1083-1101.
Holland, J. (1985). The Self Directed Search. Odess, FL: Psychological Assessment Resources.
Mehrens, W. A., and Lehmann, I. J. (1991). Measurement and evaluation in education and psychology. Ft Worth:
Holt Rinehart and Winston.
Minatoya, L. Y., & Sedlacek, W. E. (1983). The Situational Attitude Scale toward women: (SASW): A means to
measure environmental sexism. Journal of the National Association for Women Deans, Administrators, and Counselors, 47 (1), 2630.
Prediger, D. J. (1993). Multicultural assessment standards: A compilation for counselors. Alexandria, Va.
American Counseling Association.
Russo, N: F., Olmedo, E. L., Stapp, J., & Fulcher, R. (1981). Women and minorities in psychology. American
Psychologist, 36, 1315-1365.
Schwalb, S. J., & Sedlacek, W. E. (1990). Have college student attitudes toward older people changed? Journal
of College Student Development, 31, 127-132.
Sedlacek, W. E. (1988). Institutional racism and how to handle it. Health Pathways, 10 (9), 4-6.
Sedlacek, W. E. (1989). Noncognitive indicators of student success. Journal of College Admissions, 1 (Fall)
Sedlacek, W. E. (1991). Using noncognitive variables in advising nontraditional students. National Academic
Advising Association Journal, 2 (1), 75-82.
Sedlacek, W. E., & Adams-Gaston, J. (1992). Predicting the academic success of student-athletes using SAT and
noncognitive variables. Journal of Counseling and Development 70 (6), 724-727.
Sternberg, R. J. (1985). Beyond 10. London: Cambridge University Press.
Sternberg, R. J. (1986). What would better intelligence tests look like? In Measures in the College
Admissions Process (pp. 146-150). New York: The College Entrance Examination Board.
Stovall, C., & Sedlacek, W. E. (1983). Attitudes of male and female university students toward students with
different physical disabilities. Journal of College Student Personnel, 24, 325-330.
Sue, D. W., Arredondo, P., & McDavis , R. J. (1992). Multicultural counseling competencies and standards: A
call to the profession. Journal of Counseling and Development, 70, 477-486.
Tracey, T. J., & Sedlacek, W. E. (1984). Noncognitive variables in predicting academic success by race.
Measurement and Evaluation in Guidance, 16, 171-178.
Tracey, T. J., & Sedlacek, W. E. (1985). The relationship of noncognitive variables to academic success: A
longitudinal comparison by race. Journal of College Student Personnel, 26, 405-410.
Tracey, T. J., & Sedlacek, W. E. (1987). Prediction of college graduation using noncognitive variables by
race. Measurement and Evaluation in Counseling and Development, 19, 177-184.
Tracey, T. J., & Sedlacek, W. E. (1988). A comparison of White and Black student academic success using
noncognitive variables: A LISREL analysis. Research in Higher Education, 27, 333-348.
Tracey, T. J., & Sedlacek, W. E. (1989). Factor structure of the Noncognitive Questionnaire-Revised across
samples of Black and White college students. Educational and Psychological Measurement, 49, 637-647.
Washington, J. E. (1993). An investigation of attitudes of heterosexual identified resident assistant towards
students based on the sexual orientation of the student. Unpublished doctoral dissertation, University
of Maryland, College Park, Md.
Westbrook, F. D., & Sedlacek, W. E. (1988). Workshop on using noncognitive variables with minority students
in higher education. Journal for Specialists in Group Work, 13, 82-89.
Westbrook, F. D., & Sedlacek, W. E. (1991). Forty years of using labels to communicate about nontraditional
students: Does it help or hurt? Journal of Counseling and Development, 70, 20-28.
White, T. J., & Sedlacek, W. E. (1986). Noncognitive predictors of grades and retention for specially
admitted students. Journal of College Admissions, 3, 20-23.
White, T. J., & Sedlacek, W. E. (1987). White student attitudes toward Blacks and Hispanics: Programming
implications. Journal of Multicultural Counseling and Development, 15, 171-182.