Validity of the Noncognitive Questionnaire-Revised 2 in Predicting the

Academic Success of University Freshmen

Siu-Man Raymond Ting and William E. Sedlacek

Research Report # 1-00

Summary

A version of the Noncognitive Questionnaire (Tracey and Sedlacek, 1984), (NCQ-R2) was administered to 894 freshmen at a large Southeastern public university. The NCQ-R2 had adequate reliability in predicting first semester GPA and retention. Due to sample size limitations, data were analyzed for White students only. Based on a factor analysis, the factors that were related to retention included high school coursework, high school extracurricular activities and living in a multicultural society. The study provides further evidence that noncognitive variables can be useful in making college admissions decisions.
The Noncognitive Questionnaire-Revised 2 in Predicting Academic

Success of College Freshmen

Background

The overall national college dropout rate was about 33% in the past few years (American College Testing Program, 1999). Higher education has responded to such a high attrition rate. Numerous professional literatures explored factors affecting academic performance and student retention (Astin, 1993; Pascarella & Terenzini, 1991; Sedlacek, 1996; Tinto; 1993). Traditional studies adopting standardized test scores and high school grade-point-average (GPA) to predict students’ academic performance produced comparatively low validity (Houston, 1980, Sedlacek, 1998; Stanley, 1971). Recent studies shifted the focus to non-cognitive or nontraditional factors. An approach, the Non-Cognitive Questionnaire (NCQ; Tracey & Sedlacek, 1984, 1989), consisting of eight psychosocial variables to explain admission decisions has been well studied and applied in universities.

The Noncognitive Questionnaire

The Noncognitive Questionnaire (NCQ; Tracey & Sedlacek, 1984, 1989) was designed to assess psychosocial aspects of students that influence college success. It contains 23 items: 18 Likert-formatted, 2 multiple choice, and 3 open-ended. It consists of measures of eight variables with possible score ranges printed in brackets in the following: (a) positive self-concept (7-27), (b) realistic self-appraisal (4-14), (c) demonstrated community service (2-8), (d) knowledge acquired in a field (2-8), (e) successful leadership experience (3-13), (f) preference of long-range goals over short-term, immediate goals and ability to defer gratification to attain goals (3-13), (g) ability to understand and cope with racism (5-25), and (h) availability of a strong support person (3-15). Tracey and Sedlacek (1984) reported a 2-week test-retest reliability of a range from .74 to .94, with a median of .85 for the NCQ items. Interrater reliability on the three open-ended NCQ items ranged from .73 to 1.00. The NCQ appears to have promising content validity, strong construct and predictive validity.

Predicting Academic Performance

The NCQ was found to have effective predictability of academic performance and student retention for different student populations: Asian Americans (Fuertes, Sedlacek, & Liu, 1994); African Africans (Boyer & Sedlacek, 1988; Sedlacek & Adams-Gaston, 1992); Hispanics (Fuertes & Sedlacek, 1995); White and African Americans (Ting & Robinson, 1998; Tracey & Sedlacek, 1984, 1985, 1987), specially admitted students (Ting, 1997a; White & Sedlacek, 1986) and low-income and first generation students (Ting, 1998). Correlations with college grades and retention were significantly higher when the non-cognitive variables were used in conjunction with standardized test scores and earlier grades. University admissions officials continually are finding appropriate ways to assess applicants with nontraditional background and experiences.

Use of the Noncognitive Questionnaire in University Admissions

There are examples of adopting the NCQ for admissions, programs, and services. After using the NCQ in the admissions committee at the Louisiana State University Medical School, minority student enrollment has doubled to 21 percent with an 87 percent retention rate. Eighty percent of the committee members reported that they believed using the non-cognitive variables in admission was worthwhile (Helm, Prieto & Sedlacek, 1997). Currently, the North Carolina State University includes in its undergraduate application package the NCQ for supplementary information. Muhlenberg College, an 1,800-student liberal arts college in Allentown, Pennsylvania collects applicants’ information about their leadership in extra-curricular activities and community service. As an option, the college allows the applicants not to report SAT or ACT scores.

Use of the Noncognitive Questionnaire in Academic and Student Support Services

The NCQ was also adopted as strategies for intervention programs and services in universities. Sedlacek (1991) reported its effectiveness as assessing instrument and guidelines for individual counseling. Its usefulness for on-going groups enhancing students’ psychosocial adjustment was reported by Ting (1997b) and Ting, Grant, and Plenert (in press). Fogleman and Saeger (1985) reported a successful application in a summer enrichment program for health majors.

The NCQ-Revised 2

The NCQ has drawn many responses from universities. A few universities adopted the NCQ for admissions. The NCQ was first revised by Tracey and Sedlacek (1989). The nature of the noncognitive variables are dynamic and they need to be studied continuously to include new variables and examine existing ones. For example, Ting and Robinson (1998) found other psychological, environmental, and personal and socio-economic background variables predicting academic success of college students. Therefore, further refinement and inclusion of new variables are needed.

Other Studies Predicting College Success & Related Literature

Studies showed that other noncognitive variables affecting student grades and retention include student involvement (Astin, 1993), academic and social integration (Milem & Berger, 1997; Tinto, 1993). study skills (Nisbet, Ruble, & Schurr, 1982), and socio-economic background, institutional and environmental variables (Ting & Robinson, 1998).

Sternberg (1993) defines intelligence as the ability to adapt to, shape, and select the environmental context, which includes physical, biological and cultural aspects. His theory of intelligence offers a conceptual framework that support the NCQ by suggesting three types of intelligence exist (Sternberg, 1985). Componential intelligence is defined as the ability to interpret information in a structured and well-defined context. Among the types of intelligence, an individual needs only componential intelligence to interpret information in standardized aptitude tests such as the SAT. The traditional admission system in higher education tends to concentrate on standardized tests, which represent this type of intelligence. Sedlacek (1996) suggested that, because the educational system is not optimally developed for students of color, many students might have developed other types of intelligence, namely contextual and experiential intelligence. Contextual intelligence refers to instrumental acts of advising, consulting, and perhaps influencing others through advice. Experiential intelligence is the ability to see issues from different points of view, interpret information in changing contexts, and to resolve practical problems. Sedlacek hypothesized that the NCQ measures the attributes of experiential and contextual intelligence.

Despite the mental processes that are common across environmental contexts; people’s ability, motivation, or decision to apply these processes across contexts may not be equal. As a result, college students may not appear equally in their intelligent behavior. Yet, our higher education focuses on using standardized test scores and grades to reflect how well the students learn. The NCQ captured the essence of Sternberg’s intelligence theory, which can affect students’ academic performance and retention. However, other dimensions including components of experiential intelligence such as students’ studying styles, ways of learning, and skills to resolve practical problems can be further explored. Also, students’motivation to study, decision to involve in college life, and their responses to different physical and cultural environment can be examined to bear new ideas and items for the NCQ. For example, Ting and Robinson (1998) found that socio-economic background, planned working hours, financial needs, and personal development and general education goals affected freshmen’ GPA and retention.

Purpose of the Current Study

The current study attempts to revise and expand the NCQ by exploring new variables, design the NCQ-R2, and study its reliability, construct validity and predictive validity (use the NCQ-R2 scores to predict freshmen’s GPA).

Statement of Research Question

What is the reliability, construct validity, and predictive validity of the NCQ-R2?

Method

Designing the NCQ-R2 and Conducting the Pilot Study

Based on the literature review, the authors designed new items for the NCQ-R2. A research assistant also interviewed undergraduate students at a southeastern public land-grant research university to collect information for the new instrument. Finally, new items were written and were studied in a pilot study (n = 46). The findings of the pilot study were used for changes and refinement of the instrument. Seventy-nine items related to non-cognitive variables and a few personal and socio-economic background items were included in the current study.

Participants

A total of 894 responses were collected (735 Whites, 79 African Americans, 11 Hispanics, 44 Asian Americans, 6 Native Americans, and 19 unknown). In the sample, 519 were males, 363 were females, and 5 unknown. Mean age was 18.15. Only the White sample was included in this report.

Procedure

Data Collection. In the fall of 1999, the authors collected 894 random responses from freshman English classes at a southeastern public land-grant research university. All freshmen were required to take these classes. The students completed the NCQ-R2. The participants were asked to consent to a review of their academic record. Then the authors obtained their first semester GPA from the university records.

Data Analysis. The NCQ-R2 was studied for its reliability, construct validity, and predictive validity. To examine the construct validity, the authors conducted a principal component factor analysis to ascertain if the NCQ-R2 items loaded on the proposed noncognitive dimensions. Only the White students were studied owing to the small number of African American participants. To find out the reliability of the NCQ-R2, the authors studied Cronbach Alpha Correlation Coefficients of each scale based on the factor analysis.

Next, the authors studied the predictive validity of the NCQ-R2 as a predictor of collegiate success. College success was defined here as students’ GPA and student retention. The authors employed step-wise multiple regression analysis to predict first-semester GPA of the White sample with the SPSS window (version 8.0) for the analyses. A step-wise discriminant analysis was used to study student retention (continued enrollment = 1; dropping-out = 0).

Results

Construct Validity and Reliability

Owing to the small sample size of African Americans, only a principal components factor analysis on 57 non-cognitive items was conducted for Whites. Table 1 shows the factors obtained, item loadings, and reliability of the scales from the factor analysis. These results support seven of the eight non-cognitive variables suggested by Tracey and Sedlacek (1984): living in a multicultural society (named as coping with racism in the original NCQ) (Factor I), knowledge acquired in a field (Factor II), leadership experiences (Factor V), positive self-concept (Factor VI), preference for long term-goals (Factor IX), realistic self-appraisal (Factor X), a strong support person (Factor XI). New factors included: high school coursework (Factor III), involvement in high school extracurricular activities (Factor IV), study method and effectiveness (Factor VII), interest and ability to relate to others (Factor VIII), support of academic plan (Factor XII), and emotional intelligence (Factor XIII).

Discussion

The results of the current study support that with regard to the Whites (a) the NCQ-R2 had adequate reliability and construct validity; (b) for the first semester GPA, the NCQ-R2 had good predictive validity; and (c) the NCQ was found to have high predictive validity for first-semester student retention. It is important that a few new non-cognitive variables were found to be strongly related to student retention. These variables included high school course work, involvement in high school extra-curricular activities, and living in a multicultural society.

Table 1

Factors, Largest Loadings Items, and Reliability for the White Sample

Item	Loading
Factor I: Living in a Multicultural Society 5.3% of total variance (.72)	.65
I expect to have little contact with students from other races.^a	.68
I am comfortable interacting with people from other races or cultures.	.72
My friends are exclusively the same race as I am.^a	.63
I feel comfortable when I am a minority among other races in a social situation.	.77
I like to make friends with people of other races.	.40
I am empathic with a diversity of people.
Factor II: Knowledge Acquired in a Field 5.3% (.78)	.69
I have studied things about my major on my own.	.55
I have talked about my career goal with someone who works in that career.	.78
I know what I want to be doing 10 years from now.	.69
I have already learned something in my proposed major/major outside of high school.	.80
I am not sure about my career choice.^b
Factor III: High School Course Work 4.9% (.74)	.54
I studied things in high school that enhanced my college study. ^b	.78
I took as many course as I could in high school to prepare myself for college.	.79
I took more than required courses in high school to prepare for college.	.76
I took vigorous and demanding courses in high school.	.32
I feel discouraged about my academic performance in high school.^a
Factor IV: Involvement in High School Extra-curricular Activities 4.5% (.72)	.65
I was a student leader in my high school.^b	.38
I know my teachers personally in high school.	.75
I participated in many extra-curricular activities in high school.	.74
Hours spent per week in extra-curricular activities in senior year of high school.	.57
Hours spent per week in community service in senior year of high school.
Factor V: Leadership Experiences 4.4% (.73)	.60
I am sometimes looked up to by others.	.65
In groups where I am comfortable, I am often looked to as a leader.	.33
I was a student leader in my high school.^b	.59
When I believe strongly on something, I act on it.	.69
My friends look to me to make decisions.	.32
I am not good at getting others to go along with me.^a	.37
I find opportunities to learn new things.
Factor VI: Positive Self –Concept 4.1% (.70)	.59
I expect to have a harder time than most students here.^a	.69
It should not be very hard to get a B average here.	.40
I feel that I am not academically well prepared for collge.^a b	.54
I am as skilled academically as the average applicant of this university.	.51
I have confidence to get good grades here.	.36
Expected highest level of education in a lifetime.	.30
My background should help me fit in here.
Factor VII: Study Method and Effectiveness 4.0% (.73)	.43
I know the areas where I am weak and I try to improve them.^b	.71
I cannot concentrate when I am studying.^a	.35
I have confidence to get good grades here.	.44
I feel that I am not well prepared academically for college.^a b	.76
I believe that my studying method is effective.	.37
I can motivate myself to achieve a task even when I am emotionally distressed.
Factor VIII: Interest and Ability to Relate to Others 3.7% (.60).	.59
I enjoy working with others.	.68
I keep to myself pretty much.^a	.41
I am not good at getting others to go along with me. ^a b	.59
I expect to find lots of people like me here.	.58
My background should help me fit in well here.
Factor IX: Preference of Long Term Goals 3.6% (.61)	.47
I prefer to be spontaneous rather than to make plans. ^a	.76
I usually mark important dates on my calendar.	.39
If tutoring is made available on campus at no cost, I will attend regularly.^b	.77
I often make lists of things to do.
Factor X: Realistic Self-Appraisal 3.3% (.59)	.35
I know the areas where I am weak and I try to improve them.^b	.68
I want a chance to prove myself academically.	.37
I try to find opportunities to learn new things.	.44
I find I get more comfortable in a new place as soon as I make good friends.	.53
My current goals are related to academics.	.51
Factor XI: A Strong Support Person 3.2% (.59)	.72
If I run into problems concerning school, I have someone who would listen to me and help me.	.61
I am always aware of my own feelings.^b	.74
I can find someone to support me when I need it.
Factor XII: Support of Academic Plans 3.0% (.75)	.84
My friends and relatives always think that I should go to college.	.81
My family always wanted me go to college.
Factor XIII: Emotional Intelligence 2.8% (.58)	.75
I can manage my emotions well.	.75
I can motivate myself to achieve a task when I am emotionally distressed.	.55
I am always aware of my own feelings.	.33
I can think clearly and maintain focused under pressure.	.46
Factor XIV: Expected Involvement with Faculty and Academic Support Services 2.4% (.38)
I don’t expect to get to know faculty personally during my first year.^a b	.37
I feel that I am not academically well prepared for college.	.33
I am not good at getting others to go along with me.	.42
I prefer to be spontaneous rather than to make plans.	.35
If tutoring is made available on campus at no cost, I will attend regularly.^b	-.34
Factor XV: Unnamed 2.4% (-.15)
I expect the faculty to treat me differently from the average student here.	-.74
I am empathic with a diversity of people.	.34

Note: Only the items with loadings above .30 are reported.

Cronbach Alpha Correlation Coefficients are reported in bracket

^a reverse scored items

^b items used for more than one factor

Predictive Validity

The items that significantly added to prediction in the analyses and the overall multiple correlation coefficients are summarized in Table 2. In the analyses, the variables found to be predictive for whites were high school coursework (item 63, and 69 as listed in Table 2), Positive self-concept (items 16 ), preference for long term goals (items 5, and 38), and study method and effectiveness (item 72). The overall variance explained for white students’ fall GPA was .38.

Table 3 reports a discriminant analysis showing a significant relationship between some noncongitive variables and student retention for whites (r = .31, p < .0001). The predictive variables were high school course work (items 47, 69), positive self-concept (items 16), involvement in high school extra-curricular activities (items 83 and 84), and living in a multicultural society (item 23) (See Table 3).

The results imply that that the NCQ-R2 can be considered for making admissions decisions for White. In the current report, SAT scores were not included for the predictions. Combining SAT scores and non-cognitive variables, the authors will complete the analysis when students’ spring semester grades and enrollment information are available. Also, longitudinal analyses will be conducted to examine the predictability of non-cognitive variables until graduation.

Table 2

Significant NCQ-R2 Items and Their Beta Weights Predicting Fall GPA of the White Sample

Item	Unstandardized Coefficients (Beta)
69. I feel discouraged about my academic performance in high school.	.16***
72. I believe that my studying method is effective.	-.08**
5. I prefer to be spontaneous rather than to make plans.	.08**
63. I took vigorous and demanding courses in high school.	-.05**
16. I expect to have a harder time than most students here.	.07*
38. I often make a list of things to do.	-.05*

***p < .0001; **p < .005 ; *p < .05

Table 3

Discriminant Analysis of Student Retention

Items in steps	Lambda	df1	df2	F
75. I try to find lots of people like me here.	.969	1	732	23.08
84. Hours spent per week on community service in senior year of high school	.954	2	731	17.70
83. Hours spent per week on extra-curricular activities in senior year of high school.	.943	3	730	14.71
47. I took more than required courses in high school to prepare for college.	.934	4	729	12.94
16. I expect to have a harder time than most students here.	.926	5	728	11.68
23. My friends are exclusively the same race as I am.	.918	6	727	10.88
39. I expect the faculty to treat me differently from the average student here.	.912	7	726	10.03
69. I feel discouraged about my academic performance in high school.	.906	8	725	9.45

All steps are significant at p < .0001

References

American College Testing Program (1999). National dropout rates. Iowa City, IA: American College Testing.

Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco: Jossey-Bass.

Boyer, S. P., & Sedlacek, W. E. (1988). Noncognitive predictors of academic success for international students: A longitudinal study. Journal of College Student Development, 29, 218-222.

Fogleman, B. S., & Saeger, W. (1985). Examining Sedlacek’s nontraditional variables of minority student success in a summer enrichment program for health careers, Journal of the National Medical Association, 78(1), 545-549.

Fuertes, J. N., & Sedlacek, W. E. (1995). Using noncognitive variables to predict the grades and retention of Hispanic students. College Student Affairs Journal, 14(2), 30-36.

Fuertes, J. N., Sedlacek, W. E., & Liu, W. M. (1994). Using the SAT and noncognitive variables to predict the grades and retention of Asian American university students. Measurement and Evaluation in Counseling and Development, 27, 74-84.

Helm, E. G., Prieto, D. O., & Sedlacek, W. E. (1997). Simulated minority admissions exercise at Louisiana State University: An evaluation. Journal of the National Medical Association, 89 (9), 601-605.

Houston, L. N. (1980). Predicting academic achievement among specially admitted Black female college students. Educational and Psychological Measurement, 40, 1189-1195.

Milem, J. F., & Berger, J. B. (1997). A modified model of college student persistence: Exploring the relationship between Astin’s theory of involvement and Tinto’s theory of student departure. Journal of College Student Development, 38, 387-400.

Nisbet, J., Ruble, V. E., & Schurr, K. T. (1982). Predictors of academic success with high-risk college students. Journal of College Student Personnel, 23, 227-235.

Pascarella, E. T., Terenzini, P., & Wolf, L. (1986). Orientation to college and freshman year persistence/withdrawal decisions. Journal of Higher Education, 57, 155-175.

Sedlacek, W. E. (1991). Using noncognitive variables in advising nontraditional students. The Journal of the National Academic Advising Association, 11, 75-82.

Sedlacek, W. E. (1996). Employing noncognitive variables in admitting students of color. In I. H. Johnson, & A. J. Ottens (Eds.). Leveling the playing field: Promoting academic success for students of color. (New Directions for Student Services No. 74, pp. 79-91). San Francisco: Jossey- Bass.

Sedlacek, W. E. (1998). Multiple choices for standardized tests. Priorities, 10, Winter, 1-15.

Sedlacek, W. E., & Adams-Gaston, J. (1992). Predicting the academic success of student-athletes using SAT and noncognitive variables. Journal of Counseling and Development, 70 (6), 24-27.

Stanley, J. C. (1971). Predicting college success of the educationally disadvantaged Science, 171, 640-647.

Sternberg, R. J. (1985). Beyond I.Q. London: Cambridge.

Sternberg, R. J. (1993). The concept of intelligence and its role in lifelong learning and success. American Psychologist, 52 (10), 1030-1037.

Ting, S. R. (1997a). Estimating academic success in the First year of college for specially admitted White students: A model of combining cognitive and psychosocial predictors. Journal of College Student Development, 38, 401-409.

Ting, S. R. (1997b). Excellence-Commitment-and-Effective-Learning Group (ExCEL): An intervention program for academically high-risk students. The National Academic Advising Association (NACADA) Journal, 17 (2), Fall, 48-51.

Ting, S. R. (1998). Predicting first-year grades and retention of college students of first-generation and low-income families. Journal of College Admissions, 158, Winter, 14-23.

Ting, S. R., Grant, S., & Plenert, S. L. (in press). Excellence-Commitment-and-Effective-Learning Group (ExCEL): An intervention Approach for First-year College Students’ Success. Journal of College Student Development.

Ting, S. R. & Robinson, T. L. (1998). First-year academic success: A prediction combining cognitive and psychosocial variables for White and African American students. Journal of College Student Development, 39, 599-610.

Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). Chicago: University of Chicago Press.

Tracey, T. J., & Sedlacek, W. E. (1984). Noncognitive variables in predicting academic success by race. Measurement and Evaluation in Guidance, 16, 171-178.

Tracey, T. J., & Sedlacek, W. E. (1985). The relationship of noncognitive variables to academic success: A longitudinal comparison by race. Journal of College Student Personnel, 26, 410.

Tracey, T. J., & Sedlacek, W. E (1987). Predicting college graduation using noncognitive variables by race. Measurement and Evaluation in Counseling and Development, 19, 177-184.

Tracey, T. J., & Sedlacek, W. E. (1989). Factor structure of the noncognitive questionnaire--Revised across samples of Black and White college students. Educational and Psychological Measurement, 49, 637-648.

White, T. J., & Sedlacek, W. E. (1986). Noncognitive predictors: Grades and retention of specially admitted students. Journal of College Admissions, 3, Spring, 20-23.