Tuesday, June 19, 2007

How useful is SAT, really?


A major study released Monday by the University of California suggests that high school grades may be good at predicting not only first-year college performance, as commonly believed, but performance throughout four undergraduate years. The same study suggests that the SAT adds little predictive value to admissions decisions and is hindered by a high link between SAT scores and socioeconomic status — a link not present for high school grades.

And further, the study finds that all of the information admissions officers currently have is of limited value, and accounts for only 30 percent of the grade variance in colleges — leaving 70 percent of the variance unexplained.

Taken together, the study questions many assumptions widely held in admissions. And while the last year has seen numerous studies on the impact of standardized testing in admissions (with a range of conclusions), the new study is from Saul Geiser and Maria Veronica Santelices through the University of California at Berkeley’s Center for Studies in Higher Education, and is based on data from all University of California campuses. Past studies by the center have been influential in the evolving debate over admissions standards — and anything involving the University of California tends to get attention, given the system’s influence and top campuses.

There is more in this report by Scott Jaschik in Inside HigherEd. It's quite amazing that nearly 70 percent of the variability in the students' performance remains unexplained!

While it is not of the same scope as the systematic, large scale study above, this study from IIT-M also found a good correlation between students' B.Tech CGPA and their marks in Classes X and XII; more importantly, it found very little correlation between the students' CGPA and their entrance exam rank. Some of you may recall that this study played a key role in our recent discussion on anti-women bias in JEE.

3 Comments:

  1. Blue said...

    Having taken the SAT, let me be the first to say that it is not useful. ^__^

    Perhaps it measures a certain type of problem-solving ability, but I would agree that it does not predict college success.

    Particularly when so many students take the test after their parents force them through training, tutoring, etc. Despite any initial marker of intelligence, college success is largely dependent on how students manage to motivate and apply themselves on their own, which the SAT does not measure.

  2. Ritwik said...

    Abi,

    There is one key difference between the inferences that can be drawn from the US study and the IIT study.The SAT study suggests that there is inherently something incomplete with the examination, because it is an examination which gives out a particular score, with no cut-offs as such, unlike JEE which is a qualifying examination. Thus, what the IIT study hints at is this - given that two students are good enough to clear the JEE (conditional probablity), there are higher chances that the one who performed better in the boards will outperform, in college, the one who scored better in the JEE. It will be incorrect to extend this inference to mean that board results are a better predictor of engineering success at the IITs in general (absolute probability), and that the board results would be a good enough replacement for the JEE as an admission criterion.

    When you're comparing people who have cleared the JEE, you're basically comparing people who would have all easily scored 1450+ on the SAT, with some coaching for the Verbal section. Inferring from a study on such a group is quite different from inferring from a group of average University of California students, who would have had a SAT score of around 1200, if I'm not mistaken.

    There is another reason that SAT scores may be highly correlated with socio-economic status. I think it may be due to the ridiculously tough and quixotic verbal section, as opposed to the ridiculously easy and equally quixotic math section. If instead of overall SAT scores, a comparison of performances in the two individual sections is done across socio-economic segments, I have a feeling that my guess may be validated.

  3. Pratik . said...

    Why doesnt someone talk of correlation between the nature of board exams, IIT JEE and undergrad?

    There is a strong correlation between the nature of the board exam and undergrad exams - both tend to test whether students know the basics of the courses they read, rather than how deep they can think in applying those basics in solving problems. JEE is a competitive exam whereas the others are qualifying exams.

    JEE, on the other hand, is more about problem solving than simply knowing the definitions. One can know a myriad concepts; but it wont hold them in good stead in JEE unless they can actually apply them. The implication is, you can read much less stuff and get away with it, as long as you understand the stuff and can apply it.

    So, if one studies mechanics and electromagnetism, in depth and practice hard on related problems, rather than reading the full syllabus, he is likely to do much better in the JEE physics test than a student who goes over the entire syllabus but doesnt study any of the topic in so much depth. The tables will be turned in case of the board exams.

    Then of course, there is the question of bunching of ranks around a given marks. Correlations between ranks and GPA doesnt take this into account. If you are not going to take that into account, I suggest you choose a 100 students between 90.1 and 90.3% in the board exams and rank them. Then correlate those ranks based on the board exam scores and the undergrad performance. I bet the correlations would be nowhere as good as the ones you find right now.

    One can debate which of the two types of exam is more desirable. Each have their advantages and drawbacks. Without going into all that, suffice it to say that trying to find corrrelations between JEE and other types of exam is the same as correlating palak paneer with kadhai chicken.

    Incidentally, I would rather eat them both than bother about correlating them. ;-)