How good are entrance exams in 'discovering' merit

When asked to define intelligence, some people simply throw up their hands, and define it as 'what intelligence tests measure'! It's not too far off the mark to say that merit is 'what merit tests measure'. Perhaps this is a good time to look at how it is measured in India, and examine the kinds of anomalies they produce (with a particular emphasis on JEE, the Joint Entrance Exam, conducted by the IITs).

Entrance exams produce raw scores and/or ranks (or, percentiles); the implication, of course, is that someone with a higher score (or, better rank) is more more 'meritorious'. Thus, rank-ordering is a key element of these exams.

Consistency across exams: The first problem, of course, is that different entrance exams, with different designs, could lead to completely different rank ordering of the same set of students.

Consistency within the same exam: Our exams produce noisy, unreliable results. And, not much effort has gone into making them less noisy.

The use of a small number of problems (several hundreds, at the maximum) to rank-order hundreds of thousands of students. In such exams, we all know that even a single mistake can set your rank back; how badly you are affected by a single mistake depends on which part of the distribution you are at. On a Bell curve, if your score is high, a single mistake may not affect your rank much [because you occupy those rarefied heights where only a few humans survive]. It could affect you seriously if you are around the average, where tons of people are bunched up.

This problem is much worse in JEE, because even the ones who get through (i.e., get a JEE rank, called the All India Rank - AIR) are people who are able to attempt only a small fraction of the questions. In the year I took it, I attempted barely 25% of the questions in chemistry, as well as in math (physics was slightly better, at about 50%!). The JEE questions continue to be brutal.

What about standardized tests? GRE uses questions that have been tested for difficulty on a sample population, and are weighted accordingly. Also, GRE can be taken multiple times, so it gives you a chance to find ('discover') your correct ('optimum'? 'best'?) level in a second or third chance, in case you feel that you messed up your first chance. In our country, BITSAT (conducted by BITS, Pilani) appears to be a standardized test. Thus, 'your station in life', as measured by these tests, is far more reliable and reproducible.

How about JEE? AIEEE? State-level CETs? Since they are offered only once a year, they fail the test of being 'available through the year'. Is any difficulty-weighting done? Not in AIEEE and CETs; as for JEE, I don't know. [Someone who knows about this may perhaps clarify this point.]

Ability to predict academic performance: Just how seriously should we take this 'ranking through a single exam'? One good way to answer it would be to check how well a student's rank predicts his/her academic performance in college. To my knowledge, there is no large scale study, tracking students' progress through their degree courses [worth a minor rant]. The only study that I know of appears to be a small one; I am sure the devil is in the details [which we don't have access to], but its broad conclusions are quite instructive:

‘There is a strong correlation between the marks of Classes X, XII and the CGPA during B Tech. The correlation factor is close to 1.’’ This means, the chances of a good student in school doing well in B Tech is almost 100 per cent.

‘There is little correlation between AIR and CGPA.’’ This means, toppers in the JEE are not at the top during their B Tech programme.

Built-in bias: If you want to do even half-way well in JEE, you need pretty intensive coaching, which is (a) quite expensive, and (b) available in large cities (or, places -- such as Kota, Rajasthan -- that are dedicated to this enterprise). Thus, it has a built-in bias against (a) poor students who cannot afford coaching, and (b) against students in rural and semi-urban areas.

Again, it's important to have data, which is locked up with IITs. How many of IITs' students come from an urban (middle and upper-middle class) background? How many of them go through coaching classes? For how many years? Only IITs can answer these questions.

Record of merit discovery across the entire population: State level CETs and AIEEE have the mandate of testing students on what is covered in their regular curriculum (AIEEE has the reputation of being 'tougher' than CETs). JEE, on the other hand, has been "getting increasingly closer to the syllabus of science graduate programmes, making aspirants more and more dependent on coaching classes" [quote from the same IE report].

The mandate of CETs and AIEEE gives them a much better chance of 'discovering' merit among groups that find themselves underrepresented in JEE: girls, OBCs, SC/ST and so on. More and more of them make it through open competition (general category) in the professional courses in state universities, while the percentage of women in IITs, for example, is still stuck in single digits.

* * *

IMO, JEE is not a good way of measuring merit because (a) its results are noisy and unreliable, (b) it requires the kind of acquaintance with advanced problems that can be acquired only through intensive coaching, and (c) it does a poor job of 'discovering' merit among huge fraction of our population.

Having said that, the alternatives are also not all that great. State-level CETs, for example, go to the other extreme by being not intellectually challenging. At the moment, AIEEE seems to occupy the middle ground; but it, too, shares with JEE the problem 'merit discovery through a single exam'. A different 'middle ground' is the norm in some states (such as Tamil Nadu) that use both CET and board exam scores are used for assessing merit.

Currently, we seem fated to live with these defective and sub-optimal choices.

* * *

I don't know about China, but Japan and South Korea (and perhaps other Asian countries, too?) seem to have the entrance-exam culture that makes their students learn complex, advanced topics before they enter college. Among the European countries, I know France uses entrance exams for its elite 'Grands Ecoles'. In the US, SAT is the exam taken by all the students, and to my knowledge, it's a standardizing exam in the mould of GRE. Moreover, SAT scores are only one of many factors considered by the US universities in their admissions process.

Which method of merit discovery should we be using (and make an effort to move towards)? Why?