« More Useless Condescending Claptrap | Main | Pathetic Protestations from Le Moyne »
March 09, 2005
Another Study that Measures What it Measures.
I read at Number 2 Pencil about a study that Bates College recently released about their optional SAT submission policy. It compares the SAT scores, graduate rates, etc. for submitters and non-submitters. Apparently everyone is automatically taking it to mean that the SAT is not an accurate predictor for college achievement. My first thought on reading this was, “how on earth are they comparing SAT scores if the non-submitters never submitted them.” Obviously there must have some sort of optional surveying that was done. When this is the case, the problem of self-selection comes up.
I went to the Bates website and tried to find the paper, but all I could find was a press release, a powerpoint presentation of figures, and the associated notes. From looking through the figures, though, I was able to calculate that the pool of people that they used for the study were people from the class of ’92 to the class on ’93. (This runs from the time period where all standardized testing, not just the SAT, was made optional to the last class that they would be able to get graduation rates on.) This class composes 3888 students. 2487 (63%) were submitters and 1471 (37%) were non-submitters. They seem to use this pool to compare graduate rates, admission rates, GPA, academic scoring, field of study, etc.
However, the pool of people they use to compare SAT scores with these other values is 3275 (84% of the possible subjects.) 91% of the submitters participated, but only 68% of the non-submitters participated. This leaves plenty of room for the effects of self-selection. Plus, I will point out that since they're comparing the SAT values for the self-selected group to the other values for the entire group, this makes their comparisons suspect. That sort of thing is only valid if the sample is actually random.
Self-selection could affect the results in many ways. Perhaps students were more likely to participate in the study if their SAT score was higher, or perhaps students were more likely to participate in the study if their college GPA was higher, or perhaps students in non-math and sciences were more likely to participate in the study because they care less about revealing their SAT score, etc. Who knows what the results of these things were, but I can’t believe that the study doesn’t even address this at all. For example, a comparison of fields of study or GPA’s of participants and non-participants in each group would be helpful.
Update: I did some more number crunching, and it is possible to calculate that the average GPA of submitting participants is 3.21 (compared to a GPA of 3.11 of all submitters) and the average GPA of the non-submitting participants is 3.10 (compared to a GPA of 3.06 of all non-submitters). Therefore all of the participants had better GPA's than the non-participants, which means a the difference is now .11 rather than .05, which is the number that SHOULD have been compared to the SAT gap of 160. Also, I realized that since the graduation rate for both groups is about 87%, it is likely that the majority of the submitters that didn't participate, also didn't graduate. So most likely they used the stats they had on submitters who stayed for a certain length of time, but the non-submitters are all self-selected. It's also revealing to note that they do a graph of SAT scores vs. GPA for each group, and there as the SAT score increases from the 1000's to the 1500's, the GPA increases by about .5 points for both groups. Thus, in fact, it seems pretty obvious that SAT score IS a good predictor for GPA.
I found some things said in the notes quite interesting.
In this and previous studies, we asked statistical experts at Bates to check and critique our work. Michael Murray, a renowned international economist at Bates who designs national economies and central banking systems for third world countries, said to me, "Bill, you shouldn't be comparing submitters and non-submitters!" I thought, "Oh no, into what statistical blind alley is Michael leading me." He went on, "You should be comparing the enrolled non-submitters with the students you would have had to admit if you didn't have 1500 non-submitter applicants from which to choose the very best." He is right, of course, and at Bates and most colleges, that would comprise the entire wait list and a decent slice of the refuse pool.
He then goes on to not discuss this issue’s affect on the study at all. Murray seems to have made an incredibly good point here.
From 1992 to the present, 129 (about 3% of the total enrolled students) SAT I Non-submitters submitted SAT IIs. We have significant volumes of AP's, A levels, and IB's, but most of them come in late in the senior year for placement and advanced credit use, so they are not part of this research.
I think it would be a great idea if they were a part of this research. No one that takes the SAT 5 times is going to get the same score each time. (For instance, I took the ACT twice within the same year and got a 34 and a 32.) That doesn’t mean that the SAT is not a good measure of intelligence, only that there are going to be variations in many things that will show up in the score. If you have a bunch of students with the same intelligence, knowledge, and test taking skills that each take the SAT one time, you’re going to have differences between the scores, even if there’s no differences between the students. Quite often there are students who do better on the ACT than the SAT, and so submit that score, and vice versa. If we compared the scores of SAT submitters and non-submitters on other tests, we might be able to detect this effect.
Also, comparing high school GPA of submitters and non-submitters would be of interest. All of this stuff could have been done with a few more questions on the survey.
But something important should be pointed out here. Optional testing is often assumed to be a device for an affirmative action policy, to open the admissions process from a narrow statistical review to a more complex and subtle reading. And it does that. But white students using the policy outnumber the students of color by about five to one.
This is like taking a truckload of apples and a bag of oranges and saying “well, yeah, 50% more of the oranges are rotten, but the number of rotten apples outnumbers the number of rotten oranges by 50-1.” This is a person with a PhD in statistics? Either he’s outright stupid, or he’s one of those people blinded to any logic that contradicts his opinions on affirmative action. Either way, I’m not feeling real good about his study.
In California, that bell-weather state so often several years ahead of the rest of us. I refer you to Eugene Garcia's report of several years ago on Hispanic admission to the public institutions in California. The U-Cal public university admission rate for Hispanic students has been over the years less than 4%, and Hispanic students comprise 50% of the K-12 school cohort. Does this pass a common sense test of access to a public university system, to have a 4% admit rate for 50% of the school population? Does it pass a test of social ethics? I am not pointing a finger at California, but asking a common sense question about our country: are we getting the students the education they need to be competitive?
This statement kind of makes me think he’s an AA zealot. Does the guy realize that in California lots more of the kids are immigrants or children of immigrants who don’t speak English well or at all, not to mention being immersed in a culture where the dreams and aspirations of the population are very different from others.
On average, Submitters score about 90 points above the Non-Submitters in Verbal SAT, and 70 points above non-submitters in Math SAT, for a total SAT gap of 160 points. This TSAT gap has been amazingly stable for the entire history of the policy, and if there reasons for that, we cannot see them.
I’d be interested to see the Verbal and Math SAT statistics compared with the graduation rate, GPA etc. Perhaps we would find that the Verbal score is a good predictor, wheras the Math isn’t.
While testing seems to have some very basic correlation with GPAs, non-submitters seem to outperform submitters with the same SAT scores, but for both groups, the lines are pretty flat, because virtually everyone is succeeding.
This is kind of a problem. This study would be more effective at a school where there are a significant group of people who are not succeeding.
And here is the glaring exception. Bates alumni earn graduate degrees at quite high rates: about 70% of all Bates alumni will earn at least one graduate degree. At the Master's Degree level, the percentage of submitters and non-submitters are quite close. But in fields that require another standardized test for admission, there are big, visible gaps between submitters and non-submitters: MBAs, PhDs, MDs and JD's. I mean this as a honest and not a rhetorical question: are these the best, or just the best test-takers? Let that question go proxy for a lot of what we need to understand better than we do.
I like how here he posits the question of whether the people getting graduate degrees are the best, or just the takers; but in the press release, they say, “in fields where success does not depend on further standardized testing—including business executive officers and finance careers—submitters and non-submitters are equally represented.”
In conclusion, I’d like to point out that, like a standardized test, statistical studies only measure what they measure, which is not necessarily what the researcher wants them to measure.
Posted by illuminaria at March 9, 2005 02:50 PM
Comments
Harvard Business School stopped requiring the GMAT, but then after a decade or so went back to requiring it. Not requiring standardized test scores helps the BS artists at the expense of the nerds. You can go pretty far in business as a BS artist, but eventually you get found out (e.g., Bernie Ebbers).
Posted by: Steve Sailer at March 16, 2005 02:05 PM