The COVID-19 pandemic may have fundamentally altered many aspects of education. One less appreciated area of focus is on how the college admissions process has been affected. accurate books by Jeffrey Selingo on who gets in and why, Ron Lieber on the price you pay, and Colin Diver on the influence of the ranking industry, all provide current overviews of the landscape, economics, psychology, and politics of higher education, at least up to the point when the pandemic entered our lives. However, what is really needed is solid research on how the pandemic might have impacted higher education and the admissions process. For example, whether and why students decide to submit their SAT or ACT scores to colleges when the pandemic pushed colleges to adopt test-optional admissions policies is important to understand.

Fortunately, many different groups have started initiatives to tackle these issues. One group I’m leading is the education working group of the Association for Psychological Science global collaboration on COVID-19, where we recently had an academic rockstar panel discussing the role of psychological science in addressing the case of COVID-19 and the college admissions process. In addition, a special issue from Perspectives on Psychological Science examining graduate school admissions was published along with expert commentary.

Now, a new report by the College Board has just been released which addresses SAT score submission decisions by students and other important ways that the pandemic has impacted the higher education landscape. This report, titled “New evidence on accurate changes in college applications, admissions, and enrollments: Focus on the Fall 2021 admissions cycle,” is led by economist and College Board’s vice president of research, Jessica Howell. The report is the first of many that will come from the College Board Admissions Research Consortium, of which I am a member of the interdisciplinary research advisory committee. The purpose of the research consortium is as follows:

“Colleges are eager to understand how the impacts of the pandemic and widespread test optional admissions policies are shaping who is applying to their college, and how these impacts might continue to affect college grades and success in the future. Together with our members, College Board has kicked off the Admissions Research Consortium (ARC), which aims to help colleges do exactly that. ARC is a multiyear, collaborative research initiative with 80 participating colleges and associations, including the Association for Institutional Research (AIR), American Association of Collegiate Registrars and Admissions Officers (AACRAO), National Student Clearinghouse (NSC), and College Board. ARC is also guided by a Research Advisory Committee composed of academic and institutional researchers.”

This new report is focused on students in the high school graduating class of 2021 who would have been applying to college in the fall of 2020 for fall 2021 college entry. In fall 2020, when the pandemic was still in its infancy, many high schools were fully remote, students couldn’t go on campus tours or visit colleges, college guidance might have been missing or disrupted, vaccines were not yet available, and it was unclear if colleges would be open or virtual in the future. On top of all that, colleges’ test optional policies had just been announced. As a result, the changes observed in the fall 2021 application cycle, relative to prior years, capture the combined impact of all these factors on students and colleges.

I had the opportunity to ask Jessica Howell some questions about this new landmark report, which follows. The three main findings from this research brief address general shifts in student and college behavior, whether and how students chose to submit their SAT scores, and whether test-optional policies changed student body diversity. All these syllabus are addressed by data provided by the colleges and universities included in the consortium study.

How did the students and colleges change their behavior during the pandemic and the first year of widespread test-optional college admissions?

Compared to the fall 2020 application cycle, which largely took place before the COVID-19 pandemic, the colleges in our study received more applications, admitted more students, and increased enrollments in the fall of 2021.

There were some colleges that were test optional before the pandemic, but when COVID forced test centers to close, virtually all colleges had to impose a policy that didn’t penalize students who were unable to take the SAT. With the widespread adoption of these test-optional admissions policies and less certainty around how college admissions processes would work compared to prior years, it is perhaps not surprising that nearly all institutions participating in the study received more applications than the prior year, although application growth under test-optional admissions is documented in some research but not all studies of pre-pandemic test-optional admissions. Institutions experienced a nearly 18% growth in applications that ranged from 10% application growth at selective public four-year institutions to 30% application growth at very selective private four-year institutions.

Many colleges in our study reported greater uncertainty about how many students they admitted would end up enrolling on their campuses, so institutions generally admitted more students (6% growth) for fall 2021 than in the prior year. Students did enroll, often at greater rates than colleges expected, so enrollment at these colleges grew by nearly 8% in the fall of 2021. The very selective private institutions in our sample (with admission rates below 25%) are the exception to these admission and enrollment patterns, admitting 12% fewer applicants than the prior year and, thus, keeping enrollment nearly flat in fall 2021 relative to fall 2020.

How did students navigate the choice about whether or not to submit their SAT or ACT scores on their college application?

Fall 2021 applicants to colleges in our study fall into three categories regarding test submission in a test-optional environment: (1) roughly 50% had an SAT or ACT score and chose to submit it to colleges, (2) nearly 30% had an SAT score that they chose not to submit to colleges, and (3) about 20% did not have an SAT score or had an ACT score that they chose not to submit to colleges.

Among those with a test score and a choice to make about whether or not to share it on their college applications, we find that the biggest driver of the decision is whether their score is high or low relative to typical scores at the college to which they’re applying. Applicants whose scores are relatively high are very likely to submit their scores, while applicants whose scores are relatively low are not very likely to submit their scores. Companion research based on student surveys and focus groups also revealed that relative test scores were central to how students make this decision. Very few other factors appear to influence students’ test score submission decision. We find some evidence that students with lower high school grades are more likely to submit their test scores (presumably to bolster their academic record). Conditional on test scores and high school grades, we find almost no difference in test score submission decisions across students with different demographics like parental education, race/ethnicity, and so on. Differences in score submission patterns by race, parental education, and income documented in previous research are attributable to differences in academic achievement among score submitters and non-submitters. Once you control for test scores and the college to which a student is applying, students with different demographic attributes have nearly the same probability of submitting their test score in a test-optional environment.

Did test-optional change diversity at these colleges?

Prior research on test-optional college admissions finds either no change in racial and socioeconomic diversity or small changes. In our new research, we find that, because the enrollment of all subgroups of students increased between fall 2020 and fall 2021, the proportional representation of student subpopulations—by race and socioeconomic status—changed very little at these institutions. Black, Hispanic, and Native students made up about 25% of college enrollees in the sample before and after the pandemic and test-optional admissions policies were in place. Similarly, students from disadvantaged schools and communities made up the same proportion of college enrollees before and after. So, overall, we don’t find a change in trend to the racial/ethnic representation of students in this first year of near universal test-optional policies. The exception to this pattern is among the very selective private four-year colleges, who saw modest gains (about 3 percentage point growth) in both racial and socioeconomic diversity in fall 2021 relative to the prior year. These enrollment patterns in racial and socioeconomic diversity are consistent with separate analyses based on a near-universal set of U.S. four-year institutions, so these results are generalizable beyond this study sample.

It is important to note that 2021 was not a “normal” time with a handful of colleges changing their policies as in the prior research. In this case, test-optional policies were certainly not the only change that students and colleges were navigating, so we have to attribute the changes we document in this new research to the many pandemic-related disruptions that were also occurring simultaneously (e.g., a global health crisis, a domestic economic crisis, remote learning and learning losses, and mental health challenges) in addition to nearly all U.S. colleges and universities changing to a temporary test-optional policy.

Conclusions and future research

In sum, this first brief demonstrates that Admissions Research Consortium colleges have more/different students enrolled than they’ve historically served, so it will be interesting to see if there are differences in student performance, retention, and longer run outcomes like major choice and degree completion. Jessica Howell lets me know that this is what is up next in the research pipeline, and this consortium is ongoing and will publish more research on student outcomes as the group continues to track fall 2021 enrollees and study the fall 2022 cycle, serving as a dynamic process for both students and institutions, hopefully informing policy decisions in that process with solid evidence.

Regarding how students decide whether they submit their SAT or ACT scores to colleges, one fascinating finding from this report is that it aligns with prior research conducted by Howard Wainer who was also able to look up the scores of students who chose not to submit their scores under a test-optional admissions policy. It affirms that lower-scoring students tend not to submit scores and suggests that we’ll likely see lower college performance among students who didn’t submit their scores than among those who did. However, at this point, whether these findings will replicate the Wainer study is an open empirical question.

Why don't you read the doughnut question and tell me what your opinions are....

This is one of the most famous SAT questions. It's called the doughnut question. And it shows you a doughnut. In the figure above, what is the greatest number of non overlapping regions into which the shaded region can be divided with exactly two straight lines? And people use this--we used this at the kick off news conference that launched Fair Test way back in 1985. And David Owen, who wrote the book None of the Above, showed this to the assembled 40 or 50 press people in the room--who ooed and awed at it. Fifteen years ago, people hadn't seen this. And how simply you solve it, if you have been well- coached.

You read it, what is the greatest number? And you see that, well, this is the last math item in this section. Therefore, it's a very difficult question. You know from looking at the SAT that on every SAT section, exception for memorizing comprehension, items go from the easiest to the hardest. So at the beginning of the section, an obvious answer is right. At the end of a section, an obvious answer is a distracter. It's wrong. You never go for it.

And what coaching courses like the Princeton Review and others teach you, is that you can analyze--without knowing how to solve the problem, you can have a very good chance of getting the right answer. You look at it and say, what is the greatest number? Well the greatest number in the answer's six. That's wrong. That's the distracter answer. It's designed to take somebody who doesn't understand it and make him guess wrong. But you look at it and say, well, the simplest thing I can do is I can divide it into four pieces by cutting vertically and cutting horizontally. But that's too easy too. That's got to be wrong. But I could get four. So, I must be able to get three or two because they're smaller than four. And six is wrong because it's the easy answer. The answer's five. That is the right answer.

I now know how to draw the lines so you can do it. You draw them so they intersect in the middle, inside of the doughnut. So you get two very small triangles and some large ones. But you don't have to know how to solve that problem. And it has nothing to do with math. It has nothing to do with aptitude. And it most certainly has nothing to do with merit. Unless you define merit as being coached.

Can you walk us through a question from the SAT?

P, Q, R and S are four towns. P is farther north than Q and R. S is farther south than P. Q is farther south than R. Which town is the farthest south? P, Q, R, S or, it cannot be determined.

What kind of a question does it strike you as?

A, very coachable. B, not really math, certainly nothing you learn in high school and nothing that will be useful in college. Plausibly a little bit of logic, I'll accept that. But I think more, just a little trick.

It's the kind of thing they do a lot. There are three towns--A, B and C. The distance from A to B is four. The distance from B to C is three. What's the distance from A to C? And of course if you're kind of linear, you would say, well it's seven. It's the two together. But of course, they don't have to be in order. So it could be one. Does it measure what you learned in high school? No. Is that useful in college? No. It's a little trick thing. You'll get it wrong once. Someone will explain it. And then you'll be ready for it.

Why develop a question like that if it's only measuring a test-taking technique?

Because the point of writing an SAT question is not measuring what you learn in high school or how well you'll do in college. It's separating out kids. There are kids who will get that right. And they're generally the kids who have been in math courses where they play with this kind of stuff. Which is to say, upper income. And there are kids who will get it wrong because they don't play with this stuff.

So the question is very good at separating kids. And that's why they have it here.

Another question..

Money is to bank--a little socioeconomic problem here, but let's go with it--money is to bank, as food is to basket, park is to city, cash is to store, book is library and article is to magazine.

So what you're supposed to do is say, I put money in a bank. And I guess money is kept in a bank in the same way that a book is kept in a library. Maybe it's a vocab question. Do you know the word bank or the word money. I can't believe that it's a logic question. And I can't believe the kid who gets that right is going to be better in college than the kid who gets it wrong. Especially since there are some wrong answers that are pretty attractive here. You know, you sort of think of banks and money. And there's an answer with cash. And you sort of--you know, you're stressed. This is important. You're in a rush. You're in the middle of the SAT, which is a pretty important test. It's nine in the morning on a Saturday. You're probably hung over. And a lot of kids who get that wrong--it's not because they don't speak English. And it's not because they won't do well in college.

Our experience is, a kid who doesn't do well on the SAT--it's not because he gets the toughest questions wrong. It's because they make lots of careless mistakes on easy questions. They get sucked into trap answers a lot. Because they don't have their footing. They don't understand the question. It's not that they can't do it.

So a lot of the course isn't focusing on the toughest questions. It's focusing on making sure you don't get that question wrong.

The doughnut question....

You got a doughnut here. And the question reads: "In the figure above, what's the greatest number of non-overlapping regions into which the shaded region--the doughnut--can be cut with two straight lines? In other words, how many pieces can you cut the doughnut into with two straight lines?"

This is the last question on the test. It's Saturday morning. You're very stressed. You're very tired. And this is really important. So what Joe Bloggs does is, he'll just cross lines. Right? The easy answer there is four. I can make four pieces pretty easily. What are the odds that on the toughest question on the SAT that you've done enough work? Right? They're zero. No way.

And again, a good test-taker's sitting there. And he answers four and goes, god, there must be tougher than this. And he's right. So you cancel four. And of course, since you're able to get four, you cancel three and two also. That's not the greatest number you're able to cut it into. You're able to cut it into at least four. The answer's got to be five or six. And then you sit back for a second and you say, what else would Joe Bloggs do? He'd say, well, they want the greatest number possible. So maybe it's six. Maybe it's the greatest number. So it's not that either. The answer's got to be five.

On the one hand you might way, well that's kind of a goofy way to take the test. That's all testmanship. On the other hand you might say, what is this question telling you about a kids math skill or his ability to do college level work? This is just a good question. And goofy questions deserve goofy preparation.

Here's another one. There are 3 roads from Plattsville to Ocean Heights. And 4 roads from Ocean Heights to Bay Cove. If Martina drives from Plattsville to Bay Cove and back, passes through Ocean Heights in both directions and does travel any road twice, how many different routes for the trip are possible? 72, 36, 24, 18 and 12?

Like, what is this telling you about your son. Like, is it telling you he's stupid that he got it wrong? Is it telling you he shouldn't go to college or he should? What is it telling you? And I would claim it tells you almost nothing. A great question for Games Magazine but a lousy question for Harvard.

Could you look at the doughnut question and tell me what that question is measuring?

In the figure above, what is the greatest number of non-overlapping regions into which the shaded region can be divided with exactly two straight lines? And it gives you four choices.

What it's trying to do is measure mathematics reasoning, spatial relations and one's ability to conceptualize.

Anything else? Is that a true test of how one will perform in college? Should an admissions officer use that information in choosing between candidates?

Well, hopefully no one will base any decisions on a single item on a single test administered on a Saturday. That's why our tests have about 60 mathematics items and 70 verbal items. So--no, an admissions officer should not base any decision on a single item. And what we try to do it state further than no one should base any decision on a single test score. They should use a lot of information.

But what we're trying to do is, we're trying not to recapitulate or repeat the types of computational basic skills, items in terms of addition or subtraction or dividing. What that type of an item and many others on our test are trying to do is, going beyond simple computation. Look at critical thinking skills and reasoning ability. Some of it's visual and perceptual. And others are simply mathematics and the ability to look at analogies or look at memorizing comprehension's and draw meaning from the text.

Are you saying that it's basically not fair given the differences in society?

I'm saying, given the differences across different groups in every way we want to measure it, whether we measure it with the SAT, with the courses taken, with high school grades, college grades or graduation--those differences exist and they exist in any measure that we have including the National Assessment of Education Progress at fourth grade and beyond.

And so when you're prohibited from looking at race and ethnicity--when those differences are so much a part of educational opportunity, of opportunity to learn and of quality of education and teaching--it really is a crime.

What the crime is, is not that the SAT is used for admissions. The crime is that courses like the advanced placement program, qualified teachers, the opportunity to be engaged in more rigorous courses and be expected to perform at higher levels--that opportunity is not uniform across all schools and across all communities, irrespective of where one lives.

BOISE (IdahoEdNews.org) — The percentage of Idaho juniors who met both benchmark scores on the SAT – a measure of college and career readiness – slid for the second year in a row, according to data the State Department of Education released on Tuesday. 

In 2022, 28.3% of juniors met both benchmarks on the SAT (for math and evidence-based memorizing and writing) – 0.8% lower than in 2021 and 3.7% lower than in 2020. Even amid pandemic-related school shutdowns in 2020, 90% of Idaho juniors still took the SAT. 

The benchmark percentages in math have fallen over the past two years as well. This year, 30.5% of students met the benchmark, 0.7% fewer than in 2021 and 2.5% fewer than in 2020. 

Benchmark percentages for evidence-based memorizing and writing showed a 0.1% increase from 2021. Scores from the last two years, however, were both more than 4% below the 58% of students who met the benchmark in 2020. 

The SDE characterized the new SAT scores as “holding steady” from the previous year in a Tuesday press release.

Sherri Ybarra, superintendent of public instruction, pointed out that many students are no longer required — by the state or by their university — to take college entrance exams.

“We’ve known for some time that many Idaho students do not prepare for the SAT because it is not essential to their post-high school plans,” Ybarra said. “And the priority placed on the test has declined further in accurate years as universities in Idaho and elsewhere have stopped requiring college-entrance exams, and the State Board of Education dropped the exams as a graduation requirement this year.”

How Idaho schools and districts fared

There were some bright spots in the SDE’s report. Couer d’Alene Charter Academy had the highest rate in the state, just as it did in 2021, with nearly 90% of its students meeting both benchmarks. The following schools also had notably high percentages of students who met both benchmarks:

• Renaissance High (West Ada)        77%
• North Idaho Stem Charter             >76%
• Meridian Tech. Charter                68%
• Meridian Medical Arts Charter      67%

Among all Idaho districts and charter schools with at least 20 participating students, only 11 had 50 percent or more of their students meet both benchmarks.

Here are the percentages of juniors meeting both the English and math benchmark scores for select large districts:

• Boise: 41.5%
• Bonneville: 30.5%
• Caldwell: 11.3%
• Cassia County: 21.8%
• Coeur d’Alene: 34.2%
• Emmett: 27.8%
• Idaho Falls: 26.5%
• Jefferson County: 30.6%
• Jerome: 9.9%
• Kuna: 19.5%
• Lakeland: 29.1%
• Lewiston: 24.8%
• Madison: 32.8%
• Middleton: 31%
• Moscow: 49.7%
• Nampa: 17.3%
• Oneida County: 28.6%
• Pocatello-Chubbuck: 32%
• Post Falls: 30.7%
• Twin Falls: 24.9%
• Vallivue: 20.3%
• West Ada: 40.4%

The College Board establishes benchmarks “to help students and educators assess student progress toward college readiness from year to year.” According to its website, students who meet or exceed the benchmarks have a 75% chance of earning at least a C in a variety of general education first-semester college courses. It also notes that “students scoring below the SAT benchmarks can still be successful in college, especially with additional preparation and perseverance.”

Education leaders have been questioning the validity of college entrance exams, especially since the start of the pandemic.

Idaho Education News filed a public records request for the 2022 SAT data, but the SDE provided only partial information on Tuesday; raw SAT scores were not released. EdNews has requested that data and will report on it when it becomes available. 

Idaho Education News data analyst Randy Schrader contributed to this report.

This article was originally posted on IdahoEdNews.org on July 26, 2022.