Photo by Lawrence Berkeley National Lab, Roy Kaltschmidt, CC BY-NC-ND 2.0
Prof. Jesse Rothstein, July 11, 2017
In the admissions cycle that began in November 2016, UC Berkeley carried out the second year of a pilot experiment with letters of recommendation. Unlike other highly selective universities, Berkeley has never previously asked applicants to submit letters from teachers and guidance counselors. This may limit the information available for use in holistic review, and some at Berkeley think that as the university gets more selective it is getting harder to make informed decisions with the evidence available. Others, however, are concerned that students from disadvantaged backgrounds may not have access to adults who can write strong letters, and that the use of letters will further disadvantage these students.
In the pilot experiment, a subset of applicants was invited to submit letters of recommendation if they wished. Any submitted letters were incorporated into the “second read” evaluations of their applications. I evaluate the impact of this on the outcomes of applicants from four groups underrepresented among successful applicants to Berkeley: students from families with low incomes, students whose parents did not attend college, students from low-scoring high schools, and students from underrepresented racial and ethnic groups. I use a variety of methods, including a within-subject design that compares application scores when readers had access to letters with scores from a parallel process that suppressed the letters and a regression discontinuity design that exploits sharp distinctions between otherwise similar students in the selection of students to be invited to submit letters.
My study yields three primary results:
1) Many applicants from underrepresented groups did not submit letters when invited to do so. This was concentrated among applicants whose chances of admission were very low in any case, and was much more common in 2016 than in 2015. I have not been able to uncover the reason for the change over the previous year. The impact might have been to reduce the share of admitted students from underrepresented groups by no more than two percentage points (or five percent), and likely by much less.
2) Readers able to review letters of recommendation gave higher scores, on average, to applications than did other readers scoring the same applications without access to letters. The impact was larger for applicants from underrepresented groups than for others, and was concentrated among those with moderate likelihoods of admission for whom a small boost might have been quite important. This implies that consideration of letters raised the share of underrepresented students among admissions from those who submitted letters. The effect was to raise the underrepresented share of all admitted students by as much as four percentage points. This is about twice as large as the negative potential effect of some students’ failure to submit letters, implying a positive net effect of the use of letters on the underrepresented share of admitted students.
3) An evaluation of the overall impact of an invitation to submit letters that exploits a comparison between students who just crossed the threshold for an invitation and those who just missed it is inconclusive but also suggests a positive effect of the invitation on the relative enrollment of students from underrepresented groups.
There is support here both for the concerns of those who worry that disadvantaged students will not be able to obtain letters and for the hopes of those who believe that additional information will allow for fairer decisions. Although the evidence is not fully conclusive, I interpret the weight of the evidence to indicate that the use of letters did not reduce diversity and likely increased it. Nevertheless, any effect is small, and would be easily swamped by other policy decisions. One path forward might be to take advantage of the information that can be provided by letters while using other tools (e.g., expanded outreach to high schools) to ensure that there is no net negative effect on diversity.
This report was prepared quickly to support policy decisions being made in Summer 2017, and focuses exclusively on the impact of letters of recommendation on the demographic composition of the admitted class. A full evaluation of the use of letters depends as well on whether they provide useful information to support better admissions decisions within demographic groups. In future work, I plan to investigate this question by examining the performance at Berkeley of students admitted under the letters of recommendation pilot, and by testing whether there are particular profiles of students or of letters that are associated with greater impacts on admissions decisions or on the quality of these decisions.