When it Comes to Students, What Are We Measuring?

Educators want to help students, and oftentimes that involves measuring them. The numbers stem from important questions: How much are they learning? How is their learning impacted by various educational methods and policies? But before answering those types of questions, says Lily An, we first have to ask another complex question: When it comes to understanding students, how do we best measure them to begin with?

“At the end of the day, we're all trying to help students and support student learning and student outcomes,” says An, a Ph.D. candidate in HGSE’s Education Policy and Program Evaluation concentration. “So we owe it to them to keep looking and keep digging and keep trying to improve our practice.”

An came to HGSE after a career in finance, where she worked in equity research and evaluated whether companies would meet their earnings projections. She had always wanted a career in education, though, and joined HGSE’s master’s program. Learning about psychometrics in Professor Andrew Ho’s Statistical and Psychometric Methods for Educational Measurement class, however, opened up a new area of research for An. Psychometrics applies psychology and statistics to develop measures for assessing attributes such as personality traits and attitudes. In education, it’s used to study how student abilities are measured.

“I majored in statistics in college, so I thought, ‘Wow, this actually combines all my interests: understanding education and statistics and value,’” says An. “And the way that we do that a lot of times in education is through testing, so I think it's really important to be thinking critically about what we are valuing. What are we measuring?”

As her research interests became more defined, An began to focus on how organizations decide what and how to measure their students, and what impact those educational policies have on student outcomes.

“I switched to a more expansive, external, and organizational view,” she says. “There are all these policies and institutions and systems that are affecting access and what people can do.”

After graduating from HGSE in 2018 with her master’s, she returned in 2020 to pursue her Ph.D. in education. With advisers Ho and Associate Professor Luke Miratrix, she’s currently wrapping up her dissertation and is expected to graduate in May 2025 having spent the last four-plus years focused on educational measurement, causal inference methods, and school accountability policies.

“The question I'm always asking is ‘How can educational policies and the methods of evaluating those policies be more attentive to the measures that are used?’” says An. “We know that measures are important. We know that evaluation is important, but they're not always talking to each other. So where can we highlight those opportunities and improve them?”

An has worked with a variety of different partners in her academic career, including with Kentucky when the state developed different classifications of school quality levels. She also spent the two years between her master’s and Ph.D. programs at Brown University as a research analyst working for Annenberg Institute Director John Papay, Ed.M.’05, Ed.D.’11. The research, studying the impact of high-stakes testing exit examinations like the 10th-grade Massachusetts Comprehensive Assessment System (MCAS), was an illuminating example, she says, of walking the line between a partner’s needs and the importance of scholarly practice.

An’s work with Papay was initially published in 2018, but a presentation of more recent work on the subject at an Askwith Education Forum/PIER Public Seminar on the MCAS ballot measure by Papay showed the impact that research has as it evolves.

“I was like, ‘I remember that graph,’” An recalls of Papay’s presentation last October. “‘I originally made that graph!’”

An credits her work at Annenberg with opening up another key part of her HGSE journey: a Partnering in Education Research (PIER) fellowship.

“That experience really motivated me to apply to PIER because I saw how John was really attentive to what Massachusetts wanted to understand, while also thinking about the issues from the research angle,” she says. “And I [thought],‘I want that. I want to be that fluent in both languages and all these spaces.’”

That fluency came through a number of different experiences in the PIER program. The two-year Institute of Education Sciences fellowship, for example, allowed her to work with the state of North Carolina in their Standards, Accountability, and Research Division, an “invaluable” experience that deepened her appreciation for the work states are doing to better understand the needs of their students and how good policy can follow data.

“The PIER program teaches you how to interact with data partners and how to balance your partner’s needs with your diverse research interests as a researcher,” An explained. An also credited her advisers and Senior Lecturer Carrie Conaway, former chief strategy and research officer for the Massachusetts Department of Elementary and Secondary Education, for supporting her throughout her time at HGSE.

“I feel so lucky to be in this Ph.D. program, to have had the advisers and the faculty mentorship and the experiences that I've had,” says An. “I have had lots of opportunities to work with states and policymakers that I don't think the average doctoral student gets.”

The lessons, weighed and measured in any number of ways, always come back to gratitude.

“I think the last four years have shown me that people in education are all really trying to do the best that they can, grappling with measures or ideas that they're unfamiliar with,” An says. “And there's just such a role for people with some expertise to weigh in and support those conversations.”