Professor says writing test for selective admissions is misused

A self-scoring writing exercise introduced this year as part of the admissions process at five of Philadelphia’s top public high schools is being misused, an education teacher said.

Joshua Wilson, an associate professor at the University of Delaware who studies automated essay scoring, said the writing tool is intended to identify struggling learners and inform classroom instruction, not take decisions. high-stakes decisions about the future of students.

“No one has done any research as to whether it can be used to make placement decisions,” Wilson said of the writing tool, MI Write, which is a product manufactured by Measurement Inc., Based in Durham, NC Using it in this way is a “mistake,” he said.

“I think the use of Mi Write in this context is very problematic and should be reconsidered,” Wilson told the Philadelphia School Board at its Thursday meeting.

In response to Wilson’s criticism, district spokeswoman Monica Lewis issued a statement to Chalkbeat saying the district had been “assured by the vendor that the use of this tool to score trials as part of the process. application would be appropriate “.

Lewis said one of the five schools, Parkway Center City Middle College, had used a computer-rated writing sample for “positive feedback” admissions for several years.

Measurement, Inc. did not immediately respond to an email request for comment. But one document on his site says MI Write scores should not be used to grade students.

“It would be very unfair to give a mark to a student based on his assessment of an essay,” the document said. “A student could benefit from the transfer [the scoring engine’s] notes to notes, while another could be penalized. MI Write was designed to help students practice their writing skills and improve them based on feedback.

This year, the school district revamped its selective admissions process with the goal of improving access for traditionally underserved students. Instead of giving school heads the final say on who gets admitted, all eligible students are entered into a lottery.

Each school can still set standards for grades, attendance and behavior. But missing the typewritten sample writing limit can eliminate a student from the lottery.

Concerns about the handwriting test are just one of many issues raised about the new process. Several parents and students have registered to speak at the meeting on this topic.

Enrollment of blacks and Latinos in the district’s two most coveted and rigorously selective schools, Central High and Masterman Laboratory and Demonstration School, has declined in recent years, an issue highlighted in nationwide protests that followed a policeman who killed George Floyd last year. Alumni, students and some teachers from both schools demanded that the district act. Black enrollment in both schools is below 20%, in a district where more than half of the students are black.

The goal of the new system, officials said, is to address racial disparities in admissions and open up opportunities for more students by trying to minimize any bias that may arise when principals shape their next one. class.

“We’ve been working hard to avoid data points involving human assessment,” said Karyn Lynch, district chief of student support services, when she announced the new process in October, on the opening day of the district. the application window for admissions in fall 2022.

At Thursday’s board meeting, she strongly advocated the new process “as removing privilege so other students can finally have the opportunity they haven’t had before.” We have also heard from many enthusiastic students who are suddenly aware of a school selection process that they did not participate in in the past but participate in now.

Under the new system, the district also gave preference to students living in five zip codes where students are under-represented in top schools.

Wilson, the professor, called the effort to make the system more equitable a “noble goal,” but said the question was “how to make action match intention in a more meaningful way. do it”.

Besides Central and Masterman, the other three schools requiring the writing test are Palumbo Academy, Parkway Center City, and Carver High School of Engineering and Science. Carver and Parkway Center City are predominantly black, while Palumbo is one-third black and one-third Asian American.

Prior to the change, students were admitted to all five schools based on grades, attendance, demeanor, and a writing sample.

Standardized test results were an important factor before the pandemic, but they have not been administered since the spring of 2019. Officials have phased out their use for this cycle, but have not indicated whether they will resume their use at the time of the pandemic. ‘to come up. Standardized test scores generally correlate with socioeconomic status, putting underserved black and Latino students at a disadvantage.

The MI Write test used this year gives students a prompt, mostly on topics that don’t require detailed background knowledge. Questions may include “What is your favorite subject and why?” Or “If you could go back in time and meet someone, who would it be?” Students have 90 minutes to write an essay that is immediately scored on a scale of six to 30.

To qualify for Palumbo, Carver and Parkway Center City, the student needs a score of 17. For Central and Masterman, the score is 22, depending on the district.

Wilson raised concerns about how the district determined MI Write cutoff scores for admission. “Where’s the research that validates the cup scores?” ” he said. “I think Philadelphia has the burden of showing proof that these cup scores are appropriate.”

Wilson said 17 “is just below the median, and probably not that high a bar.” In a 90-minute sample, he said, “that would probably be achievable for most kids unless they really lack basic skills.”

At the board meeting, member Mallory Fix-Lopez said she heard that some students were getting fractional scores, such as 21.9, and asked Lynch if the scores would be rounded so that students reach the limit for Masterman and Central. Lynch said no.

Fix-Lopez said the students she spoke to who missed the threshold by a few tenths of a point are blacks and Latinos, “the population we were particularly looking for to resolve equity issues.” She said she would like to see demographics on otherwise qualified students who have fallen into that kind of gap.

Student Nora Ouarirdi told the jury that she got a 21.1. “I cried,” she said. “For the students who have worked incredibly hard, that doesn’t seem particularly fair. “

Wilson also noted that the instrument does not score for content. “I can write a beautifully written off topic response and get a good score,” he said.

Students do the computerized exercise in class and immediately learn their score. The students take the test now; the testing window is between November 29 and December 18.

Timothy Boyle, director of Science Leadership Academy Middle School, said 43 of his students took the test last week and almost all scored over 17. Based on what he knew about the students, he said he wondered what the scores really revealed about them. .

An A-level student scored a 21, one point lower than the score that would make them eligible for Central or Masterman. Another student, with a GPA of 1.5, scored 20, placing them above the cutoff for three of the five schools, although they may be ineligible due to other factors. “How does that translate into more fairness and more opportunity? Boyle asked.

The scoring system is based on an algorithm derived from what trained human reviewers produced based on the six-stroke writing model, which includes rubrics for ideas, organization, sentence fluency, choice words and style conventions. Each trait can be assigned a score between 1 and 5, adding to a maximum score of 30.

Wilson said the main use and benefit of self-scoring tests is to encourage teachers to assign more writing, as this helps them identify student weaknesses and quickly guide improvements. “Grading and evaluating writing takes so long,” he said.

Boyle also applauds the effort to make the system fairer. But he said if district leaders suspected a bias on the part of his educators to explain demographic disparities in his more selective schools, he needed to directly address that problem.

“They say they don’t know if this [new process] going to work, ”he said. “It’s like they’re throwing things at the wall and confusing action and strategy.”

The selection process for high schools includes three levels – the top five “criteria-based”, “city-wide admission” schools with a particular theme or goal, and those with catchment areas. who have special programs.

In the statement, made in response to questions from Chalkbeat, Lewis said this year the district has received more applications for high schools compared to last year, suggesting the effort to expand access is working. She said 15,382 students – both inside and outside the district – submitted 62,591 applications (each student can choose five schools), which represents “365 more students participating in the process than the ‘last year”.

Lynch and Superintendent William Hite, who will end his 10-year term in August, said the intention is to continually refine the process.

“We know that change can be difficult and we also know that the time has come for change focused on equity,” the district statement said. “This year’s changes are a starting point” that will lead to “further improvements … as we strive to be a stronger, more equitable school district.”