Sean Feeney’s correspondence with U.S. News regarding problems with ranking
Begin forwarded message:
From: “Feeney,Sean” <feeneys@ewsdonline.org<mailto:feeneys@ewsdonline.org>>
To: “Hines, Kenneth” <KHines@usnews.com<mailto:KHines@usnews.com>>
Cc: “rmorse@usnews.com<mailto:rmorse@usnews.com>” <rmorse@usnews.com<mailto:rmorse@usnews.com>>, “Kanas,Dr. Elaine” <kanase@ewsdonline.org<mailto:kanase@ewsdonline.org>>
Subject: RE: 2015 Best High School’s Rankings
Ken:
Further investigation into this situation has led me to believe that the methodology used for this year’s ranking system is dramatically different for New York State than in the past. Rather than use Cohort Data, you simply looked at the year’s snapshot of test takers. This is NOT what US News has done in the past, and it is not what New York State uses for accountability purposes. Rather, your previous ranking methodologies have looked at a cohort of students and based proficiency measures on how that cohort has performed. For the 2015 Rankings (which are based on 2012-13 data), the 2009 Cohort would be the object of inspection.
In the 2014 US News Rankings, Wheatley was recognized as having a 99% proficiency in English and a 99% proficiency in mathematics. This was consistent with what we know of our students’ performance on NYS examinations. It is also a reflection of the 2008 Cohort. For the 2015 Rankings, Wheatley was recognized as having a 99% proficiency in English and an 89% proficiency in mathematics. This is NOT an accurate reflection of the performance of the 2009 Cohort. In fact, for a school that has consistently demonstrated 99% and above proficiency for these two values, a 10-point drop must have some sort of explanation!
So what happens when one calculates proficiency in the manner US News chose to this year? Well, schools that challenge their students to reach high get “punished.” Let me explain.
As you probably know, New York offers multiple Regents examinations as high school graduation requirements. In fact, it offers three different mathematics examinations and one English examination. Only ONE of the mathematics examinations is required for graduation, however. To calculate proficiency ratings, you simply looked at the number of students taking the exams in 2012-13. That was not an issue with English, for there was only one examination. For mathematics, however, this became the cause for the decrease in Wheatley’s proficiency rating. Wheatley has long encouraged its students to challenge themselves upward. Our target for all students is the Regents Diploma with Advanced Designation, not the regular Regents Diploma. To earn this diploma, students must pass all three mathematics examinations. Our success with our students is why we have one of the highest Advanced Designation Diploma Rates in the State (approximately 85%, well above the 31% State average). When reaching for this Advanced Diploma, some students will take the third mathematics Regents (Algebra 2/Trig) multiple times in order to overcome an initial failure. Of course, if we withheld our weaker students from these higher level mathematics courses, our passing rates (and the US News 2015 proficiency rate) would have been much higher. By failing to look at Cohort data and instead simply refer to the number of students taking the exams this year, the methodology effectively punishes schools that push their students upward.
This year’s methodology is a stark departure from past US News Raking methodologies. It is also a stark departure for the manner in which NYS accounts for school performance. Clearly, we expect this error to be corrected publically so that Wheatley – and other school impacted by this significant departure from past practices – can obtain its appropriate and deserved place in the ranking. Our community expects nothing less!
Attached, please find a copy of the 2012-13 school accountability report card for The Wheatley School.
Thank you for your prompt attention to this matter.
Sean
—
Sean C. Feeney
Principal
The Wheatley School
11 Bacon Road
Old Westbury, NY 11568
Follow my blog at http://www.thewheatleyway.org<http://www.thewheatleyway.org><http://www.thewheatleyway.org>
Follow Wheatley on Twitter @WheatleySchool
Like Wheatley on Facebook: http://www.facebook.com/WheatleySchool<http://www.facebook.com/WheatleySchool>
From: Hines, Kenneth [mailto:KHines@usnews.com]
Sent: Wednesday, May 13, 2015 5:18 PM
To: Feeney,Sean
Subject: RE: 2015 Best High School’s Rankings
USNEWS is not making the full data set used to do the ranking available. What is available is the data for each school on our website, but we do not provide electronic files.
From: Feeney,Sean [mailto:feeneys@ewsdonline.org]
Sent: Wednesday, May 13, 2015 5:05 PM
To: Hines, Kenneth
Subject: Re: 2015 Best High School’s Rankings
I understand that. What are the underlying values of the linear regression? I am struggling to understand how we can be praised by NYS yet fail to make it through your Step 1 (with a more lenient cushion).
South Side and Wheatley are both unranked this year. Our schools are typically regarded as some of the best in the State.
What data were used for the regression? Is the table available?
Sean
On May 13, 2015, at 16:27, Hines, Kenneth <KHines@usnews.com<mailto:KHines@usnews.com><mailto:KHines@usnews.com>> wrote:
The value of 5.1 is a residual which calculated the difference between expected and observed values based on a linear regression – this is substep 1.4 which is on page 9 of our technical appendix.
From: Feeney,Sean [mailto:feeneys@ewsdonline.org]
Sent: Wednesday, May 13, 2015 4:13 PM
To: Hines, Kenneth
Subject: RE: 2015 Best High School’s Rankings
Thank you for your reply, Kenneth.
I continue to have some questions, however. I am puzzled by what appears to be contradictory information at the State level.
Recognizing what your ranking is trying to communicate in its first two steps – the need for a school to be achieving above the State expectations overall AND for disadvantaged students – I would like to refer you to Wheatley’s designation as a “Reward School” for the 2012-13 accountability year (see http://www.p12.nysed.gov/accountability/2012-13AccountabilityDesignations.html). This designation was given to those schools that achieve at the top levels without significant gaps among subgroups.
Using the 2012-13 scores available, I was able to replicate the State Test PI value of 119.2 (I received 119.1, but close enough). I would like greater clarification on the 5.1 value you have listed as the “gap between actual and expected PI” for Wheatley. This appears to be our residual value, although the calculations behind its value are not specified. Can you shed some light on this?
Furthermore, I wonder if Wheatley puts itself at a disadvantage by encouraging all students to take all three NYS mathematics examinations. Our weakest performance is in the Algebra 2/Trig examination. Although we have one of the highest “Aspirational Index” values for NYS, more students fail this exam than any other exams. If our weakest students did not take the exam (which is the case in many other schools), would that help our residual value?
Your thoughts are appreciated. Perhaps we can schedule a telephone conversation.
Sean
—
Sean C. Feeney
Principal
The Wheatley School
11 Bacon Road
Old Westbury, NY 11568
Follow my blog at http://www.thewheatleyway.org<http://www.thewheatleyway.org><http://www.thewheatleyway.org>
Follow Wheatley on Twitter @WheatleySchool
Like Wheatley on Facebook: http://www.facebook.com/WheatleySchool<http://www.facebook.com/WheatleySchool><http://www.facebook.com/WheatleySchool>
From: Hines, Kenneth [mailto:KHines@usnews.com]
Sent: Wednesday, May 13, 2015 12:00 PM
To: Feeney,Sean
Subject: RE: 2015 Best High School’s Rankings
Hello Sean,
The website was reporting the difference between predicted and actual performance as the “risk adjusted performance index.” We’ve updated our website<http://www.usnews.com/education/best-high-schools/new-york/districts/east-williston-union-free-school-district/the-wheatley-school-13654/test-scores> to change the label of “risk adjusted performance index” to “gap between actual and expected performance index.”
As the technical appendix<http://www.usnews.com/pubfiles/best-high-schools-technical-appendix.pdf> states on page 9 the risk-adjusted performance index is the ratio of the residual to one-third of the state’s standard deviation. Your schools actual risk-adjusted performance is .957 [=5.1/(0.33*16.14), which is less than 1.
The Wheatley School fell short of passing step 1 of the methodology. Please go to the detailed methodology PDF (http://www.usnews.com/pubfiles/best-high-schools-technical-appendix.pdf) and read about Step 1 starting on page 7. Then go to page 57 of the PDF that shows a profile of NY’s schools in Step 1.
If you link up the grid on page 57 of the PDF for The Wheatley High School on economically disadvantaged of 3% (http://www.usnews.com/education/best-high-schools/new-york/districts/east-williston-union-free-school-district/the-wheatley-school-13654) and state test performance index = 119.2 (http://www.usnews.com/education/best-high-schools/new-york/districts/east-williston-union-free-school-district/the-wheatley-school-13654/test-scores) you can see that your school did not place high enough on a “relative basis” – to be a blue diamond to have passed Step 1 for that level of state test performance index and that poverty level %.
Regards,
Kenneth Hines
Data Analyst
U.S. News and World Report
1050 Thomas Jefferson Street N.W.
Washington D.C. 20007
From: Morse, Bob
Sent: Tuesday, May 12, 2015 2:46 PM
To: Hines, Kenneth
Subject: FW: 2015 Best High School’s Rankings
From: Feeney,Sean [mailto:feeneys@ewsdonline.org]
Sent: Tuesday, May 12, 2015 2:35 PM
To: Official
Cc: Kanas,Dr. Elaine; Wasserman,Greg
Subject: 2015 Best High School’s Rankings
To Whom it May Concern:
A review of your newly-released 2015 Best High Schools list has left me puzzled as to why The Wheatley School (in Old Westbury, NY) has been omitted from your list.
After a careful review of the data you list for Wheatley on your site and the methodology used to rank schools, I fail to understand the omission.
As you know, schools need to pass three “tests” in order to be ranked by your index.
Step 1: Identify HS that Perform Better than Expected on State Accountability Assessments
Your methodology indicates schools with a Risk-Adjusted PI value greater than 1.0 are doing better than expected and will move on to Step 2. Wheatley’s Risk-Adjusted Performance Index is listed as 5.1. By this account, we should have been considered for the second step.
Step 2: Identify HS that Perform Better than the State Avg for their Least Advantaged Students
Your methodology indicates schools with a “Gap Between School and State Among Disadvantaged Students” value greater than 0 (zero) are places where disadvantage students outperform expectations. These schools should be considered for the Step 3. Wheatley’s Gap Between School and State Among Disadvantaged Students value is 9.5. By this account, we should have been considered for the third step.
Step 3: Identify HS that Performed Best in Providing Students with Access to Challenging College-Level Coursework
Your methodology indicates schools who have passed the first two steps of the process and have a College Readiness Index greater than 19.42 will be ranked nationally and receive a “medal.” Wheatley’s College Readiness Index is 64.4. Clearly, Wheatley succeeds in this step as well.
Despite being able to see the appropriate 2015 Wheatley data and match it to the criteria used in the methodology, Wheatley is listed as “Unranked” on your site http://goo.gl/m62pGC.
I would appreciate insight into the aforementioned situation.
Regards,
S. Feeney
—
Sean C. Feeney, Ph.D.
Principal
The Wheatley School
11 Bacon Road
Old Westbury, NY 11568
Follow my blog at http://www.thewheatleyway.org<http://www.thewheatleyway.org><http://www.thewheatleyway.org>
Follow Wheatley on Twitter @WheatleySchool
Like Wheatley on Facebook: http://www.facebook.com/WheatleySchool<http://www.facebook.com/WheatleySchool><http://www.facebook.com/WheatleySchool>
Round the Inkwell 
One Response to “Sean Feeney’s correspondence with U.S. News regarding problems with ranking”
[…] Seeking answers, the principals who coauthored the Washington Post piece contacted US News. This chain of email shows how they were […]