Are schools gaming NAPLAN to manipulate academic performance?

Schools may be strategically manipulating participation in the National Assessment Program – Literacy and Numeracy (NAPLAN) to improve their public performance results, according to new research from UNSW Business School.

The research found that schools with lower initial test scores relative to other schools teaching similar student bodies show higher rates of non-participation in NAPLAN tests following the public release of performance data on the My School website.

This website, which is maintained by the Australian Curriculum, Assessment and Reporting Authority, is designed to provide transparency and accountability of Australia’s school education system through the publication of nationally consistent school-level data.

However, the research paper, Unintended consequences of school accountability reforms: Public versus private schools, published in the Economics of Education Review, found that certain schools are more likely to exclude lower-performing students – a move which would enhance a school’s overall test results.

The study said this increase in non-participation was largely driven by formal parental withdrawal, a process that parents can initiate by applying directly to the school principal.

Students may be withdrawn from the testing program by their parent/carer for reasons such as “religious beliefs or philosophical objections to testing”, according to the NAPLAN website, which states that this is decided by parents/carers in consultation with their child’s school.

“We found that the fraction of students withdrawn from testing in years after the 2010 launch of My School went up, while the fraction who were absent or exempt from testing remained roughly steady – with poorly-performing students far more likely than other students to be withdrawn from testing,” said Gigi Foster, a Professor in the School of Economics at UNSW Business School, who co-authored the research paper together with University of Melbourne Associate Professor Mick Coelli.

“We further find that this increase in withdrawal rates occurred in schools that were initially reported on My School to be poor performers relative to peer schools,” she said.

This behaviour, commonly referred to as “gaming the system,” undermines the objective of NAPLAN to provide a fair assessment of student performance across Australia, Prof. Foster asserted.

Private schools are more likely to game the system

The research identified a sector-specific response, with private schools more likely to adjust their testing pools compared with public or Catholic schools. This indicates a higher tendency among private schools to manipulate participation rates to maintain their reputations.

More specifically, the research found that students at private schools are more than twice as likely than their peers at state schools to be pulled out of NAPLAN tests in subsequent years if they received low grades in previous years.

“We provide some suggestive evidence that these higher rates of withdrawal in lower-performing schools were more prominent in independent private schools – which are famous for charging parents a pretty penny for the privilege of enrolling their kids – than in public schools,” said Prof. Foster.

“These findings are consistent with a situation in which the increased withdrawal is used as a tactic to manipulate the image of a school’s quality: excluding more poor performers from testing makes the school look better than it otherwise would look on My School.”

How parents play a role in NAPLAN testing participation

The research paper noted that parents can formally apply to the school principal for the withdrawal of their child from NAPLAN testing “in the manner specified by the local testing authority” (usually the state’s department of education).

NAPLAN testing protocols indicate that withdrawals are intended to address issues such as religious beliefs and philosophical objections to testing, but the research paper observed that providing a considered explanation for withdrawal appears to be unnecessary.

Furthermore, the process of applying for withdrawal was not actively promoted by governments or the testing authority. “We believe that knowledge of the process was provided to parents directly by schools,” said the researchers, who noted that parents apply for withdrawal directly to the school, not to the testing authority or government.

Parents have reported pressure from schools to withdraw children from NAPLAN testing in surveys conducted by state education authorities, while the popular press have also reported on claims by parents that schools have instructed children not to sit the NAPLAN tests “in order to boost their chances of obtaining higher overall scores”, the research paper stated.

“The volume of such reports led the NSW Minister for Education to warn that principals or teachers found encouraging children not to sit the tests may face disciplinary action.”

Unintended consequences of manipulating NAPLAN

Prof. Foster explained that she and co-author A/Prof. Coelli, were familiar with overseas research evaluating government programs designed to make school performance more transparent to parents, and thereby to raise the accountability of schools.

“The idea of such programs is to help improve educational outcomes, since with more information, parents would be expected to select higher-performing schools for their children to attend, thereby exerting competitive pressure on low-performing schools to either lift their game or close,” she said.

However, she said, these types of programs could have unintended consequences – if not designed with careful thought. As low-performing pupils at poorly performing schools were more likely not to sit the NAPLAN tests, Prof. Foster said this meant that the My School program may have had the unintended consequence of hiding from public view the low skills in English and numeracy of some of Australia’s weakest school students.

“This is somewhat ironic since the whole point of a school accountability program is generally to improve the education that students – and particularly disadvantaged students, whose parents may have few sources of information about school quality apart from the program – receive,” she said.

Data analysis and policy implications

In conducting their analysis, the researchers utilised data from the My School website spanning from 2008 to 2015. This data set included standardised test scores and participation rates from a balanced panel of 6981 schools over eight years.

While the intention behind publicising school performance data was to drive improvements and transparency, Prof. Foster said, the results indicated that it could also incentivise undesirable behaviours.

“The government could fix a maximum percentage of students who may be excluded from testing on any given testing day, although the monitoring costs of this may be prohibitive and it may result in further unintended consequences, such as schools reducing the number of allowed exclusions amongst average performers – perhaps then creating pressure on sick or injured students to sit the tests – in order to create more ‘slots’ for weak performers to sit out the tests,” she said.

“Another option would be for the My School site to report the percentages of students excluded from testing at each school, while adding a note informing parents that a comparatively high fraction of students excluded from testing may signal that a school’s true average performance is lower than what it appears to be on the My School tables.”