|
|||
|
|||
|
|
|
|
|||||||||||
|
1. |
The ANAO recommends that DEEWR establish for future Digital Education Revolution program funding agreements, an obligation for non-government education authorities to provide an annual acquittal of program funds, including an independent audited statement that the funding was expended for the purpose of achieving the deliverables and performance benchmarks in accordance with the agreement. DEEWR response: Agreed |
|
2. |
The ANAO recommends that, in order to strengthen external reporting and help steer program direction, DEEWR establish a balanced set of Portfolio Budget Statements key deliverables and performance indicators to measure the effectiveness of the Digital Education Revolution program. DEEWR response: Agreed |
|
3. |
The ANAO recommends that DEEWR consult with education authorities to design and conduct an audit of a sample of schools funded under Digital Education Revolution program, in early 2012 to assist in: a) providing assurance on the accuracy of information reported by education authorities on computer installations; b) confirming whether the schools have achieved the 1:1 computer to student ratio; and c) identifying any reasons for schools not achieving the 1:1 computer to student ratio, including any funding deficiencies. DEEWR response: Agreed |
3.21 The Committee held a public hearing on Wednesday, 14 October 2011 with the following witnesses:
3.22 As this program is in its final stages, the Committee focused on the appropriateness of the key performance indicators and whether evaluation to assess educational outcomes has been adequately addressed. In addition, the Committee wanted to ensure the program had delivered value for money and that lessons learned had been implemented across new programs.
3.23 The Committee took evidence on the following issues:
3.24 As noted above, the ANAO’s report contained three recommendations aimed at improving the administration of the NSSCF. Referencing these in his opening statement, the Auditor-General outlined the report findings which suggested that DEEWR’s administration of the program ‘had been effective in supporting progress through a partnership approach towards the computer fund’s objective of increasing the computer-to-student ratio’. However, the Auditor-General also noted the report found that ‘there were aspects of the Department’s oversight that could have been strengthened’.[16]
3.25 In response to the statement made by the Auditor-General, DEEWR noted the ANAO’s valuable contribution in the continuous improvement of administrative processes. DEEWR also advised that they consider that the first two recommendations made by the ANAO have been fully implemented, with the third relating the 2012 calendar year to be fully implemented in due course.[17]
3.26 The Committee asked DEEWR for further detail on the implementation of each of the three recommendations.
3.27 In response to the ANAO’s recommendation to strengthen future program funding agreements with non-government education authorities, DEEWR noted that:
…since the audit report… the basis of the program has changed to that of a national partnership. However, under the DER funding agreements, which are part of the national partnership arrangements, non-government education authorities are, in fact, required to submit six-monthly progress reports, outlining their progress to reach a computer-to-student ratio of one to one by the end of 2011. All non-government education authorities have complied with these requirements, and the most recent progress report was received by us on 15 July 2011.[18]
3.28 The second recommendation suggested establishing a balanced set of portfolio budget statements key deliverables and performance indicators to measure the effectiveness of the program.[19]
3.29 DEEWR considered they have fully responded to this recommendation with the inclusion of additional information in the 2011-12 DEEWR Portfolio Budget Statement (PBS). The key performance indicators listed in the PBS are:
3.30 The third ANAO recommendation suggested action to be taken in early 2012 aimed at increasing assurance over schools' achievement of the computer-to-student ratio of one-to-one through an audit of a sample of schools.[21]
3.31 According to DEEWR, this recommendation is being advanced as part of the overall evaluation strategy for the fund. A mid-program review is planned in 2012 to cover a number of aspects and ‘will incorporate an audit in conformance with the recommendations of the Audit Office’.[22]
3.32 The ANAO provided commentary on a significant funding issue arising from the rush to meet the Government’s 100 day commitment, whereby the on-costs associated with the deployment and support of the computers had not been agreed. The Government initially thought that states and territories would cover these costs, however, the COAG did not endorse the proposal. Dr Paul Grimes, then Deputy Secretary of the Department of Finance and Deregulation, was appointed to undertake a review to determine the full costs of computers funded through the NSSCF.[23] Following the conclusion of the review and agreement from COAG, the Government announced a further $807 million in funding for the program.[24]
3.33 In trying to determine why these costs were not agreed as part of the initial funding announcement, the Committee asked what advice the Department had given to the Government prior to the first COAG meeting.
3.34 Noting that the audit canvassed these events in detail, and are otherwise on the public record, Dr Evan Arthur, Group Manager, National Schools and Youth Partnerships Group, provided a historical account as follows:
Immediately after the election of the Rudd government, there was a COAG meeting which agreed to the rollout of the computers, but there was a reservation, as I recall, in terms of funding of the legitimate additional costs. At a subsequent COAG meeting…the Commonwealth agreed that it would meet legitimate additional costs for installation of the computers. …A process was then agreed by which we would quantify what those legitimate additional costs were. That was the process which was managed by Dr Grimes. … On receipt of the Grimes report, the Government decided to accept the quantification of the costs contained within the Grimes report.[25]
3.35 Additional questions on this matter were taken by the Department on notice. The Department’s response to the Committee had not been provided at the time of the publication of this report.
3.36 In regard to the overall program rollout, DEEWR asserted that beyond the initial disagreement over funding for on-costs, there have been ‘no substantive difficulties within the administration of the program’. While over the course of the program questions have arisen, DEEWR consider these ‘have all been handled in an entirely cooperative way’. [26]
3.37 DEEWR reinforced their view of the program’s successful progress with the advice that ‘at no point has the program failed to meet its timetable of pre-existing commitments’.[27]
3.38 The ANAO noted the concept of ‘value for money’ was raised by the Government at the very early stages as an expectation of this program. Following consultation with central agencies, a unit price of $1000 per computer was established. By way of example, the ANAO report explained that based on this unit price, a school requiring 10 computers would receive $10,000 in application round funding.[28]
3.39 Taking into account the highly competitive ICT market in Australia, the ANAO report noted DEEWR encouraged government education authorities to undertake centralised purchasing processes for schools to achieve maximum purchasing power.[29]
3.40 The ANAO found the DEEWR approach to encouraging value for money was generally sound. The option to use residual funding for complementary ICT equipment provided flexibility for education authorities and schools as well as a strong incentive to achieve value for money.[30]
3.41 The Committee asked DEEWR for their view on this policy and whether they would consider using this type of mechanism, or a similar refined version, for future programs.
3.42 In regard to the current program, DEEWR commented that rather than micromanage expenditure of funds at school level, this policy decision allowed flexibility to apply the funds to a range of complementary purposes.[31]
3.43 The second part of the question relating to further promulgation of this policy was put on notice. The Department’s response to the Committee had not been provided at the time of the publication of this report.
3.44 Noting the flexible arrangements, the Committee asked DEEWR whether any data was being collected identifying school ICT purchasing decisions. The Committee was interested in the overall benefit being realised by students. This included whether schools are providing laptops and if so, are these being made available to students outside of school hours.
3.45 According to DEEWR, the majority of deployments have been Netbooks. However, some schools elected to install desktops and there has been an emerging trend toward investment in slates (iPads or similar). Overall, a very wide range of choices have been made, depending on both end-user requirements and the procurement model used.[32]
3.46 DEEWR advised that a range of procurement and deployment models have been established by state and territory education authorities, and that these models in turn go some way in determining the options available to schools.[33]
3.47 DEEWR used the following examples to demonstrate the different approaches taken:
3.48 The ANAO’s report provides additional information with two case studies outlining the alternate models adopted by the NSW and Victorian education departments. [36]
3.49 DEEWR claimed that the DER program ‘has also made significant changes to the ICT environment within schools’. Having an easily accessible, supporting network in place is a precondition of effective use of technology in education. According to DEEWR, this has been achieved as a result of the one-off funding injection for on costs.[37]
3.50 The Department took questions on notice from the Committee in regard to the ICT supporting infrastructure. Specifically:
The Department’s response to the Committee had not been provided at the time of the publication of this report.
3.51 The Committee noted that in recent months there have been a number of media articles claiming that parents of students in Queensland schools are required to pay additional fees for laptop computers provided under the NSSCF.[38] The Committee asked DEEWR to respond to these claims, and advise whether those fees are in any way associated with the DER program or any deficiencies in the NSSCF?
3.52 DEEWR refuted the reports, stating:
…there is a very clear position from the governments on this issue, which is that there should not be any fees associated with the provision of computers funded by the Commonwealth. The Commonwealth, as a result of the COAG agreements … is… meeting the total cost of ownership of the device for four years.[39]
3.53 However, DEEWR noted that the Commonwealth Government has ‘no role in restricting the choices schools make and how they fund their activities’. Expanding on this DEEWR commented on three situations in which schools may be charging parents:
3.54 Further, DEEWR explained that if the fees are to fund the school’s ongoing maintenance of computers outside those provided under the NSSCF (either prior to the establishment of the fund or to years other than Years 9 to 12), the Commonwealth could not prevent that, but there is a stipulation that it has to fully discussed and with the agreement of the parent body.[41]
3.55 DEEWR considers that where appropriate consultation has taken place, it would be unreasonable for the Commonwealth Government to interfere with a school’s internal economy beyond stipulating that ‘there should be no components of those fees which are a cost associated with the provision of the Commonwealth computers’.[42]
3.56 Further to DEEWR’s first point that some school’s may charge fees in order to provide computers with higher specifications, the Committee wanted to confirm that the NSSCF notional price provided for technology of a suitable level for activities being undertaken in classrooms.
3.57 DEEWR’s response confirmed that the funding envelope provided caters for a ‘very highly capable device’. DEEWR noted that as the devices are universally sourced from overseas, the appreciation of the Australian dollar has also been of benefit. DEEWR did note that some schools may have elected to purchase more expensive computers to support speciality software for classes such as music or graphic design. [43]
3.58 The ANAO report noted the importance of effective and timely identification of need and capacity to support the program’s roll-out. To meet the Government’s ‘100 day commitment’ DEEWR moved quickly to develop and distribute a preliminary survey. On 18 January 2008, education authorities were advised that completed surveys were due back to DEEWR by 7 February 2008. DEEWR acknowledged the rushed timeframe.[44]
3.59 The ANAO reported that there were 460 instances (16 per cent of 2929 schools) where schools had provided anomalous data. However, they also noted that for the majority of these instances, the size of data discrepancies was in the vicinity of 10 computers. The ANAO’s report suggested where data discrepancies exceeded 10 computers, DEEWR could have asked education authorities to review and confirm or amend data provided.[45]
3.60 Noting the historical issues with data discrepancy, the Committee asked what assurances DEEWR could give that the data provided in the 2012 planned review will be accurate.
3.61 DEEWR acknowledged the suggestions in the ANAO report regarding mechanisms to improve data collection. However, DEEWR stated that they ‘do not have information that there are discrepancies in the data’. DEEWR explained that in order to make decisions on funding only two sets of figures were used from the data collected. These figures were then put through a number of iterative checks with education authorities to confirm their veracity.[46]
3.62 Following DEEWR advice on the accuracy of data, the Committee sought to confirm whether the number of computers to be provided to schools was based on the number of students as at the end of the 2007 school year or the beginning of the 2008 academic year.
3.63 DEEWR advised that the figures were taken from the annual Commonwealth census data on the number of students in the Australian school system. DEEWR noted that some schools may have considered the figures to be not entirely accurate, but that overall within this ‘$2 billion-plus program’ there has been evidence of ‘significant savings in the price paid for equipment’, with increase purchasing capacity as a result of exchange rate movement. In summary, DEEWR stated:
There are more than enough dollars provided in this program for any issues around the margins of those figures to be addressed.[47]
3.64 The ANAO found that educational authorities had reported solid progress on the installation of computers. Most recently, at the 2011-2012 Supplementary Budget Estimates hearings held on 20 October 2011, DEEWR advised that at the last formal reporting date of 30 June 2011, installation was on target at 75 per cent. According to DEEWR ‘educational authorities have publicly stated and repeated assurances to the Department and the Government that they will meet the time frames’.[48]
3.65 In relation to the installation figure of 75 per cent, the Committee asked if this meant the computers were delivered to schools and operational. Further, noting that the 30 December 2011 target has been extended to early 2012, the Committee asked DEEWR to predict when all computers will have been installed.
3.66 DEEWR confirmed that the 30 June 2011 figures were for computers that had been delivered and were operational. In regard to all computers being in place and functioning to bring the student to computer ratio to 1:1 for Years 9 to 12, DEEWR advised that they expect students to have their computers at the start of the new school year.[49]
3.67 The Committee asked whether more recent data on the progress of installation was available. The Member for Lyne commented that in October 2011 there appeared to have been a ‘flood of computers land’ in his electorate.[50]
3.68 DEEWR advised the Committee that while they do have more recent information, a decision had been taken by the Government that ‘it would only publish information derived from the six monthly consolidated reports from all education authorities’.[51]
3.69 The ANAO report notes the NPA sets out high level governance arrangements for the delivery of the program, including performance benchmarks (KPIs), but these primarily relate to computer installation. The ANAO noted the difficulties of evaluating a 'multi-jurisdictional program focused on changing teaching and learning in schools’. [52]
3.70 The Committee acknowledged DEEWR's implementation of the ANAO's Recommendation No.2, but noted these were quantitative measures that don't provide an indication of whether the program is meeting the stated objective of preparing students for the digital world.
3.71 The Member for Robertson commented on the important social benefits being achieved in her electorate as a result of the program:
I do not know that you [DEEWR] get to see the faces of the students who receive them or to knock on doors and have a mother come to you and say, “My son has changed his whole attitude to education because this is the first new thing he has ever had in his life”. That is the sort of testimony to this program that I have experienced in my electorate and particularly in the suburb of Kariong where many families have been very advantaged by this. Their kids have got the advantages they need to progress into the future.[53]
3.72 The Committee asked DEEWR if any KPIs are being developed to indicate that students are significantly improving their performance across a range of subject areas because of their engagement with new technologies.
3.73 DEEWR noted that ‘technology is only a means to an end’. In support of this statement, DEEWR drew the Committee’s attention to the documented outcomes the Department developed for the program, which focus on educational outcomes rather than the provision of computers.[54]
3.74 Further, DEEWR contended as there are multiple inputs into ‘good or bad educational outcomes’, it is extremely difficult to isolate the influence of a single factor, in this case technology. DEEWR referred to high-level studies by the former British Government agency, Becta[55] into the correlation between introduction of technology and results. While the results appeared positive, DEEWR cautioned overemphasise on Becta’s findings in a ‘very fraught methodological area’.[56]
3.75 The Committee referred DEEWR to a recent article in the Courier Mail[57] where the significant improvement in NAPLAN results of the Doomadgee State School was in part attributed to ICT. The Committee suggested tools such as NAPLAN may be able to provide the longitudinal work.
3.76 DEEWR noted that a number of studies have identified the dominant variables influencing results as principals, leadership and teaching quality.[58] Offering a personal perspective Dr Arthur commented:
… if you combine those strengths with the kind of potential that technology offers, I personally am sure that you can get exceptionally good results from that. I am just being cautious in the sense that I would not want to claim that we can demonstrate that to a level of proof which would satisfy academic peer review rigour.[59]
3.77 In summary, DEEWR advised they understand the importance of the issue and are continuing work in this area.[60]
3.78 Noting DEEWR’s response, the ANAO drew the Committee’s attention to chapter five of the ANAO’s report, which outlines the longitudinal study being undertaken by the NSW Department of Education and Training in partnership with the University of Wollongong. The study is ‘looking at issues and effects from the program in relation to pedagogy, student engagement and outcomes’.[61] The ANAO report suggests DEEWR leverage of this work.[62]
3.79 DEEWR advised that in their six-monthly reports education authorities are required to report on the four themes developed at the commencement of the program: the installation of computers, leadership, teacher capability and digital resources. Respondents are also asked to provide case studies ‘that can be used and built on in the evaluation of good practice in the classroom’.[63]
3.80 According to the ANAO’s report, the timetable for the implementation of the DER program led to a focus on key administrative activities, leaving the development of an evaluation framework to be considered later following completion of more detailed program planning. At the time of the ANAO audit a final evaluation framework had not been released. The ANAO concluded that 'earlier investment in evaluation methodologies and associated data as the program evolved would have provided a stronger foundation for measuring the impact of the DER program'.[64]
3.81 DEEWR informed the Committee that an evaluation strategy has been developed in consultation with the Commonwealth, state and non‑government authorities, and the Australian Information and Communications Technology in Education Committee[65]. DEEWR advised it has been agreed that the evaluation will:
3.82 DEEWR also advised the Committee that they are in the process of identifying a service provider to undertake the mid-program review and the audit in 2012.
3.83 Referring back to the program objective to ‘prepare students for further education, and training, and to live and work in a digital world’, the Committee was interested in what efforts had been made to engage the ‘digital world’. More specifically, the Committee wants to be sure that the skills being developed as a result of the DER meet the requirements of post‑secondary education providers (universities as well as the vocational and education training sector) and potential employers.
3.84 Beyond the schools and education authorities, the Committee asked if any consultation had been undertaken with these post-secondary stakeholders in terms of helping to identify performance indicators that would demonstrate that there has been some development in technology capability of the students.
3.85 The Department undertook to respond to this question on notice. The Department’s response to the Committee had not been provided at the time of the publication of this report.
3.86 Overall, the Committee acknowledges that despite the early challenges, DEEWR has managed the program effectively to meet announced implementation timeframes. In other areas, such as KPIs and evaluation, the Committee believes DEEWR could have done more.
3.87 In particular, the Committee does not agree with DEEWR that the ANAO’s Recommendation No.2 regarding performance indicators has been fully implemented. The Committee acknowledges the difficulties associated with measuring high level qualitative achievements as a result of individual programs, such as DER’s stated aim ‘to contribute sustainable and meaningful change to teaching and learning in Australian schools’.
3.88 However, the Committee considers that if such high level aims are to be stated then it is reasonable to expect that a corresponding system of measurement be developed. If this cannot be done in full efforts should be made to develop indicators toward the high level outcome for the program, even if they only provide a partial gauge of the programs contribution. Given the size of the funding allocated to the DER program the Committee considers efforts in this regard even more important.
3.89 To assist with improved performance measurement, the Committee agrees with the ANAO’s comments in their report that there is merit in DEEWR leveraging off the evaluation work of state and territory education authorities. However, the Committee feels that DEEWR should go beyond this and also develop relationships with appropriate research bodies to study the program’s qualitative achievements. Such bodies could include universities and other peak representative organisations in the education sphere.
3.90 Further, the Committee concurs with the ANAO’s advice that the evaluation mechanisms should be developed at the start of a program. While accepting that the initial ‘100 day’ timeline placed pressure on the Department, the Committee contends that DEEWR could have leveraged previous program experience to produce an evaluation model earlier. The Committee notes that DEEWR has more recently been working with stakeholders to develop an evaluation strategy, but remains concerned that some arrangements are still being decided so close to the deadline for the full implementation of the computer roll-out.
3.91 The Committee was concerned with the suggestion in the ANAO’s report that the initial payment acquittal arrangements did not adequately protect the Australian Government’s interests. The Committee acknowledges DEEWR’s advice that this matter has been rectified following the move to the National Partnership Agreement on the Digital Education Revolution. The Committee trusts that the Department has learnt from this and has processes in place that ensure stronger future program funding agreements that include appropriate and timely acquittal mechanisms.
3.92 In terms of DEEWR’s procurement strategy, the Committee commends the Department’s initiative to encourage flexibility and value for money by allowing any residual funding to be applied to complementary ICT. The Committee would like to see this type of thinking applied to suitable similar programs across Government.
3.93 Ten questions on notice were submitted to DEEWR. While acknowledging the limited timeframe for responses, the Committee is nonetheless disappointed that no responses had been received at the time of report finalisation. The Committee had a particular interest in obtaining answers to the questions on the broader reach of the program, for example:
3.94 The Committee notes that there are a number of DEEWR initiatives underway to boost schools’ ICT infrastructure and capacity to use the technology, as well as a sizeable program administered by the Department of Broadband, Communication and the Digital Economy (DBCDE) to integrate the benefits of the National Broadband Network[67]. There are also many state and territory programs, such as the Connected Classrooms program in NSW. The Committee emphasises the importance of leveraging investments in computers or infrastructure to ensure classrooms a fully networked, and is encouraged to see initiatives towards this end. Ensuring that classrooms are as connected as possible is essential to maximise the educational outcomes for our children into the future.
3.95 With this combined multi‑billion dollar investment across government agencies and levels of governments, the Parliament and the public are entitled to be informed of the progress and outcomes in a timely manner. Therefore, the mid-program review should be made public soon after its completion. It is also important that there is comprehensive and transparent reporting of the program as a whole. The Committee therefore reemphasises the comments and recommendation made in Chapter 2 of this report - that more work needs to be done on improved cross-agency and cross‑jurisdictional financial reporting as part of the Commonwealth Financial Accountability Review.
Recommendation 2 |
|
| The Joint Committee of Public Accounts and Audit recommends the Department of Education, Employment and Workplace Relations publicly release in full the findings from the mid‑program review scheduled for 2012 within three months of completion. | |
Navigation: Previous Page | Contents | Next Page
