Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Problems flagged months before HPE hardware failure hit ATO systems

Leon Spencer | June 12, 2017
At least 77 events related to components that failed in the December 2016 outage were logged months earlier.

Under the turn-key deal, instead of having direct access to the SAN hardware, the ATO relied upon HPE to provide a full service. The agency also contracted with Leidos as service integrator.

Leidos operates a virtual dashboard over myriad ATO IT systems, and provides a problem management process should issues arise with parts of our IT infrastructure.

While procedures were in place to provide manual fail over for selected applications in the event of a failure, full automated fail over for the entire suite of applications and services in the event of a complete SAN failure in Sydney was not part of the storage solution for the SAN, the report stated.

 

What's next?

According to the report, the ATO has developed a new storage strategy to enhance its IT stability and resilience. This involves rebuilding its primary and back up storage systems with the newest technology from the HPE product portfolio working in conjunction with its 3PAR SAN technology.

"All production system workloads are now utilising the enhanced storage system," the ATO said. "Once data transfer activities are completed, the existing 3PAR SAN will be replaced by a new 3PAR and the current 3PAR SAN decommissioned by late July 2017 for forensic analysis."

While the ATO is still working with HPE in relation to the storage infrastructure, insights from its experiences relating to the outage will also inform its future IT acquisitions, the agency said.

"As contracts come up for renewal, we need to balance service, stability, resilience and cost," the ATO said. "Our IT program continues to prioritise government policy reforms and ATO corporate priorities, with a primary focus on another successful Tax Time for 2017.

"Future sourcing of IT is also influenced by whole of government initiatives, including closer collaboration with the Digital Transformation Agency," it said.

The release of the report comes just days after Australian Commissioner of Taxation, Chris Jordan, told Senate Estimates committee members that the ATO had reached a commercial settlement with HPE, the detailed terms of which are subject to contractual confidentiality.

The move to strike a settlement agreement following the hardware failure and subsequent systems outage echoes the settlement IBM reached with Australian Government over the troubled eCensus project with the Australian Bureau of Statistics (ABS) last year.

Jordan also took the opportunity to reveal the preliminary findings of various investigations into the storage hardware failure last year, citing a combination of factors, suggesting that the design of the SAN infrastructure played a part in its failure.

"The SAN design and configuration meant we had an over emphasis on performance features rather than stability or resilience - a relatively small disk drive failure had a large impact - only 12 of some 800 disk drives failed, but they impacted most ATO systems," Jordan said.

 

Previous Page  1  2  3  4  Next Page 

Sign up for Computerworld eNewsletters.