A different big data challenge awaited Dr Phillip Lindsay when he moved to New Zealand Genomics Limited (NZGL) as CIO.
NZGL is a government funded initiative to provide genomics research services for the NZ research community, as well as the clinical and medical fields. It is based at the University of Otago, one of three collaborating universities within NZGL, together with Auckland and Massey universities.
One of its roles is to provide bioinformatics support, which is delivered within a private cloud and executed on a high performance computing cluster to genomic researchers based in the three universities as well as commercial and non-commercial clients across New Zealand.
"In a nutshell our challenge was to be able to transfer large files around the country and in a predictable time frame, and make use of the bandwidth that was there," says Lindsay. "We are doing new stuff now, where these large files need much cleaner network pathways to be efficient."
"Technology is getting faster, bigger, better and producing more data," he notes, and the sequencing machines they support produce high volumes of data, up to four terabytes.
Lindsay says for a variety of reasons, there has been underinvestment in campus WAN gateways so trying to move massive sequencing data files through the enterprise-grade firewalls in place was not going to work.
All organisations have built their network where the lowest common denominator for security is the corporate environment, says Lindsay. "That is all fine in a corporate environment. Even if you are moving payroll file from A to B, there will be a few kilobytes, not a terabyte."
Our challenge was we are not going to change the security policies of organisations and it would not be appropriate to do that because the arrangement they have are appropriate for their environments, he says.
NZGL asked Research and Education Advanced Network New Zealand (REANNZ), of which it is a member, to come up with a system that would create a high-capacity firewall bypass connecting NZGL to client sites while maintaining security and removing the impact of dropped packets in throttling back transmission rates.
"Traditional best-effort IP networks work well for most applications but the trend in science is to become ever more data-intensive," says REANNZ CEO Steve Cotter.
"What we proposed to them was a concept that my team had developed when I was with the Lawrence Berkeley National Laboratory prior to coming to New Zealand."
"We came up with a concept of the science DMZ which was using the same concept for a security DMZ where you move part of the network outside the firewall," explains Cotter. "We were doing the same thing with the high performance infrastructure that they needed. So we created a data enclave outside the firewall that was still trusted and secure but provided a high-speed onramp to the backbone.
Sign up for Computerworld eNewsletters.