About three years ago the Financial Industry Regulatory Authority (FINRA) began plotting a migration to Amazon’s cloud. Most companies when they do so target low-hanging fruit: Applications that can be relatively easily lifted and shifted to the cloud.
Not FINRA. “We started with the migration of our most critical systems to AWS,” explains Steve Randich, executive vice president and CIO of FINRA, and a former CIO of the NASDAQ stock exchange and Citibank. “We moved our most mission critical, data-intensive services first.” Randich called them FINRA’s “crown jewels.”
FINRA records every order and quote in the New York Stock Exchange daily. That’s about 75 billion individual events per day. FINRA processes in one day the magnitude of data that Visa and Mastercard process in six months, Randich says.
FINRA stores this information so that it can analyze trends over days, weeks and months. That amounts to trillions of records and about 20 petabytes of storage. FINRA’s IT “center of gravity” is now in Amazon Web Services, he says.
Why move to the cloud
The U.S. Securities and Exchange Commission, the primary government regulator for the stock market, has rules upcoming that will require an even greater amount of data to be collected and analyzed compared to what is required today. This has been one of the driving factors toward FINRA’s use of the cloud, Randich says.
The organization’s legacy proprietary data warehouse appliances worked reasonably well, but Randich questioned if they were capable for serving FINRA’s future use cases. “When looking forward, we very quickly determined that we needed something different,” he said during a keynote state at AWS’s recent re:Invent conference in Las Vegas. (Watch Randich talk about FINRA’s cloud journey in the video below at the 1:33:20 mark of the re:Invent keynote.)
FINRA evaluated many providers. Legacy infrastructure vendors tried to convince him that a database of this scale could not run in the public cloud. After an evaluation and proof of concept process FINRA found AWS to be “several years ahead of the closest competitor,” a gap that Randich says is increasing.
He had four principles in moving to the cloud:
-Self-sufficiency: Randich didn’t want to rely on consultants or vendors. FINRA built expertise in-house to manage its AWS cloud.
-Public vs. private cloud: Many in the financial services world prefer a private cloud. “Our view is, why own, manage and support all that commodity hardware?” he asks. “Why not be even with Moore’s Law?” Randich says “infrastructure people who want to stay within their comfort zone” advocate for private cloud.
-Open source: Randich focused development efforts on HBase and Hive, open source database standards instead of proprietary databases from vendors that would require licensing.
Sign up for Computerworld eNewsletters.