Netflix has developed a microservices orchestrator called Conductor and is now offering it via open source.
Inspired by the company's need to run business processes for streaming titles worldwide, Conductor was built as a cloud-based orchestration engine. It manages workflows, can pause and restart processes, and uses a JSON DSL-based blueprint to define execution flow. It also features a user interface to visualize process flows and can scale to millions of concurrently running process flows.
Conductor has been in use at Netflix for almost a year. "To date, it has helped orchestrate more than 2.6 million process flows ranging from simple linear workflows to very complex dynamic workflows that run over multiple days," Netflix said in a bulletin. The company is open-sourcing the technology in order to learn from others with similar needs and enhance the software.
With microservices, business process automations are driven by orchestrating across services, Netflix said. "Conductor enables orchestration across services while providing control and visibility into their interactions. Having the ability to orchestrate across microservices also helped us in leveraging existing services to build new flows or update existing flows to use Conductor very quickly, effectively providing an easier route to adoption."
Conductor is powered by a state machine service called Decider that works with a distributed queue to manage tasks. The tasks communicate via an API layer, which exposes APIs over HTTP. As workflow events occur, Decider combines the workflow blueprint with the current workflow state, identifies the next state, schedules tasks, and updates the workflow status.
Netflix is looking ahead to further possibilities, including supporting the AWS Lambda compute service or similar functions for serverless simple tasks, tighter integration with container orchestration frameworks and logging of execution data for each task.
Sign up for Computerworld eNewsletters.