Open source projects often stem from businesses attempting to solve an internal problem, before being offered up to a wider community. And increasingly in recent years it has been the internet giants such as Facebook, Twitter, LinkedIn and Google which have offered the technology which underpins their huge, hyper-scale computing environments.
Owning some of the biggest data centres in the world, these companies are forced to manage data on an unprecedented scale, requiring some pioneering techniques. And, in some cases, these can be seen to gradually filter down to a wider range of businesses.
And why reveal their trade secrets to the wider world? In return for offering others a glimpse of their internal systems, these internet firms gain access to a vibrant community of developers that can improve their own technology for free. For example, Facebook has claimed that its Open Compute Project hassaved it $2 billion in data centre costs.
Here are some of the innovative technologies from the big web companies that are now seeing wider use.
1. Google - MapReduce
Google has released over 20 million lines of code and hundreds of open source projects.
One of its most notable creations has been the MapReduce programming model, which allowed it to crunch huge data sets across large clusters of servers. While it is no longer used at Google, MapReduce's legacy has been the inspiration for the open source Hadoop platform, alongside the Google File System. Hadoop has become widely used in the years since its creation by former Yahoo employee Doug Cutting, with a number of IT vendors selling their own services based on the software.
2. Google - Kubernetes
Containerisation has been one of the biggest buzz words of recent years, though many have pointed out that the technology is not new. Google reportedly used around two billion containers to manage applications in its data centres, relying on its secretive Borg and Omega technologies to run workloads internally for years.
And these platforms have provided the basis for its open source Kubernetes container cluster management platform, which has been made available publically since June 2014. Kubernetes has been picked by a range of large businesses looking for a lightweight alternative to virtual machines.
3. Google - TensorFlow
At the heart of Google's impressive search capabilities for Google Photos, voice recognition tools and Google Translate sits its AI system Tensor Flow.
The machine learning tool was open sourced last month in order to help accelerate wider developments around the technology, which is still in its infancy.
Sign up for Computerworld eNewsletters.