Confluent Real Time Data Streaming Management »

The way that we analyze data is changing. What took days to peruse now takes a few minutes to process, thanks to big data. However, as the amount of data gets bigger, much like our expanding universe, businesses are left with the same old tools that are inadequate to solve newer problems. Since time is equal to money, large companies cannot afford to misconstrue the facts. Everything has to be in real-time.


One startup is on a quest to transform how businesses manage their data. Confluent, a Palo Alto-based company founded by former engineers at LinkedIn, is leading the data stream management industry. Its secret weapon? Meet Apache Kafka, an open source messaging system that is real-time, very scalable, and fault tolerant. In fact, it’s too fast that it’s able to transport high volumes of data with millisecond latency.

jay-krepsConfluent is the brainchild of Jay Kreps, Neha Narkhede, and Jun Rao. The trio met while working at LinkedIn, where they developed Apache Kafka as an internal system to analyze the social network’s events—around 300 billion user events per day. At LinkedIn, events happen when a user views another profile, likes a post, or shares an article. The problem was too much data was lost and there was no other way to track it. Fortunately, Kreps’s had an idea of a moving system that can display high volumes of data in real-time.

The success of Apache Kafka attracted numerous companies like Netflix, Uber, Airbnb, Goldman Sachs, PayPal, Yahoo, Square, and Cisco. The project was then open-sourced and donated to the Apache Software Foundation, which skyrocketed its adoption. Seeing a business opportunity to commercialize the project, Jay Kreps, Neha Narkhede, and Jun Rao founded Confluent. LinkedIn became its first investor. Today, Confluent has raised over $31 million from top investors.

Kafka has been a labor of love and it’s been thrilling to see the technology mature and advance. We began working together on Kafka and stream processing in 2010, and now Kafka processes 867 billion messages a day at LinkedIn,” said Jay Kreps in July. “Companies from all sectors are looking to evolve their data architecture to enable real-time stream data processing, and with the funding, we’ll be able to accelerate the development of the Confluent Platform to help companies best leverage the power of Apache Kafka.”

By Gene Briones

Leave a Comment