Profile banner
Vaibhav Goyal

Vaibhav Goyal

Principal Engineer at Nielsen, focused on data and distributed systems

vaibhav-goyal
Gurgaon, Haryana, India
Joined October 2024
Favicon imagein.linkedin.comFavicon imagemedium.com

Network

477 connections

Summary

Vaibhav Goyal is an experienced software engineer specializing in building scalable and high-performance data processing systems, particularly with Apache Kafka. His work at LocoNav involved designing and managing Kafka clusters that processed billions of messages daily with high availability and low latency for fleet management and IoT solutions. He has experience with both the ingestion layer and stream processing using Kafka Streams. medium+1
He possesses strong expertise in data engineering and distributed systems, as evidenced by his detailed blog posts on managing self-hosted Kafka, handling back pressure, and implementing robust ingestion architectures for complex and high-volume data from diverse IoT devices. His technical skills include GoLang, Kafka, AWS, and Prometheus/Grafana for monitoring. medium+2

Work

Education

Writing

Art of ingesting data at LocoNav

June 1, 2023

Explores the challenges of ingesting heterogeneous data from millions of IoT devices at LocoNav, detailing the GoLang-based ingestion layer, parsing mechanisms, local buffering, and the role of Kafka in handling data surges and bi-directional communication.

Favicon imagemedium.com

Processing 2 Trillion messages with self-hosted Kafka

May 1, 2023

Details the architecture and implementation of a self-hosted Kafka cluster at LocoNav that processed over 2 trillion messages, emphasizing scalability, fault tolerance, and low-latency data processing for fleet management solutions.

Favicon imagemedium.com