![]() The replenishment plans are then published through different Kafka Topics to Kafka Consumers in near real-time. This is analyzed in a planning engine that factors in existing inventory, forecasts, lead times, shipping times, etc. Walmart can process tens of billions of messages from 100 million different product SKUs in less than three hours, with peaks of up to 85 GB per minute. Using Kafka along with Apache Spark in a micro-batch architecture, data is streamed via 18 Kafka Broker servers each managing 20+ Topics (feeds) with 500+ partitions. Get more details from this Kafka Summit presentation.Īs for real-time replenishment of its massive distribution centers, Walmart built a messaging system that minimized cycle times and complexity, maintained high accuracy and speed, and was both elastic and resilient. To reduce latency and unreliable data, Walmart designated a specific partition for every item store. Partitions allow Kafka to scale, but doing so efficiently and cost-effectively required a lot of under-the-hood testing and optimization by Walmart’s data engineers. Rather than trying to force everyone on to a single standard, Walmart’s central IT team created a “smart transformation engine” that converted all ingested data into a common standard for storage (see below). There were more than 10 event sources, all using different schemas. For its inventory system, Walmart had to solve three challenges: Its real-time replenishment system, which resupplies Walmart warehouses when goods are low, also relies on Kafka.īuilding these systems to support Walmart’s scale and complexity was not trivial. Walmart houses its real-time inventory data using Kafka Streams and a Kafka connector to ingest the data into Apache Cassandra and other databases. Walmart is well-known for the size, speed, and efficiency of its global supply chain, which draws from 100,000 vendors and ships to 12,000 stores and delivery locations.Ĭrucial to orchestrating this is Kafka. The largest brick-and-mortar retailer in the world, Walmart is also one of the most innovative IT users, investing $12 billion annually on IT, just behind Amazon and Google’s parent company, Alphabet. Key Stats (May 2022) : Tens of billions of messages from 100 million SKUs in three hours for real-time replenishment system. Walmart: Vaunted Real-Time Supply Chain Powered by Kafka JP Morgan Chase optimizes its data operations using a data mesh).Netflix keeps its massive data infrastructure cost-effective.LinkedIn scaled its analytical data platform to beyond one exabyte.Spotify upgraded its event streaming and data orchestration platforms.Facebook’s unified Techtonic file system creates efficiency from exascale storage.(Read other blogs in our series on Data Engineering Best Practices, including how: And I’ll also explain how companies that aren’t operating with the scale, budgets, and engineering manpower of a Walmart, LinkedIn or Uber can still efficiently manage and optimize their Kafka systems. In this blog, I’ll detail how Big Tech companies manage and optimize their cutting-edge Kafka deployments. Nevertheless, Kafka’s reputation as being complicated for companies to set up and manage and challenging to optimize is not undeserved. Overall, Kafka remains dominant due to its vaunted reliability, massive scalability, wide compatibility with other data and analytics tools, and flexibility, as it can be run on-premises, hosted in any number of public cloud providers, or as a fully-managed cloud-native service such as Confluent. While some Big Tech companies like Spotify have responded by moving off Kafka, many others like Twitter continue to deploy Kafka or expand their use. All claim some combination of easier manageability, lower cost, and/or similar near-real-time performance as Kafka. Since Kafka was open-sourced in 2011, a plethora of alternative event streaming, messaging and pub/sub (publish-subscribe) systems have risen to challenge Kafka: Flink, RabbitMQ, AWS Kinesis, Google Pub/Sub, Azure Event Hub, and others. Meanwhile, Chinese social media company Tencent (maker of the popular WeChat and QQ instant messaging apps) processes more than 10 trillion messages per day using Kafka. ![]() LinkedIn, which created and open-sourced Kafka and still drives much of its development, processes 7 trillion messages per day using Kafka (a 2019 statistic that is no doubt much higher today). Uber, which processes trillions of messages and multiple petabytes of data each day with Kafka, calls it the “cornerstone of our technology stack.” ![]() The popular open-source messaging/streaming system, Apache Kafka, is a key enabler for some of the most data-driven and disruptive companies today.
0 Comments
Leave a Reply. |