How Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020 : Spotify Engineering
February 11, 2021 Published by Neville Li, Claire McGinty, Sahith Nallapareddy, & Joel Östlund In this post we’ll discuss how Spotify optimized and sped up elements from our largest Dataflow job, Wrapped 2019, for Wrapped 2020 using a technique called Sort Merge Bucket (SMB) join. We’ll present the design and implementation of SMB and how we incorporated it into our data pipelines. Introduction Shuffle is the core building block for many big data transforms, such as a join, GroupByKey, or other reduce operations. Unfortunately, it’s a...
Source: atspotify.com