Page 2 - Apache Beam News Today

PipelineDP allows datasets containing personal information to be aggregated in a way that preserves the privacy of individuals.

Google is extending differential privacy capabilities to the Python language, with an open source tool for creating pipelines that aggregate data containing personal information in a way that preserves the privacy of individuals. PipelineDP, developed in partnership with OpenMined and accessible from the project website, is still in an experimental stage. With differential privacy, useful […]

PipelineDP allows datasets containing personal information to be aggregated in a way that preserves the privacy of individuals.

Streaming data records are typically small, measured in mere kilobytes, but the stream often goes on and on without ever stopping. Many of the best platforms for managing, storing, and analyzing streaming data are Apache open source projects.

Streaming data, also called event stream processing, is usually discussed in the context of big data. It is data that is generated continuously, often by thousands of data sources, such as sensors or server logs. Streaming data records are often small, perhaps a few kilobytes each, but there are many of them, and in many […]

February 11, 2021 Published by Neville Li, Claire McGinty, Sahith Nallapareddy, & Joel Östlund In this post we’ll discuss how Spotify optimized and sped up elements from our largest Dataflow job, Wrapped 2019, for Wrapped 2020 using a technique called Sort Merge Bucket (SMB) join. We’ll present the design and implementation of SMB and how we incorporated it into our data pipelines. Introduction Shuffle is the core building block for many ...