Wednesday, February 17, 2021

PRO WORKSHOP: Fugue: A New Way to Think and Solve Distributed Computing Problems
Join on Hopin
Han Wang
Han Wang
Lyft, Staff Engineer, Tech Lead

Through my career, I have seen and solved so many different computing problems. I find the challenge is not we can't solve large scale problems, the challenge is not we are lack of great tools. The real challenge is lack of consistency: solving different scale problems requires different programming models and frameworks, solving ETL and machine learning problems requires different skillsets, mindsets and even different languages.

In this session, I am going to introduce the Fugue project, whose purpose is to:
* unify the key concepts of distributed computing and machine learning problems
* unify the key interfaces for end to end ETL and machine learning pipelines
* decouple user's code and mindset from the underlying compute engines such as Spark and Ray and Fugue itself

I am also going to go through a couple of real examples how Fugue can shape your mind and optimize your code.

Fugue is already open sourced at: