data pipelines

Saturday, February 13, 2021

- PST
Scaling Scala @ Spotify
Join Stream

For the last few years, we at Spotify have been developing and using Scio, an open-source Scala framework, to develop data pipelines. During that time, Spotify has been successfully deploying and running thousands of unique Scio jobs in production.

The scale of our deployments is one of the challenges we have to face, in terms of the amount of data we process, of course, but most importantly, in the number of engineers who will interact with our platform.

This talk will be an exploration of our experience using Scala in a large and rapidly growing company, and the unique strengths of the language one may leverage to reach true scalability.