AstriCon 2018

Machine learning is an exploding field - with the arrival of cloud GPU instances and libraries like TensorFlow and PyTorch, models can be trained in ways that were never practical before. For us working in RTC, we can now leverage those tools to both solve issues with realtime communications and create new ones. Who’s interested in crime?In this talk we’ll discuss the future of ML with RTC. We’ve recently seen rapid advances in ASR, TTS, and machine translation. Incorporating these directly into our platforms is now a simple API call away. Next up is optimization of audio and video using models like Google’s RAISR, allowing us to realtime upscale the image stream on the client side with no artifacting. Similarly we can do advanced noise suppression using tools like Mozilla’s RRNoise or, even more exciting, only send a subset of audio and recreate speech on the far side using a model of the speakers voice.Machine learning works best when there is voluminous amounts of well-structured data – something that we generate each call in spades! We can also use machine learning to analyze data and watch metrics during a session to re-route or optimize bitstreams accordingly. After the call, we can take a look at the statistics to improve future performance.But what if you weren’t the ethical, upstanding citizen you are today? These tools can also be used for nefarious purposes. Imagine receiving a call from someone you loved saying there were in the hospital, so you quickly leave your house unlocked. And are we OK putting voice over actors out of work? We’ll talk through some of the societal repercussions of these advances.