Namaste! Remember the quote „an apple a day keeps the doctor away“? This rather silly saying expresses that you should make sure that you get enough vitamins on an everyday basis. In Deep Learning it is advised to keep up to date. I always tell people that one of the greatest challenges in Deep Learning is not the Neural Network architecture and training. It is staying up to date. And boy, there is so much going on!
DeepMind and WaveNet.
DeepMind is doing excellent work in the field of AI. Their approach to Deep Reinforcement Learning finally solved the game Go. And they are also responsible for WaveNet, a Neural Net that is capable of generating raw audio signals. WaveNet can be used for synthesizing speech and even music. In a very convincing manner.
When you read their paper about WaveNet you learn many things. One of them is that it manages to deal with temporal relationships in audio-signals with convolutions. Dilated convolutions to be exact. Remember convolutions? CNNs? Yes, the idea of embedding one mathematical function into a second. This helped boosting Deep Learning for Digital Signal Processing a lot in the past. ImageNet competition. Today convolutions are well established and you cannot imagine a single Deep Learning day without.
Making WaveNet faster.
Today I read yet another WaveNet paper. On the one hand, WaveNet is very convincing when it comes to synthesis. Principal problem solved. This is already a great plus. On another hand, training is very straightforward. Because of the convolutions, training can be massively parallelized. But on the third and final hand (hope you counted the hands, too), WaveNet in its original manifestation is really slow when deployed into production. Enter Parallel WaveNet!
A slight change in architecture yielded a 1000-times increase in performance. Before that long audio waveforms had to be synthesized sample by sample. In a sequential – that is non-parallel – way. This is different with Parallel WaveNet. How? The developers set up a student-teacher-relationship and train in a way similar to Generative Adversarial Neural Networks (GANs). Similar but not the same. There is a big difference. The student is not supposed to fool the teacher as in the GAN setting. Instead the student is taught to cooperate. Very nice idea!
But wait… Why would this be faster? Well, the student has a completely different architecture than the teacher. The teacher is original WaveNet, which can already solve the task at hand. But it is massively slow. The student on the other hand can be parallelized and at the same time trained with the knowledge of the teacher. Great!
Keep on reading.
It is fantastic what is happening in the Deep Learning world. You can spend many hours per week just studying what is going on. And there is more everyday. But what is most important? Putting all into practice. Make sure that once in a while you do some prototyping and talk about it. Doing so, you will learn the most!
Stay in touch.
I hope you liked the article. Why not stay in touch? You will find me at LinkedIn, XING and Facebook. Please add me if you like and feel free to like, comment and share my humble contributions to the world of AI. Thank you!
If you want to become a part of my mission of spreading Artificial Intelligence globally, feel free to become one of my Patrons. Become a Patron!
A quick about me. I am a computer scientist with a love for art, music and yoga. I am a Artificial Intelligence expert with a focus on Deep Learning. As a freelancer I offer training, mentoring and prototyping. If you are interested in working with me, let me know. My email-address is tristan@ai-guru.de - I am looking forward to talking to you!
