Machine Listening

Course Details

• Credit-Lecture-Lab 3-3-0
• Course Completion Classification: Combined Masters/Doctorate
• Teaching materials
  • Lecture slides
  • Textbooks (references) you will find useful:
    • Spectral Audio Signal Processing by Julius O. Smith
    • Pattern Recognition and Machine Learning by Christopher Bishop
    • Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville
  • Papers from relevant conferences and/or journals will be used as references.

Course Outline

Machine Listening (or Computer Audition) is one of the most widely used fields of artificial intelligence, in addition to Computer Vision. There are already many machine listening services such as speech recognition algorithms, like Siri, and automatic music search through audio fingerprinting, penetrating deep into our lives. In this course, you will learn through a series of lectures about the fundamentals of state-of-the-art the machine learning algorithms used to create artificial hearing or machine listening systems, including:

• LTI systems, convolution theorem, sampling theorem
• Mel spectrogram, MFCC, Spectral envelope, Linear prediction, Phase vocoder
• Onset detection, F0 detection, Chroma, CQT, Key & Chord Estimation
• Deep learning, Back-propagation, CNNs, RNNs, Transformers
• Acoustic scene classification
• Audio-text multimodal learning
• Neural harmonic-plus-noise model

Students will have a chance to actually implement such algorithms in the lab sessions. Finally, we aim to build a real-world system that can be applied to audio/music/auditory perception through the final project.

Syllabus

• Assignments (50%) / Midterm exam (20%) / Final project (30%)
• Attendance Policy
  • Students who are absent for over 1/3 of the class will receive a grade of ‘F’ or ‘U’ for the course. (Exceptions can be made when the cause of absence is deemed unavoidable by the course instructor.)
• Prerequisites
  • Basic knowledge in signal processing and Python experience are not required but preferred. Labs/assignments are in Jupyter Notebook (or Google Colab) format.
• Lecture Plan
  1. 09/04 Introduction
  2. 09/11 Digital Audio Signal Processing I
     1. Lab1: Python, Google Colab, Sinusoid, DFT, FFT, STFT
     2. Assignment 1: due 9/18
  3. 09/18 Digital Audio Signal Processing II
     1. Lab2: LTI systems, convolution theorem, sampling theorem
     2. Assignment 2: due 9/25
  4. 09/25 Acoustic Feature Extraction I
     1. Lab3: Mel spectrogram, MFCC, Spectral envelope, Linear prediction, Phase vocoder
     2. Assignment 3: due 10/16
  5. 10/02 No class (Hangawi holiday)
  6. 10/09 No class (Hangul day)
  7. 10/16 Acoustic Feature Extraction II & Music Analysis
     1. Lab4: Onset detection, F0 detection, Chroma, CQT, Key & Chord Estimation
     2. Assignment 5: due 10/23
  8. 10/23 Linear/logistic regression
     1. Lab5: PyTorch, Linear/logistic regression
     2. Assignment 6: due 10/30
  9. 10/30 Midterm Exam
     1. Project proposal: due 11/13
  10. 11/06 Deep learning I
      1. Lab6: PyTorch, Deep learning, Back-propagation
      2. Assignment 6: due 11/13
  11. 11/13 Project Proposal Presentation
  12. 11/20 Deep learning II
      1. Lab7: CNNs, RNNs, Transformers
      2. Assignment 7: due 11/27
  13. 11/27 Deep learning for audio applications I
      1. Lab8: Acoustic scene classification
      2. Project Milestone: due 12/04
  14. 12/04 Project Interim Presentations
      1. Assignment 8: due 12/11
  15. 12/11 Deep learning for audio applications II
      1. Lab9: Audio-text multimodal learning
  16. 12/18 Deep learning for audio applications III
      1. Lab10: Neural harmonic-plus-noise model
  17. 12/22 Project Final Presentations
      1. Project Writeup: due 12/27