CSC413/2516 Winter 2021
Neural Networks and Deep Learning


Course syllabus and policies: Course handout.

Teaching staff:

  • Instructor and office hours:
  • Head TA: Harris Chan

Contact emails:

Please do not send the instructor or the TAs email about the class directly to their personal accounts.

Piazza: Students are encouraged to sign up Piazza to join course discussions. If your question is about the course material and doesn’t give away any hints for the homework, please post to Piazza so that the entire class can benefit from the answer.

Lecture and tutorial hours:

  Time Location
Lecture Tuesday 3-5 pm YouTube
Tutotiral Thursday 3-4 pm Zoom

Online lectures and tutorials: The access to online lectures and tutorials will be communicated via course mailing list. Course videos and materials belong to your instructor, the University, and/or other sources depending on the specific facts of each situation, and are protected by copyright. Do not download, copy, or share any course or student materials or videos without the explicit permission of the instructor. For questions about recording and use of videos in which you appear please contact your instructor.



Announcements:

  • Jan 17: Homework 1 handout updated to Version 1.1.
  • Jan 17: Homework 1 handout is now online and is due Jan 28th.

Course Overview:

It is very hard to hand design programs to solve many real world problems, e.g. distinguishing images of cats v.s. dogs. Machine learning algorithms allow computers to learn from example data, and produce a program that does the job. Neural networks are a class of machine learning algorithm originally inspired by the brain, but which have recently have seen a lot of success at practical applications. They’re at the heart of production systems at companies like Google and Facebook for image processing, speech-to-text, and language understanding. This course gives an overview of both the foundational ideas and the recent advances in neural net algorithms.


Assignments:

  Handout Due
Homework 1 pdf Jan. 17(out), due Jan. 28
Programming Assignment 1 TBD Jan. 21(out), due Feb. 04
Homework 2 TBD Jan. 28(out), due Feb. 11
Programming Assignment 2 TBD Feb. 04(out), due Feb. 25
Homework 3 TBD Feb. 18(out), due Mar. 11
Programming Assignment 3 TBD Mar. 04(out), due Mar. 18
Programming Assignment 4 TBD Mar. 18(out), due Apr. 01
Homework 4 TBD Mar. 25(out), due Apr. 08
Course Project Guideline soon due Apr. 12



Midterm Quiz: Feb. 09

The midterm quiz will cover the lecture materials up to lecture 4. The exact details will be announced soon.


Calendar:

Suggested readings included help you understand the course material. They are not required, i.e. you are only responsible for the material covered in lecture. Most of the suggested reading listed are more advanced than the corresponding lecture, and are of interest if you want to know where our knowledge comes from or follow current frontiers of research.

  Date     Topic Slides Suggested Readings
Lecture 1 Jan 12 Introduction & Linear Models Slides Roger Grosse’s notes: Linear Regression, Linear Classifiers, Training a Classifier
Tutorial 1 Jan 14 Multivariable Calculus Review ipynb Python notebook: ipynb, you may view the notebook via Colab.
Lecture 2 Jan 19 Multilayer Perceptrons & Backpropagation   Roger Grosse’s notes: Multilayer Perceptrons, Backpropagation
Tutorial 2 Jan 21 Autograd and PyTorch    
Lecture 3 Jan 26 Distributed Representations & Optimization    
Tutorial 3 Jan 28 How to Train Neural Networks    
Lecture 4 Feb 02 Convolutional Neural Networks and Image Classification    
Tutorial 4 Feb 04 Convolutional Neural Networks    
Lecture 5 Feb 09 Interpretability    
Midterm Quiz Feb 09      
Tutorial 5 Feb 11 How to Write a Good Course Project Report    
Reading Week Feb 16      
Lecture 6 Feb 23 Optimization & Generalization    
Tutorial 6 Feb 25 Best Practices of ConvNet Applications    
Lecture 7 Mar 02 Recurrent Neural Networks and Attention    
Tutorial 7 Mar 04 Recurrent Neural Networks    
Lecture 8 Mar 09 Transformers and Autoregressive Models    
Tutorial 8 Mar 11 NLP and Transformers    
Lecture 9 Mar 16 Reversible Models & Generative Adversarial Networks    
Tutorial 9 Mar 18 Information Theory    
Lecture 10 Mar 23 Generative Models & Reinforcement Learning    
Tutorial 10 Mar 25 Generative Adversarial Networks    
Lecture 11 Mar 30 Q-learning & the Game of Go    
Tutorial 11 Apr 01 Policy Gradient and Reinforcement Learning    
Lecture 12 Apr 06 TBD    

Resource:

Type Name Description
Related Textbooks Deep Learning (Goodfellow at al., 2016) The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning.
  Information Theory, Inference, and Learning Algorithms (MacKay, 2003) A good introduction textbook that combines information theory and machine learning.
General Framework PyTorch An open source deep learning platform that provides a seamless path from research prototyping to production deployment.
Computation Platform Colab Colaboratory is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.
  GCE Google Compute Engine delivers virtual machines running in Google’s innovative data centers and worldwide fiber network.
  AWS-EC2 Amazon Elastic Compute Cloud (EC2) forms a central part of Amazon.com’s cloud-computing platform, Amazon Web Services (AWS), by allowing users to rent virtual computers on which to run their own computer applications.