Course syllabus and policies: Course handout.

Teaching staff:

• Instructor and office hours:
  • Jimmy Ba, Tues 5-6
  • Bo Wang, Fri 10-11
• Head TA: Harris Chan

Contact emails:

• Instructor: csc413-2021-01@cs.toronto.edu
• TAs and instructor: csc413-2021-01-tas@cs.toronto.edu

Please do not send the instructor or the TAs email about the class directly to their personal accounts.

Piazza: Students are encouraged to sign up Piazza to join course discussions. If your question is about the course material and doesn’t give away any hints for the homework, please post to Piazza so that the entire class can benefit from the answer.

Lecture and tutorial hours:

	Time	Location
Lecture	Tuesday 3-5 pm	YouTube
Tutotiral	Thursday 3-4 pm	Zoom

Online lectures and tutorials: The access to online lectures and tutorials will be communicated via course mailing list. Course videos and materials belong to your instructor, the University, and/or other sources depending on the specific facts of each situation, and are protected by copyright. Do not download, copy, or share any course or student materials or videos without the explicit permission of the instructor. For questions about recording and use of videos in which you appear please contact your instructor.

• Apr 6: Homework 4 is updated to v1.2 and due date is now Apr. 8 Apr. 11.
• Mar 30: Course Project due date is now Apr. 12 Apr. 18.
• Mar 26: Homework 4 handout is now online and is due April 8th.
• Mar 19: Programming Assignment 4 handout, starter code 1, starter code 2, and starter code 3 are online and are due April 1st.
• Mar 06: Programming Assignment 3 handout, starter code 1 and starter code 2 are due Mar 20th.
• Feb 21: Programming Assignment 2 handout and starter code v1.1 is updated with clarifications and a new due date of Feb 28th.
• Feb 20: Homework 3 handout is now online and is due Mar 13th.
• Feb 1: Project Guideline handout is now online.
• Jan 28: Homework 2 handout is now online and is due Feb 11th.
• Jan 22: Programming Assignment 1 handout and the starter code are now online. Make sure you create a copy in your own Drive before making edits, or else the changes will not be saved.
• Jan 17: Homework 1 handout updated to Version 1.1.
• Jan 17: Homework 1 handout is now online and is due Jan 28th.

It is very hard to hand design programs to solve many real world problems, e.g. distinguishing images of cats v.s. dogs. Machine learning algorithms allow computers to learn from example data, and produce a program that does the job. Neural networks are a class of machine learning algorithm originally inspired by the brain, but which have recently have seen a lot of success at practical applications. They’re at the heart of production systems at companies like Google and Facebook for image processing, speech-to-text, and language understanding. This course gives an overview of both the foundational ideas and the recent advances in neural net algorithms.

	Handout	Due
Homework 1	pdf	Jan. 17(out), due Jan. 28
Programming Assignment 1	starter code(make a copy in your own Drive), pdf	Jan. 22(out), due Feb. 04
Homework 2	pdf	Jan. 28(out), due Feb. 14
Programming Assignment 2	starter code(make a copy in your own Drive), pdf	Feb. 05(out), due Feb. 28
Homework 3	pdf	Feb. 20(out), due Mar. 13
Programming Assignment 3	starter code 1, starter code 2(make a copy in your own Drive), pdf	Mar. 06(out), due Mar. 20
Programming Assignment 4	starter code 1, starter code 2, starter code 3 (make a copy iin your own Drive), pdf	Mar. 19(out), due Apr. 01
Homework 4	pdf	Mar. 26(out), due Apr. 08 Apr. 11
Course Project	handout	due Apr. 12 Apr. 18

Midterm Quiz: Feb. 09

The midterm quiz will cover the lecture materials up to lecture 4. The exact details will be announced soon.

Suggested readings included help you understand the course material. They are not required, i.e. you are only responsible for the material covered in lecture. Most of the suggested reading listed are more advanced than the corresponding lecture, and are of interest if you want to know where our knowledge comes from or follow current frontiers of research.

	Date	Topic	Slides	Suggested Readings
Lecture 1	Jan 12	Introduction & Linear Models	Slides	Roger Grosse’s notes: Linear Regression, Linear Classifiers, Training a Classifier
Tutorial 1	Jan 14	Multivariable Calculus Review	ipynb	iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 2	Jan 19	Multilayer Perceptrons & Backpropagation	Slides	Roger Grosse’s notes: Multilayer Perceptrons, Backpropagation
Tutorial 2	Jan 21	Autograd and PyTorch	ipynb	iPython notebook: ipynb, you may view the notebook via Colab.
Lecture 3	Jan 26	Distributed Representations & Optimization	Slides	Roger Grosse’s notes: Automatic Differentiation, Distributed Representations, Optimization
Tutorial 3	Jan 28	How to Train Neural Networks	Slides, code demo
Lecture 4	Feb 02	Convolutional Neural Networks and Image Classification	Slides	Roger Grosse’s notes: ConvNets, Image Classification. Related papers: Yann LeCun’s 1998 LeNet, AlexNet.
Tutorial 4	Feb 04	Convolutional Neural Networks	ipynb	iPython notebook: ipynb, you may view the notebook via Colab
Lecture 5	Feb 09	Interpretability	Slides	Related papers: Sanity Check for Saliency Maps, SmoothGrad, Towards a rigorous science of interpretable machine learning.
Midterm Quiz	Feb 09
Tutorial 5	Feb 11	How to Write a Good Course Project Report	Slides
Reading Week	Feb 16
Lecture 6	Feb 23	Optimization & Generalization	Slides	Roger Grosse’s notes: Generalization, Exploding Vanishing Gradients Related papers: Dropout, ResNet
Tutorial 6	Feb 25	Best Practices of ConvNet Applications	Slides
Lecture 7	Mar 02	Recurrent Neural Networks and Attention	Slides	Roger Grosse’s notes: RNNs, Exploding Vanishing Gradients. Related papers: LSTM, ResNet, Neural machine translation, Show, attend and tell.
Tutorial 7	Mar 04	Recurrent Neural Networks	Slides, ipynb	You may view the notebook on Colab
Lecture 8	Mar 09	Transformers and Autoregressive Models	Slides	Related papers: Transformers, BERT pre-training, PixelRNNs, WaveNet, PixelCNNs.
Tutorial 8	Mar 11	NLP and Transformers	Slides
Lecture 9	Mar 16	Reversible Models & Generative Adversarial Networks	Slides	Related papers: Generative adversarial networks, CycleGANs, RealNVP.
Tutorial 9	Mar 18	Information Theory	Slides
Lecture 10	Mar 23	Generative Models & Reinforcement Learning	Slides	Related papers: RealNVP, Variational Auto-encoder, Policy Gradients for Robotics, Proximal Policy Optimization Algorithm.
Tutorial 10	Mar 25	Generative Adversarial Networks	Slides
Lecture 11	Mar 30	Q-learning & the Game of Go	Slides	Related papers: Deep Q Network, AlphaGo, AlphaZero.
Tutorial 11	Apr 01	Policy Gradient and Reinforcement Learning	Slides, code demo
Lecture 12	Apr 06	Recent Trends in Deep Learning

Type	Name	Description
Related Textbooks	Deep Learning (Goodfellow at al., 2016)	The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning.
	Information Theory, Inference, and Learning Algorithms (MacKay, 2003)	A good introduction textbook that combines information theory and machine learning.
General Framework	PyTorch	An open source deep learning platform that provides a seamless path from research prototyping to production deployment.
Computation Platform	Colab	Colaboratory is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.
	GCE	Google Compute Engine delivers virtual machines running in Google’s innovative data centers and worldwide fiber network.
	AWS-EC2	Amazon Elastic Compute Cloud (EC2) forms a central part of Amazon.com’s cloud-computing platform, Amazon Web Services (AWS), by allowing users to rent virtual computers on which to run their own computer applications.