Brand-New Attention Chapter

We have added the brand-new Chapter: Attention Mechanisms:

• Attention Cues

  • Attention Cues in Biology
  • Queries, Keys, and Values
  • Visualization of Attention

• Attention Pooling: Nadaraya-Watson Kernel Regression

  • Generating the Dataset
  • Average Pooling
  • Nonparametric Attention Pooling
  • Parametric Attention Pooling

• Attention Scoring Functions

  • Masked Softmax Operation
  • Additive Attention
  • Scaled Dot-Product Attention

• Bahdanau Attention

  • Model
  • Defining the Decoder with Attention
  • Training

• Multi-Head Attention

  • Model
  • Implementation

• Self-Attention and Positional Encoding

  • Self-Attention
  • Comparing CNNs, RNNs, and Self-Attention
  • Positional Encoding

• Transformer

  • Model
  • Positionwise Feed-Forward Networks
  • Residual Connection and Layer Normalization
  • Encoder
  • Decoder
  • Training

PyTorch Adaptation Completed

We have completed PyTorch implementations for Vol.1 (Chapter 1--15).

Towards v1.0

The following chapters have been significantly improved for v1.0:

• Introduction
• Modern Recurrent Neural Networks

Chinese Translation

The following chapters have been translated into Chinese (d2l-zh v2 Git repo, Web preview):

• Introduction
• Preliminaries
• Linear Neural Networks
• Multilayer Perceptrons
• Deep Learning Computation
• Convolutional Neural Networks
• Modern Convolutional Neural Networks

Turkish Translation

The community are translating the book into Turkish (d2l-tr Git repo, Web preview). The first draft of Chapter 1--7 is complete.

Framework Adaptation

We have added PyTorch implementations up to Chapter 11 (Optimization Algorithms). Chapter 1--7 and Chapter 11 have also been adapted to TensorFlow.

Towards v1.0

The following chapters have been significantly improved for v1.0:

• Linear Neural Networks
• Multilayer Perceptrons
• Deep Learning Computation
• Convolutional Neural Networks
• Modern Convolutional Neural Networks
• Recurrent Neural Networks

Finalized chapters are being translated into Chinese (d2l-zh v2)

Other Improvements

• Fixed issues of not showing all the equation numbers in the HTML and PDF
• Consistently used f-string
• Revised overfitting experiments
• Fixed implementation errors for weight decay experiments
• Improved layer index style
• Revised "breaking the symmetry"
• Revised descriptions of covariate and label shift
• Fixed mathematical errors in covariate shift correction
• Added true risk, empirical risk, and (weighted) empirical risk minimization
• Improved variable naming style for matrices and tensors
• Improved consistency of mathematical notation for tensors of order two or higher
• Improved mathematical descriptions of convolution
• Revised descriptions of cross-correlation
• Added feature maps and receptive fields
• Revised mathematical descriptions of batch normalization
• Added more details to Markov models
• Fixed implementations of k-step-ahead predictions in sequence modeling
• Fixed mathematical descriptions in language modeling
• Improved the d2l.Vocab API
• Fixed mathematical descriptions and figure illustrations for deep RNNs
• Added BLEU
• Improved machine translation application results
• Improved the animation plot function in the all the training loops

Highlights

We have added both PyTorch and TensorFlow implementations up to Chapter 7 (Modern CNNs).

Improvements

• We updated the text to be framework neutral, such as now we call ndarray as tensor.
• Readers can click the tab in the HTML version to switch between frameworks, both colab button and discussion thread will change properly.
• We changed the release process, d2l.ai will host the latest release (i.e. the release branch), instead of the contents from the master branch. We unified the version number of both text and the d2l package. That's why we jumped from v0.8 to v0.14.0
• The notebook zip contains three folders, mxnet, pytorch and tensorflow (though we only build the PDF for mxnet yet).

Highlights

D2L is now runnable on Amazon SageMaker and Google Colab.

New Contents

The following chapters are re-organized:

• Natural Language Processing: Pretraining
• Natural Language Processing: Applications

The following sections are added:

• Subword Embedding (Byte-pair encoding)
• Bidirectional Encoder Representations from Transformers (BERT)
• The Dataset for Pretraining BERT
• Pretraining BERT
• Natural Language Inference and the Dataset
• Natural Language Inference: Using Attention
• Fine-Tuning BERT for Sequence-Level and Token-Level Applications
• Natural Language Inference: Fine-Tuning BERT

Improvements

There have been many light revisions and improvements throughout the book.

Highlights

• D2L is now based on the NumPy interface. All the code samples are rewritten.

New Contents

• Recommender Systems

  • Overview of Recommender Systems
  • The MovieLens Dataset
  • Matrix Factorization
  • AutoRec: Rating Prediction with Autoencoders
  • Personalized Ranking for Recommender Systems
  • Neural Collaborative Filtering for Personalized Ranking
  • Sequence-Aware Recommender Systems
  • Feature-Rich Recommender Systems
  • Factorization Machines
  • Deep Factorization Machines

• Appendix: Mathematics for Deep Learning

  • Geometry and Linear Algebraic Operations
  • Eigendecompositions
  • Single Variable Calculus
  • Multivariable Calculus
  • Integral Calculus
  • Random Variables
  • Maximum Likelihood
  • Distributions
  • Naive Bayes
  • Statistics
  • Information Theory

• Attention Mechanisms

  • Attention Mechanism
  • Sequence to Sequence with Attention Mechanism
  • Transformer

• Generative Adversarial Networks

  • Generative Adversarial Networks
  • Deep Convolutional Generative Adversarial Networks

• Preliminaries

  • Data Preprocessing
  • Calculus

Improvements

• The Preliminaries chapter is improved.
• More theoretical analysis is added to the Optimization chapter.

Preview Version

Hard copies of a D2L preview version based on this release (excluding chapters of Recommender Systems and Generative Adversarial Networks) are distributed at AWS re:Invent 2019 and NeurIPS 2019.

Fixes

• Fixed issues in sections of Machine Translation and the Dataset, Encoder-Decoder Architecture, Sequence to Sequence, Beam Search, Attention Mechanism, Sequence to Sequence with Attention Mechanism, and Transformer.

Textbook/Reference Book Adoption at Universities

We thank instructors and students for providing feedbacks during use.

USA

• University of California, Berkeley
  • STAT 157 Introduction to Deep Learning
• Georgia Institute of Technology
  • CX4240: Introduction to Computational Data Analysis
• University of North Carolina at Chapel Hill
  • INLS 690-270 - Data Mining: Methods and Applications
• University of California, Los Angeles
  • CS269 Foundations of Deep Learning
• University of California, Santa Barbara
  • CS165A: Artificial Intelligence
• University of Illinois at Urbana-Champaign
  • CS 498 Introduction to Deep Learning
• University of Maryland
  • Introduction to Deep Learning
• University of Washington
  • CSE 599W Systems for ML
• University of New Hampshire
  • CS 953 Adv Top / Data Science for Knowledge Graphs and Text

China

• University of Science and Technology of China
  • Deep Learning
• Shanghai University of Finance and Economics
  • Deep Learning
• Zhejiang University
  • Graduate course: IoT and Information Processing

Australia

• University of Technology Sydney
  • 42028 Deep Learning and Convolutional Neural Network

Spain

• Universitat Politècnica de Catalunya
  • Master Course: Deep Learning for Artificial Intelligence

Change of Contents

We heavily revised the following chapters, especially during teaching STAT 157 at Berkeley.

• Preface
• Installation
• Introduction
• The Preliminaries: A Crashcourse
• Linear Neural Networks
• Multilayer Perceptrons
• Recurrent Neural Networks

The Community Are Translating D2L into Korean and Japanese

d2l-ko in Korean (website: ko.d2l.ai) joins d2l.ai! Thank Muhyun Kim, Kyoungsu Lee, Ji hye Seo, Jiyang Kang and many other contributors!

d2l-ja in Japanese (website: ja.d2l.ai) joins d2l.ai! Thank Masaki Samejima!

Thanks to Our Contributors

@alxnorden, @avinashingit, @bowen0701, @brettkoonce, Chaitanya Prakash Bapat, @cryptonaut, Davide Fiocco, @edgarroman, @gkutiel, John Mitro, Liang Pu, Rahul Agarwal, @mohamed-ali, @mstewart141, Mike Müller, @NRauschmayr, @prakhar Srivastav, @sad-, @sfermigier, Sheng Zha, @sundeepteki, @topecongiro, @tpdi, @vermicelli, Vishaal Kapoor, @vishwesh5, @YaYaB, Yuhong Chen, Evgeniy Smirnov, @lgov, Simon Corston-Oliver, @IgorDzreyev, @trungha-ngx, @pmuens, @alukovenko, @senorcinco, @vfdev-5, @dsweet, Mohammad Mahdi Rahimi, Abhishek Gupta, @uwsd, @DomKM, Lisa Oakley, @vfdev-5, @bowen0701, @arush15june, @prasanth5reddy.